Comments on: User Space Filesystems Formerly known as CloudFS Tue, 26 Mar 2013 22:47:54 +0000 hourly 1 By: Linus vs FUSE | Ceph Linus vs FUSE | Ceph Fri, 08 Jul 2011 21:47:55 +0000 [...] as many have pointed out, calling all such systems “toys” isn’t completely fair. But then it [...]

By: Jitesh Jitesh Thu, 07 Jul 2011 22:55:14 +0000 True, but do you really expect anyone to give a statement that is universally and politically true? Should he think about cloud filesystems and distributed filesystems when making a simple point?

The point being that if the underlying infrastructure/device/whatever-term-you-put-in-here is fundamentally slow, then no one cares about time spent in extra-buffer copies anyway and FUSE is actually *not* useless.

Eventually it depends on how you choose to interpret what he said. To me: the context of the email didn’t include Cloud or distributed FS, so interpreting the statement in that context is just wrong and over-reacting. His claim is very true for local filesystems and that is the area it was intended for.

By: Jeff Darcy Jeff Darcy Fri, 01 Jul 2011 00:55:24 +0000 I don’t think “low-use interface to a fundamentally slow device” is a very good description of the I/O system attached to Intrepid,and I don’t see anything else in that email about bottlenecks being somewhere else, so I’m not sure how you managed to arrive at that interpretation.

By: Jitesh Jitesh Thu, 30 Jun 2011 22:58:56 +0000 Linus agrees with you!

Line from his email
“fuse works fine if the thing being exported is some random low-use interface to a fundamentally slow device.”

He agrees that if the bottlenecks are somewhere else, FUSE is fine, which is sort of the point you are trying to make. Peace :-)

By: Mace Moneta Mace Moneta Thu, 30 Jun 2011 16:39:19 +0000 From a performance perspective, I agree with Linus. From an ease of use and utility perspective, FUSE is great.

I use sshfs even though I have NFS configured, because of the easy adhoc functionality when needed. I use encfs because I only need to encrypt one small directory of personal information, and don’t need the system-wide overhead.

Every tool has its place and purpose.

By: Jeff Darcy Jeff Darcy Thu, 30 Jun 2011 15:43:29 +0000 You’re missing the point on several levels, P.B. I already mentioned the issue of extra data copies, but also made the point that it doesn’t relegate all user-space file systems to toys. Let’s see how many reasons there might be for that.

(1) The copies you mention are artifacts of the Linux FUSE implementation, and are not inherent to user-space file systems in general. Other systems do this more intelligently. PVFS2 does it more intelligently *on Linux*. With RDMA, communication could be direct from the application to application, without even the overhead of going through the kernel. FUSE itself could be more efficient if resistance from the kernel grognards and their sycophants could be overcome. Even if one could make the case that filesystems based on FUSE as it exists today are all toys, Linus’s statement *as he made it* would still be untrue.

(2) The copies don’t matter in many environments, especially in distributed systems. If your system is network, memory, or I/O bound anyway – whether that’s because of provisioning or algorithms – then the copies are only consuming otherwise-idle CPU cycles. This is especially true since most systems sold today are way out of balance in favor of CPU/memory over network or disk I/O anyway.

(3) There’s an important distinction between latency and throughput. The FUSE overheads mostly affect latency. If latency is your chief concern, then you probably shouldn’t be using any kind of distributed file system regardless of whether it’s in kernel or user space. If throughput is your chief concern, which is the more common case, you need a system that allows you to aggregate the power of many servers without hitting algorithmic limits. Such systems are hard enough to debug already, without the added difficulty of putting it them the kernel prematurely. I’m not against putting code in the kernel *when all of the algorithms are settled*, but projects can go well beyond “toy” status well before that.

(4) There are concerns besides performance. There are bazillions of libraries that one can use easily from user space. Many of them can not and should not ever be reimplemented in the kernel simply because that would bloat the kernel beyond belief. In some cases there would be other serious implications, such as a kernel-ported security library losing its certification in the process.

(5) Results from actual field usage trumps synthetic micro-benchmarks any day, and either trumps empty theorizing like yours. If Argonne and Pandora and dozens of others can use PVFS2 and GlusterFS and HDFS for serious work, then they’re not toys. The point is already proven. End of story.

By: P.B.Shelley P.B.Shelley Thu, 30 Jun 2011 14:29:56 +0000 How about admitting that with FUSE, data has to be copied to kernel, then your user space component, and then back to kernel to write to disk? If you claim that this overhead can result in better performance, please PROVE it, instead of citing how many successful user space filesystems you have out there. All of them can perform better if they live in the kernel!

By: Manhong Dai Manhong Dai Thu, 30 Jun 2011 14:18:16 +0000 I have been using glusterfs for 3 years. It connects a 32-node cluster to five 12T storage nodes. It is simple, scalable and very stable.

We tried lusterfs before glusterfs, it is good, but not what we want. Here is the reasons
1, It is tied to a special kernel, which limits the storage’s usability. We want to use the storage for HPC, storage, backup, etc.
2, It requires some dedicated meta nodes, however, we always want to strench each dollar.
3, Now it is owned by Oracle.

We also waited for the pNFS, but my life is too short.

Other than user-space file systems, if somebody knows a open source or free distributed kernel-space file system that can handle millions of small files well for HPC and file service, please let us know.

In my opinion, GlusterFS is the best distributed filesystem as of today. No matter how user-space file system is discriminated, it will grow bigger. Eventually it will put Linux on a bigger stage.

By: Linus Torvalds doesn’t understand user-space filesystems - Gluster Linus Torvalds doesn’t understand user-space filesystems - Gluster Wed, 29 Jun 2011 14:29:33 +0000 [...] Darcy, of Red Hat and CloudFS fame, wrote a wonderful response, which you should read first before continuing [...]

By: Jeff Darcy Jeff Darcy Wed, 29 Jun 2011 14:20:07 +0000 Shawn: Yes, there is Minix, but it’s a dangerous thing to mention in this context ;) Besides, Minix is a pedagogical OS, which is a wonderful thing but for current purposes almost the same as a toy. I think a better microkernel example would be L4, which is clearly being used for non-toy projects and is demonstrably superior to Linux in terms of both performance and security.