The first is a post-checkout script, to regenerate cscope information any time a new branch is checked out. Just put this in $your_git_tree/.git/hooks to use it. If anybody knows how to make this global instead of per-tree, or how to do the same thing for every other git operation that might add/remove source files in your tree, please let me know.

The second is a window-title script which will check to see if you’re in a git tree and, if so, add the current branch to the current directory. For example, my PROMPT_COMMAND looks like this.

‘printf “\033]0;%s@%s:%s\007″ “${USER}” “${HOSTNAME%%.*}” “$(pwd_with_branch)”‘

As a result, one of my window titles now looks like this.

jdarcy@jdarcy-dt:~/projects/glusterfs_github (hsrepl)

Here, “hsrepl” is the name of the branch I’m currently working on. You could just as easily use this to set your command prompt instead of your window title if you were so inclined.

Enjoy, and patches welcome. ;)

• Would I even want the result to be a filesystem? I’ve often said that one of the reasons I got involved with GlusterFS was because I really didn’t want to develop an entire filesystem, and I still feel that way. There’s tremendous value in the result, but if I were to do something on my own maybe I’d go for something less complicated. Object (or “blob”) stores like S3 or Swift are about twenty times easier to implement, and many people seem willing to adopt them instead of filesystems, so maybe I’d do that. With things like HTTP PATCH coming along I’d still have to deal with overlapping byte-aligned writes within a file/object (I’m just going to say “file” from here on for simplicity), but that’s OK.
• Distribution would still be based on consistent hashing. I have many of my own favorite tweaks in this area, most of which are equally applicable to GlusterFS and PBFS.
• The first major change I’d make is to get out of the business of dealing with cluster membership and consensus with respect to configuration myself. There’s other software available to do just that, so there’s no need to implement an inferior home-grown version or put up with its deficiencies.
• The other major change would be a move from a “write in place” model to more of a log-structured or “event sourcing” model. When a client writes data, the entire write is encrypted and erasure coded according to its own internal alignment, regardless of how it’s aligned with respect to the file. The pieces are then distributed among the servers responsible for that file. Voila, no more server-side merging or read-modify-write sequences.
• As pieces of the write’s contents are being written to several servers, information about the write as a whole is fully replicated to a smaller set of servers. Any one of these “catalog” servers can then, when asked about that write, identify which servers got erasure fragments and also provide any authentication/integrity metadata necessary to validate the combined result.
• Since the entire write is maintained as a single unit, that unit can be transmitted verbatim to another site. It can even be erasure-coded differently on that end, to satisfy different requirements e.g. at a primary vs. DR site. This simplifies matters greatly compared to approaches that require writes to be merged somehow before being propagated.
• As with other patch-based systems as they’re well known and understood in the source-control world, the merging happens on the client side during reads. To read a byte range within a file, you ask one of its catalog servers for all current writes overlapping that range. What you get back is potentially a list of writes, with extent and ordering information. It’s up to you to decrypt and merge to get the bytes you need. In some cases, it would make sense to re-encrypt and write back the results so that the next reader only gets a single chunk, but that’s optional. In that case, the write-back might cause some or all of the previous chunks to be marked as no longer current and (eventually) garbage-collected.

As I’m sure you can see, this is a very different kind of animal than GlusterFS – or, for that matter, any other distributed filesystem. Both the read and write paths are fundamentally different. Writes can be significantly more efficient, but reads are more complicated and rely more on high scale to deliver good aggregate performance. It would certainly be an interesting exercise, if only I had time…

Imagine you have a remote view of an ordinary room, with doors at each end. You see a man enter at one end. He crosses to the other end, pausing a couple of times and once stooping to examine something on the floor, then exits at the other end. There seems to be nothing remarkable about this. Then you watch a second man try to cross the room. When he gets half-way across, panels in the walls slide down and the man is skewered with arrows from several hidden crossbows. The screen flickers. A third man tries to cross, and doesn’t even get a quarter of the way across before a green gas seeps from vents near the ceiling and he collapses. The screen flickers again. After you watch a score more of failed attempts to cross the room, the smoke from the latest explosion reveals something: the room is criss-crossed with beams of light, some of which turn on and off, and it’s when people cross these beams that Bad Things happen. You think back to the first man, and realize that he might crossed in the only possible safe way. You briefly consider that he was lucky, but it’s just too implausible. It’s much more likely that he knew exactly what he was doing, that his pauses and stooping were carefully calculated to avoid the various traps. Only then do you realize how skillful his performance was.

Yeah, I know, it’s like those “ninja” metaphors that I usually consider so childish. I was going to try something less colorful, but Amy happened to overhear this little story as I was trying it out on Cindy, and she keeps asking me to tell it again, so maybe being childish isn’t so bad. Anyway, the point is that working on a very complex system (such as I do on GlusterFS) can be a bit like crossing that room. The code contains all sorts of hidden traps for the unwary. I’ve taken my share of arrows in the knee, that’s for sure (and now you know what inspired the metaphor). Usually you have to go through many iterations of being that second or third or tenth person before you have a day where you feel like that first one. You know all of the non-obvious requirements and constraints (traps) between you and what you’re trying to accomplish. Maybe you even relax (disarm) a couple, but mostly you just do the delicate dance around them and get to the point where your new feature works (get out alive). Then somebody watching says, “Pffft, you just walked across a room. What’s hard about that?” That’s why people who have learned – the hard way – how to bend a piece of code in ways it wasn’t originally intended to bend sometimes seem intolerant of dabblers’ or onlookers’ commentary. That’s one of the reasons I stuck with this metaphor: the complexity is hidden. I thought of using a different metaphor of a complex mechanism with many levers and such, but anyone seeing such a mechanism would realize it’s complex. In this case, you could watch the first man cross the room a hundred times and never realize he was engaged in a complex task.

If there’s a moral here, it’s twofold. First, don’t assume something’s easy just because you see someone do it without mishap or obvious effort. They might be applying a great deal of hard-won knowledge to make things go so smoothly. Second, when you’re attempting to do something in a complex environment, expect that you’ll get killed a few times. Respect the dangers, but don’t be afraid of them. You can always reload from your last save. Each time you find a trap, you learn. After enough tries, you might find a way to avoid them all, and the result might even be more satisfying than if things had been easy to begin with.

sequence=0
...
# Test 1
sequence=$((sequence+1))
srcdir=/foo/bar$sequence
sequence=$((sequence+1))
dstdir=/foo/bar$sequence
...
# Test 2
sequence=$((sequence+1))
thedir=/foo/bar$sequence

This works pretty well, but the inline manipulation of $sequence kind of bugged me so I tried to put it in a function. My first try looked something like this.

sequence=0
 
function next_value {
    sequence=$((sequence+1))
    echo $sequence
}
 
thedir=/foo/bar/$(next_value)

Yeah, I hear the laughter. For those who didn’t get the joke yet, this falls prey to bash’s handling of variable scope and subshells. The $(next_value) construct ends up getting executed in a subshell, so changes it makes to variables aren’t reflected in the parent and you end up with the same value every time. I really should have stopped there, satisfying myself with the original inline version. Sure, that version can still hit the scope/subshell issue, but only if you use functions in your own code and not as a side effect of the idiom itself. I realized that getting around the scope/subshell issue would involve something ugly and inefficient, which is why I should have stopped, but I was intrigued. Surely, I thought, there should be a way to do this in an encapsulated and yet robust way. The first idea was to stick the persistent context in a temporary file.

tmpfile=$(my_secure_tmpfile_generator)
 
function next_value {
    prev_value=$(cat $tmpfile)
    next_value=$((prev_value+1))
    echo $next_value > $tmpfile
    echo $next_value
}

OK, it’s kind of icky, but it should work. Again, I should have stopped there, but that temporary file bothered me. Surely I could do that without the file I/O, perhaps by spawning a subprocess and talking to that through a pipe. Yes, folks, I had embarked on a quest to find the most insanely complicated way to solve a pretty simple problem. The result is generator.sh and here’s an example of how to use it.

source generator.sh
start_generator int_generator 5 6
...
dir1=/foo/bar$(next_value 5 6)
dir2=/foo/bar$(next_value 5 6)

Doesn’t look too bad, does it? OK, now go ahead and look at how it’s done. I dare you. Here are some of the funnier bits.

# start_generator
ctop=$(mktemp -t -u fifoXXXXXX)
mkfifo $ctop || fubar=1

Yes, really. Not polluting the filesystem with a temporary file was part of the point here, but I ended up dropping not one but two orts instead. (Cool word, and yes, I did use a thesaurus.) To be fair, these are only visible in the filesystem momentarily before they’re opened and then deleted, but still. I tried to find a way to do this with anonymous pipes, but there just didn’t quite seem to be a way to get bash to do that right. Here’s the next fun bit.

# start_generator
$1 < $ptoc >$ctop &
eval "exec $2> $ptoc"
eval "exec $3< $ctop"

The first line invokes the subprocess, with input and output fifos. The two execs are the bash way to create read and write file descriptors for a file. They’re wrapped in evals to satisfy my goal of making things as complicated as possible by allowing the caller to specify both the generator function/program and the file descriptors to use. Eval is very evil, of course, so let’s play Spot The Security Flaw.

start_generator int_generator "do_something_evil;"
# ...causes us to eval...
exec do_something_evil;> $ptoc

I'm not going to fix this, because it's only an "insider" threat. This code already runs with the same privilege as the caller, and can't do anything the caller can't. They could also pass in a totally bogus generator function, and I'm not going to worry about that either because they'd only be shooting themselves. On to the next fun piece.

# next_value
echo more 1>&$1
read -u $2 x

Again, this is kind of standard bash stuff to write and then read from specific file descriptors. Having an example of this is one of the main reasons I didn't just throw away the script. With a little bit of tweaking, the same technique could be used as the basis for a general form of IPC to/from a subprocess, and that might be useful some day.

To reiterate: this is some of the craziest code I've ever written. It's way more complicated than other solutions that better satisfy any likely set of requirements, and the implementation threads its way through some particularly perilous bash minefields. FFS, I might as well have just used mktemp in the first place and skipped all of this. You'd have to be nuts to solve this problem this way, but maybe my documentation of the discoveries I made along the way will help someone solve a similar problem. Or maybe it's just a funny story about bash scripting gone horribly wrong.

Before I start, I’d like to point out that I know both David and Jonathan. They’re both excellent engineers and excellent people. I also don’t know the context in which David originally made his statement. On the other hand, NoSQL/BigData folks pissing all over things they’re too lazy to understand has been a bit of a hot button for me lately (e.g. see Stop the Hate). So I’m perfectly willing to believe that David’s original statement was well intentioned, perhaps a bit hasty or taken out of context, but I also know that others with far less ability and integrity than he has are likely to take such comments even further out of context and use them in their ongoing “filesystems are irrelevant” marketing campaign. So here’s the conversation so far, rearranged to show the diverging threads of discussion and with some extra commentary from me.

I think that about covers it. As I said, I disagree with the original statement in its general form, but might find myself agreeing with it in a specific context. As I see it, aggregating local filesystems to provide a single storage pool with a filesystem interface and aggregating local filesystems to provide a single storage pool with another interface (such as a column-oriented database) aren’t even different enough to say that one is definitely preferable to the other. The same fundamental issues, and many of the same techniques, apply to both. Saying that filesystems are the wrong way to address scale is like saying that a magnetic #3 Phillips screwdriver is the wrong way to turn a screw. Sometimes it is exactly the right tool, and other times the “right” tool isn’t as different from the “wrong” tool as its makers would have you believe.

However, the most interesting thing about MOOcode is its approach to permissions. Because it was designed to work in a multi-user environment, where users were often neither trusting nor trustworthy but still called each others’ code constantly, MOO needed a pretty robust permissions system. This was no lame private/protected/public model, without even the concept of an owner and thus without the ability to infer rights based on ownership. Each object, each “verb” and each property has an owner. The language includes primitives to determine the object for the previous call (caller), the operative permissions in that call (caller_perms), or even the whole call stack (callers). This lets you implement basically whatever permissions scheme you wanted. For example, “caller==this” is kind of like “protected” in C++, and “caller==#12345″ is kind of like declaring #12345 as a “friend” likewise. At the other extreme, one could go looking further up the stack to see if the current verb is being called in some particular context even if there are multiple “unexpected” calls in between.

The most unusual thing about the MOO permission system is that every verb runs with the permissions of its author – not the user who caused it to be invoked. This is kind of like every program in UNIX being “set UID” by default, which seems crazy but actually works quite well. It makes most kinds of “trojan horse” attacks impossible, for one thing. The person who has to worry about improper access to data is also the person – the verb author/owner – who can add code to prevent it. The exact workings of MOO property ownership and inheritance were a bit strange sometimes, but most MOO programmers learned the basics and were able to secure their code pretty quickly.

Because of all these features, programming in MOOcode was a very fluid and enjoyable experience. Python comes closest among the languages I know well, though I’ve dabbled with Lua and it seems even closer. If I ever decide to spend time on inventing my own language instead of using one to get something else done, it would probably be a prototype-based OOP language with extra features for concurrency/distribution, and in that context MOO’s permission model is about as good a starting point as I’ve seen. It’s too bad that most people who could benefit from studying it are probably put off by its origins in a game-like environment.

Rusty suggests that new languages should be designed with a built-in concept of undefined variables. At the very least, each type should have a value that can not be set, and that only the interpreter/compiler can check. This last part is important, because otherwise people will use it to mean NULL, with all of the previously-mentioned ambiguity that entails. The “uninitialized” value for each type should mean only that – never “ignored” or “doesn’t matter” or anything else. A slightly better approach is to make “uninitialized” only one of many variable annotations that are possible, as in cqual. Maybe some of that functionality will even be baked into gcc or LLVM (small pieces already are), providing the same functionality in current languages. Until then, the best option is to educate people about why it can sometimes be good to leave variables uninitialized until you have a real value for them.

The last couple of days, I had to travel to NYC, so I decided to experiment with using this as my travel computer. I did bring another (small) laptop along just in case, but resolved not to use it – and I didn’t, except as a reserve battery for my MiFi portable wireless gadget. However, I have run into two serious limitations. One is that I can’t use it for presentations. Besides the fact that it physically has only a (mini) HDMI whereas most projectors are still VGA, I can’t find any decent software for presentations. I’ve looked at several apps and a couple of online services. They’re all awful. Is it too much to ask that a presentation program handle a simple two-level bullet list properly? Apparently. The other problem is that I can’t really use this thing for terminal sessions. The base-unit keyboard actually lacks an escape key. As a vi user, that’s crippling. I could use emacs instead, but the handling of the control key also seems a bit erratic. I tried using the on-screen escape key in ConnectBot, but eventually settled on using Hacker’s Keyboard instead. While I was able to get some work done (Gluster guys: that’s how I did the quorum-optional patch) it was certainly not very pleasant. I’d like to avoid solutions that require rooting the device, but I might have to resort to that.

I still love using the EeePad at home, and in meetings. I just might not be able to use it as a road machine and that makes me sad. Maybe the software situation will improve over the next year or so.

So, as usual, please let me know if you see any glitches. One of the things the traffic spike did for me was show that normal stuff is working, but some stuff around the edges might still need tweaking. I know FTP access and image links to womb.atyp.us (in old posts) aren’t working. Anything else?