I’ve been writing “design notes” for my distributed storage system lately. The first one has to do with asynchronous ordered writes (or ordered asynchronous writes, if you prefer) and it’s pretty much done. Turned out to be easier to do than I’d expected, though still hard to explain. My current effort revolves around “secure writes” through untrusted intermediate hosts, by which I mean two things:

  • Unauthorized writes, or writes that have been tampered with, must be rejected.
  • Authorized writes must be performed exactly once, without being dropped or replayed.

Everyone seems to have a pet solution to the first problem. It’s the second one that’s tricky, though, especially if you want to retain the performance advantages of incorporating untrusted hosts into your distributed system in the first place, and it seems to me that nobody’s bothering to deal with it. In most systems, writing seems to be a “best effort” kind of thing: if the write actually happens that’s great, but if it doesn’t then, well, c’est la vie. And no, content-based addressing doesn’t solve the problem.

I think I’ve come up with a solution. It doesn’t involve very much overhead, and in some ways it’s actually quite elegant. However, if any of you regular visitors (I seem to have several dozen nowadays) happen to know of serious alternatives, I’ve love to hear from you. Just send me some email with suggestions (or, even better, actual citations).