I’ve written before about using Amazon’s S3 data-storage service, and even developed a minimal website backup tool for it. One of my complaints, though, has been the absolute cruddiness of the various libraries one might use to interface with it. I’m pleased to say that Boto seems to have fixed that. Boto is a Python library that provides a quite-reasonable interface for S3. For example, here’s how I get a list of my buckets.

$ python
Python 2.4.1 (#1, Oct 13 2006, 16:51:58)
[GCC 4.0.2 20050901 (prerelease) (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto
>>> conn = boto.connect_s3('my_public_key','my_private_key')
>>> blist = conn.get_all_buckets()
>>> blist
[<bucket : s3dav_2>, <bucket : wombatypus>]

The interface for listing a bucket and finding keys that match a name or pattern is not quite ideal, but still quite usable.

>>> w = blist[1]
>>> klist = w.list()
>>> for s3file in klist:
...   if s3file.name == 'flume1.jpg':
...     break
...
>>> s3file
<Key: wombatypus,flume1.jpg>

Lastly, given the key object, I can fetch the contents into a local file with one line.

>>> s3file.get_contents_to_file('myflume5.jpg')

Very nice. Since I’m often harshly critical of other people’s code, I like taking the opportunity to praise some as well, so . . . good job, Boto folks!. The interfaces for dealing with ACLs look reasonable, though I haven’t actually played with them. Specifying MIME types looks a bit ickier (apparently you have to construct the appropriate HTTP headers yourself and pass them to send_file or set_contents_from_file* as such instead of having an actual type argument to those functions) but survivable. With little bit of my own extra library glop on top of Boto, it looks like I might be able to write a decent set of tools to manipulate my S3 files. With that plus FusePython, I might be able to prototype my very own S3-based filesystem. Actually I should blog about that soon, since it’s based on a very different approach than the other S3-filesystem attempts I’ve seen out there (none of which have turned out very well IMO).

This entry was posted on Wednesday, July 9th, 2008 at 9:06 am and is filed under tech. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.