As python programmers we are sometimes faced with using an API that is, well, unpythonic.
Unpythonic? Pythonic? Huh? Have you ever tried running this:
python -m this
Maybe you’re using a C library via ctypes, or you have inherited a collection of functions. Regardless, you find yourself wishing you can use the API in a more idiomatic manner.
This happened to me recently when I started working with redis-py, the defacto standard python interface to redis.
Sidebar: this post is not really about redis, but if you don’t know about it or haven’t given serious thought to using it as a datastore, run right over to http://redis.io/ and read up on this amazing in-memory, key-value store. If you’re familiar with memcached, then you’ll have a general idea of redis, but redis goes well beyond simple key-value pair storage. It also has functionality for storing lists, sets, sorted sets and hashes. Redis is often described as an in-memory data structure server.
In using redis-py, you acquire an instance to a monolithic object that manages a connection pool as well as the underlying redis protocol, offering a method for every redis command. While being feature complete, the object is a bit unwieldy, imho. Typical usage might look like this:
>>> import redis
>>>
>>> conn = redis.StrictRedis()
>>> conn.set('foo', 'bar')
>>> print conn.get('foo')
bar
>>> for c in 'this is a test':
... conn.sadd('myset', c)
>>> print conn.smembers('myset')
set(['a', ' ', 'e', 'i', 'h', 's', 't'])
Every command for every data structure is attached to an instance of StrictRedis. If you look at the redis command set you will see there are some commands that apply to all data structures, while others apply to one data structure or another. For example, commands like exists, expire, and ttl can be used against any key in the datastore, while commands get, set, and incr, can only be applied to strings, and hget, hkeys, and hset can only be applied to hashes.
A picture begins to form for the potential for an abstract base class, Key, with concrete subclasses String, List, Set, SortedSet, and Hash.
We will explore how to create a more pythonic interface for redis, building off the powerful redis-py implementation, using minimal code by using advanced python attribute access and delegation.
A gist has been published which you are free to download and use. The rest of this post will be devoted to explaining the design and implementation of this python module.
As mentioned above, the inherent design of redis suggests an abstract base class with one concrete class for each data structure. This is the class hierarchy:
RedisBase
+– RedisString
+– RedisSet
+– RedisSortedSet
+– RedisList
+– RedisHash
The underlying redis key
Each of the commands we want to wrap in our knew classes has one thing in common: the first parameter of each method is the redis key to be operated on. We will take advantage of that when we implement the delegation mechanism. We won’t spend any more time here, but note that in RedisBase.init(), self.key is set (line 158), which will be used as the underlying redis key.
Commands to delegate to redis-py
Each class has a list of redis commands it delegates to the underlying redis-py implementation. For example, RedisBase has this list:
# http://redis.io/commands#generic
_redis_methods = ['delete', 'exists', 'expire',
'expireat', 'persist', 'pexpire',
'pexpireat', 'pttl', 'ttl',]
Each subclass adds to the list the methods it implements. For example, for RedisString:
# http://redis.io/commands#string
_redis_methods = RedisBase._redis_methods +
['append', 'bitcount', 'decr', 'decrby', 'get',
'getbit', 'getrange', 'getset', 'incr', 'incrby', 'incrbyfloat',
'psetex', 'set', 'setbit', 'setex', 'setnx', 'setrange',
'strlen']
The magic, getattribute()
def getattribute(self, name): ‘The magic that delegates redis commands to the underlying redis-py methods’
def helper(*args, **kwargs):
method = getattr(get_redis_connection(), name)
return method(self.key, *args, **kwargs)
methods = object.__getattribute__(self, '_redis_methods')
if name in methods:
return helper
return object.__getattribute__(self, name)
The method getattribute() is called unconditionally for attribute look-up on python class instances (in python 2.X, this method exists only for new style classes, i.e., those that are subclasses of object). In lines 167-169 we get the list of delegated methods and if the attribute we’re looking up is one of these methods, we return the function helper (highlighted). If the attribute is not one of the delegated methods, we proceed with normal attribute look-up in line 171.
The function helper() first finds the method named name on the global StrictRedis instance (returned by get_redis_connection(), not discussed here, but a doc string in lines 79-91 describes the function) then calls the method passing self.key as the first argument, followed by other parameters and keyword arguments, returning its value.
Scanning through the file you will notice the heart of the implementation (i.e., not counting documentation and the included test suite) is RedisBase.getattribute() and setting the _redis_methods attribute for each class. In just a handful of lines of code we have created an OO interface to redis!
Even more pythonic
You will no doubt have noticed that the redis data structures map well to built-in python data types. How about using thepython data model to make our classes even more idiomatic? For example, if we have a redis set:
class MySet(RedisSet):
pass
mset = MySet(some_key)
# use mset, lots of calls to mset.sadd()
then get its size
cardinality = mset.scard()
or check for an item
if mset.sismember(some_item):
...
It would be much nicer to be able to use python idioms:
create mset as above then get its size
cardinality = len(mset)
or check for an item
if some_item in mset.sismember:
...
It is very easy to add this functionality to our classes by implementing contains() and len() on RedisSet:
def <strong>contains</strong>(self, item):
return self.sismember(item)
def __len__(self):
return self.scard()
We can continue to add special methods to make our classes emulate python containers. For example, if we add implementations of getitem(), setitem(), delitem(), len(), and iter() to RedisHash, plus have it subclass collections.MutableMapping, we can use inst[key] syntax to get and set values in redis and otherwise have them behave fully like python dict objects!
We’ll leave that as an exercise for the reader. Fork the gist and extend it!
