As python programmers we are sometimes faced with using an API that is, well, unpythonic.
Unpythonic? Pythonic? Huh? Have you ever tried running this:
python -m this
Maybe you’re using a C library via ctypes, or you have inherited a collection of functions. Regardless, you find yourself wishing you can use the API in a more idiomatic manner.
This happened to me recently when I started working with redis-py, the defacto standard python interface to redis.
Sidebar: this post is not really about redis, but if you don’t know about it or haven’t given serious thought to using it as a datastore, run right over to http://redis.io/ and read up on this amazing in-memory, key-value store. If you’re familiar with memcached, then you’ll have a general idea of redis, but redis goes well beyond simple key-value pair storage. It also has functionality for storing lists, sets, sorted sets and hashes. Redis is often described as an in-memory data structure server.
In using redis-py, you acquire an instance to a monolithic object that manages a connection pool as well as the underlying redis protocol, offering a method for every redis command. While being feature complete, the object is a bit unwieldy, imho. Typical usage might look like this:
>>> import redis >>> >>> conn = redis.StrictRedis() >>> conn.set('foo', 'bar') >>> print conn.get('foo') bar >>> for c in 'this is a test': ... conn.sadd('myset', c) >>> print conn.smembers('myset') set(['a', ' ', 'e', 'i', 'h', 's', 't'])
Every command for every data structure is attached to an instance of
StrictRedis. If you look at the redis command set you will see there are some commands that apply to all data structures, while others apply to one data structure or another. For example, commands like
ttl can be used against any key in the datastore, while commands
incr, can only be applied to strings, and
hset can only be applied to hashes.
A picture begins to form for the potential for an abstract base class,
Key, with concrete subclasses
We will explore how to create a more pythonic interface for redis, building off the powerful redis-py implementation, using minimal code by using advanced python attribute access and delegation.
A gist has been published which you are free to download and use. The rest of this post will be devoted to explaining the design and implementation of this python module.
As mentioned above, the inherent design of redis suggests an abstract base class with one concrete class for each data structure. This is the class hierarchy:
The underlying redis key
Each of the commands we want to wrap in our knew classes has one thing in common: the first parameter of each method is the redis key to be operated on. We will take advantage of that when we implement the delegation mechanism. We won’t spend any more time here, but note that in
self.key is set (line 158), which will be used as the underlying redis key.
Commands to delegate to redis-py
Each class has a list of redis commands it delegates to the underlying redis-py implementation. For example, RedisBase has this list:
# http://redis.io/commands#generic _redis_methods = ['delete', 'exists', 'expire', 'expireat', 'persist', 'pexpire', 'pexpireat', 'pttl', 'ttl',]
Each subclass adds to the list the methods it implements. For example, for
# http://redis.io/commands#string _redis_methods = RedisBase._redis_methods + ['append', 'bitcount', 'decr', 'decrby', 'get', 'getbit', 'getrange', 'getset', 'incr', 'incrby', 'incrbyfloat', 'psetex', 'set', 'setbit', 'setex', 'setnx', 'setrange', 'strlen']
The magic, getattribute()
def getattribute(self, name): ‘The magic that delegates redis commands to the underlying redis-py methods’
def helper(*args, **kwargs): method = getattr(get_redis_connection(), name) return method(self.key, *args, **kwargs) methods = object.__getattribute__(self, '_redis_methods') if name in methods: return helper return object.__getattribute__(self, name)
getattribute() is called unconditionally for attribute look-up on python class instances (in python 2.X, this method exists only for new style classes, i.e., those that are subclasses of
object). In lines 167-169 we get the list of delegated methods and if the attribute we’re looking up is one of these methods, we return the function
helper (highlighted). If the attribute is not one of the delegated methods, we proceed with normal attribute look-up in line 171.
helper() first finds the method named
name on the global
StrictRedis instance (returned by
get_redis_connection(), not discussed here, but a doc string in lines 79-91 describes the function) then calls the method passing
self.key as the first argument, followed by other parameters and keyword arguments, returning its value.
Scanning through the file you will notice the heart of the implementation (i.e., not counting documentation and the included test suite) is
RedisBase.getattribute() and setting the
_redis_methods attribute for each class. In just a handful of lines of code we have created an OO interface to redis!
Even more pythonic
You will no doubt have noticed that the redis data structures map well to built-in python data types. How about using thepython data model to make our classes even more idiomatic? For example, if we have a redis set:
class MySet(RedisSet): pass mset = MySet(some_key) # use mset, lots of calls to mset.sadd()
then get its size
cardinality = mset.scard()
or check for an item
if mset.sismember(some_item): ...
It would be much nicer to be able to use python idioms:
create mset as above then get its size
cardinality = len(mset)
or check for an item
if some_item in mset.sismember: ...
It is very easy to add this functionality to our classes by implementing
len() on RedisSet:
def <strong>contains</strong>(self, item): return self.sismember(item) def __len__(self): return self.scard()
We can continue to add special methods to make our classes emulate python containers. For example, if we add implementations of
iter() to RedisHash, plus have it subclass collections.MutableMapping, we can use
inst[key] syntax to get and set values in redis and otherwise have them behave fully like python dict objects!
We’ll leave that as an exercise for the reader. Fork the gist and extend it!