As python programmers we are sometimes faced with using an API that is, well, unpythonic.
Unpythonic? Pythonic? Huh? Have you ever tried running this:
python -m this
Maybe you’re using a C library via ctypes, or you have inherited a collection of functions. Regardless, you find yourself wishing you can use the API in a more idiomatic manner.
Sidebar: this post is not really about redis, but if you don’t know about it or haven’t given serious thought to using it as a datastore, run right over to http://redis.io/ and read up on this amazing in-memory, key-value store. If you’re familiar with memcached, then you’ll have a general idea of redis, but redis goes well beyond simple key-value pair storage. It also has functionality for storing lists, sets, sorted sets and hashes. Redis is often described as an in-memory data structure server.
In using redis-py, you acquire an instance to a monolithic object that manages a connection pool as well as the underlying redis protocol, offering a method for every redis command. While being feature complete, the object is a bit unwieldy, imho. Typical usage might look like this:
[code language=”python” gutter=”false”]
>>> import redis
>>> conn = redis.StrictRedis()
>>> conn.set(‘foo’, ‘bar’)
>>> print conn.get(‘foo’)
>>> for c in ‘this is a test’:
… conn.sadd(‘myset’, c)
>>> print conn.smembers(‘myset’)
set([‘a’, ‘ ‘, ‘e’, ‘i’, ‘h’, ‘s’, ‘t’])
Every command for every data structure is attached to an instance ofStrictRedis
. If you look at the redis command set you will see there are some commands that apply to all data structures, while others apply to one data structure or another. For example, commands likeexists
can be used against any key in the datastore, while commandsget
, can only be applied to strings, andhget
can only be applied to hashes.
A picture begins to form for the potential for an abstract base class,Key
, with concrete subclassesString
We will explore how to create a more pythonic interface for redis, building off the powerful redis-py implementation, using minimal code by using advanced python attribute access and delegation.
A gist has been published which you are free to download and use. The rest of this post will be devoted to explaining the design and implementation of this python module.
As mentioned above, the inherent design of redis suggests an abstract base class with one concrete class for each data structure. This is the class hierarchy:
The underlying redis key
Each of the commands we want to wrap in our knew classes has one thing in common: the first parameter of each method is the redis key to be operated on. We will take advantage of that when we implement the delegation mechanism. We won’t spend any more time here, but note that inRedisBase.__init__()
is set (line 158), which will be used as the underlying redis key.
Commands to delegate to redis-py
Each class has a list of redis commands it delegates to the underlying redis-py implementation. For example, RedisBase has this list:
[code language=”python” firstline=”119″]
_redis_methods = [‘delete’, ‘exists’, ‘expire’,
‘expireat’, ‘persist’, ‘pexpire’,
‘pexpireat’, ‘pttl’, ‘ttl’,]
Each subclass adds to the list the methods it implements. For example, for RedisString:
[code language=”python” firstline=”179″]
_redis_methods = RedisBase._redis_methods +
[‘append’, ‘bitcount’, ‘decr’, ‘decrby’, ‘get’,
‘getbit’, ‘getrange’, ‘getset’, ‘incr’, ‘incrby’, ‘incrbyfloat’,
‘psetex’, ‘set’, ‘setbit’, ‘setex’, ‘setnx’, ‘setrange’,
[code language=”python” firstline=”160″ highlight=”163,164,165″]
def __getattribute__(self, name):
‘The magic that delegates redis commands to the underlying redis-py methods’
def helper(*args, **kwargs):
method = getattr(get_redis_connection(), name)
return method(self.key, *args, **kwargs)
methods = object.__getattribute__(self, ‘_redis_methods’)
if name in methods:
return object.__getattribute__(self, name)
is called unconditionally for attribute look-up on python class instances (in python 2.X, this method exists only for new style classes, i.e., those that are subclasses ofobject
). In lines 167-169 we get the list of delegated methods and if the attribute we’re looking up is one of these methods, we return the functionhelper
(highlighted). If the attribute is not one of the delegated methods, we proceed with normal attribute look-up in line 171.
first finds the method namedname
on the globalStrictRedis
instance (returned byget_redis_connection()
, not discussed here, but a doc string in lines 79-91 describes the function) then calls the method passingself.key
as the first argument, followed by other parameters and keyword arguments, returning its value.
Scanning through the file you will notice the heart of the implementation (i.e., not counting documentation and the included test suite) isRedisBase.__getattribute__()
and setting the_redis_methods
attribute for each class. In just a handful of lines of code we have created an OO interface to redis!
Even more pythonic
You will no doubt have noticed that the redis data structures map well to built-in python data types. How about using thepython data model to make our classes even more idiomatic? For example, if we have a redis set:
mset = MySet(some_key)
# use mset, lots of calls to mset.sadd()
# then get its size
cardinality = mset.scard()
# or check for an item
It would be much nicer to be able to use python idioms:
# create mset as above
# then get its size
cardinality = len(mset)
# or check for an item
if some_item in mset.sismember:
It is very easy to add this functionality to our classes by implementing__contains__()
[code language=”python” firstline=”203″]
def __contains__(self, item):
We can continue to add special methods to make our classes emulate python containers. For example, if we add implementations of__getitem__()
to RedisHash, plus have it subclass collections.MutableMapping, we can useinst[key]
syntax to get and set values in redis and otherwise have them behave fully like python dict objects!
We’ll leave that as an exercise for the reader. Fork the gist and extend it!
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.