Making An Inherited API Pythonic

Making redis pythonic

Making redis pythonic.

As python programmers we are sometimes faced with using an API that is, well, unpythonic.

Unpythonic? Pythonic? Huh? Have you ever tried running this:

    python -m this

Maybe you’re using a C library via ctypes, or you have inherited a collection of functions. Regardless, you find yourself wishing you can use the API in a more idiomatic manner.

This happened to me recently when I started working with redis-py, the defacto standard python interface to redis.

Sidebar: this post is not really about redis, but if you don’t know about it or haven’t given serious thought to using it as a datastore, run right over to http://redis.io/ and read up on this amazing in-memory, key-value store. If you’re familiar with memcached, then you’ll have a general idea of redis, but redis goes well beyond simple key-value pair storage. It also has functionality for storing lists, sets, sorted sets and hashes. Redis is often described as an in-memory data structure server.

In using redis-py, you acquire an instance to a monolithic object that manages a connection pool as well as the underlying redis protocol, offering a method for every redis command. While being feature complete, the object is a bit unwieldy, imho. Typical usage might look like this:

[code language=”python” gutter=”false”]
>>> import redis
>>>
>>> conn = redis.StrictRedis()
>>> conn.set(‘foo’, ‘bar’)
>>> print conn.get(‘foo’)
bar
>>> for c in ‘this is a test’:
… conn.sadd(‘myset’, c)
>>> print conn.smembers(‘myset’)
set([‘a’, ‘ ‘, ‘e’, ‘i’, ‘h’, ‘s’, ‘t’])
[/code]

Every command for every data structure is attached to an instance of

StrictRedis

. If you look at the redis command set you will see there are some commands that apply to all data structures, while others apply to one data structure or another. For example, commands like

exists

,

expire

, and

ttl

can be used against any key in the datastore, while commands

get

,

set

, and

incr

, can only be applied to strings, and

hget

,

hkeys

, and

hset

can only be applied to hashes.

A picture begins to form for the potential for an abstract base class,

Key

, with concrete subclasses

String

,

List

,

Set

,

SortedSet

, and

Hash

.

We will explore how to create a more pythonic interface for redis, building off the powerful redis-py implementation, using minimal code by using advanced python attribute access and delegation.

A gist has been published which you are free to download and use. The rest of this post will be devoted to explaining the design and implementation of this python module.

As mentioned above, the inherent design of redis suggests an abstract base class with one concrete class for each data structure. This is the class hierarchy:

RedisBase
+– RedisString
+– RedisSet
+– RedisSortedSet
+– RedisList
+– RedisHash

The underlying redis key

Each of the commands we want to wrap in our knew classes has one thing in common: the first parameter of each method is the redis key to be operated on. We will take advantage of that when we implement the delegation mechanism. We won’t spend any more time here, but note that in

RedisBase.__init__()

,

self.key

is set (line 158), which will be used as the underlying redis key.

Commands to delegate to redis-py

Each class has a list of redis commands it delegates to the underlying redis-py implementation. For example, RedisBase has this list:

[code language=”python” firstline=”119″]
# http://redis.io/commands#generic
_redis_methods = [‘delete’, ‘exists’, ‘expire’,
‘expireat’, ‘persist’, ‘pexpire’,
‘pexpireat’, ‘pttl’, ‘ttl’,]
[/code]

Each subclass adds to the list the methods it implements. For example, for RedisString:

[code language=”python” firstline=”179″]
# http://redis.io/commands#string
_redis_methods = RedisBase._redis_methods +
[‘append’, ‘bitcount’, ‘decr’, ‘decrby’, ‘get’,
‘getbit’, ‘getrange’, ‘getset’, ‘incr’, ‘incrby’, ‘incrbyfloat’,
‘psetex’, ‘set’, ‘setbit’, ‘setex’, ‘setnx’, ‘setrange’,
‘strlen’]
[/code]

The magic,

__getattribute__()

[code language=”python” firstline=”160″ highlight=”163,164,165″]
def __getattribute__(self, name):
‘The magic that delegates redis commands to the underlying redis-py methods’

def helper(*args, **kwargs):
method = getattr(get_redis_connection(), name)
return method(self.key, *args, **kwargs)

methods = object.__getattribute__(self, ‘_redis_methods’)
if name in methods:
return helper

return object.__getattribute__(self, name)
[/code]

The method

__getattribute__()

is called unconditionally for attribute look-up on python class instances (in python 2.X, this method exists only for new style classes, i.e., those that are subclasses of

object

). In lines 167-169 we get the list of delegated methods and if the attribute we’re looking up is one of these methods, we return the function

helper

(highlighted). If the attribute is not one of the delegated methods, we proceed with normal attribute look-up in line 171.

The function

helper()

first finds the method named

name

on the global

StrictRedis

instance (returned by

get_redis_connection()

, not discussed here, but a doc string in lines 79-91 describes the function) then calls the method passing

self.key

as the first argument, followed by other parameters and keyword arguments, returning its value.

Scanning through the file you will notice the heart of the implementation (i.e., not counting documentation and the included test suite) is

RedisBase.__getattribute__()

and setting the

_redis_methods

attribute for each class. In just a handful of lines of code we have created an OO interface to redis!

Even more pythonic

You will no doubt have noticed that the redis data structures map well to built-in python data types. How about using thepython data model to make our classes even more idiomatic? For example, if we have a redis set:

[code language=”python”]

class MySet(RedisSet):
pass

mset = MySet(some_key)
# use mset, lots of calls to mset.sadd()

# then get its size
cardinality = mset.scard()

# or check for an item
if mset.sismember(some_item):

[/code]

It would be much nicer to be able to use python idioms:

[code language=”python”]

# create mset as above

# then get its size
cardinality = len(mset)

# or check for an item
if some_item in mset.sismember:

[/code]

It is very easy to add this functionality to our classes by implementing

__contains__()

and

__len__()

on RedisSet:

[code language=”python” firstline=”203″]
def __contains__(self, item):
return self.sismember(item)

def __len__(self):
return self.scard()
[/code]

We can continue to add special methods to make our classes emulate python containers. For example, if we add implementations of

__getitem__()

,

__setitem__()

,

__delitem__()

,

__len__()

, and

__iter__()

to RedisHash, plus have it subclass collections.MutableMapping, we can use

inst[key]

syntax to get and set values in redis and otherwise have them behave fully like python dict objects!

We’ll leave that as an exercise for the reader. Fork the gist and extend it!

Daniel Popowich

Daniel Popowich

Daniel is a Sr Software Engineer for Art & Logic.
Daniel Popowich

Latest posts by Daniel Popowich (see all)

Tags:

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.