1991-2016—25 years of Art & Logic

Automated Testing in the Real World

Automated Testing in the Real World

Disclaimer: There are no rules, only guidelines. Every project and situation has its own unique needs.

Actually, that’s exactly what this article is about.

* * *

Why do we always talk about unit tests? Why is “unit tests” automatically added to the plan of any or all projects? I think it’s just because it’s an expression we’ve gotten used to hearing. There are an awful lot of “xUnit” packages nowadays.

But what is a unit test, anyway?


Factories, not Fixtures

Rosie the RiveterFor years, the most common way to provide test data for automated tests has been fixtures – hard-coded values, usually stored in text files.  For example, here’s some YAML-formatted data for a city model object:

- model: city
    id: 1
    name: Los Angeles

In an environment like a Django app, this fixture would typically be loaded into a test database, and then accessed like: la = City.objects.get(id=1).

But fixtures and the frameworks that rely on them have several drawbacks:

Their data is brittle, especially when including references like unique IDs.  Changing or adding a property later may break tests.  This drawback also means that they are not easily modifiable, which tends to lead to duplicate fixtures with unwieldy names like los-angeles-with-an-extra-long-name.yaml.

They are typically loaded en masse by test frameworks like Django’s.  This can be slow if many unnecessary fixtures are being loaded for each unit test.  It also creates brittle sets of data.  For example, if an automated test is searching for objects with a matching city name and expects to find one instance, but later a new fixture is added that also matches, the test will fail.

Because fixtures are typically automatically loaded into a database by the test framework, it’s not particularly easy or fast to change the properties of an object for a single test case, which also tends to lead to an over-abundance of fixture files.

Factories, not fixtures

Test data factories solve these problems by making data and data loading more dynamic and programmable.  A factory is written in code, and while it can simply mimic a fixture by supplying default values for an object, factory libraries offer many other useful features.  Here’s a simple example factory:

Factory.define('city', City)
  .attr('name', 'Los Angeles')

An instance of this factory could be created on the fly like: la = Factory.build('city').

Following the builder pattern, a factory generates data for all the attributes in its definition, constructs any necessary associated objects, and allows a test case to override these values.  Here’s a slightly more complex example:

Factory.define('city', City)
  .attr('name', 'Los Angeles')
  // Define 'ref' as dependent on the id and name properties
  .attr('ref', ['id', 'name'], function(id, name) {
    return id + '-' + name;

nyc = Factory.build('city', {name: 'NYC'})

Some typical features in factory libraries are:

  • integration with common ORMs; Factory.create(...) will typically build and save the object to a database
  • factory inheritance, allowing similar factories to share properties; e.g. Factory.define('city').extend('Olympic').attr('year', null)
  • lazy attributes; e.g. .attr('created_at', function() { return new Date(); })
  • associations to other factories

Factories across languages

Factory libraries have been springing up over the past handful of years.  Ruby’s factory_girl, which has been cloned to many other languages, was first released in 2008.  Several new ones for JavaScript and Objective-C have just appeared this year.

Here’s a list of factory libraries for a variety of common languages:

Test data for unit tests

A note of caution: a one line factory invocation may hide a great deal of complexity and database integration.  That may be fine for integration tests, but should be avoided for unit tests (see the blog post Factories breed complexity for a lengthier discussion).  Prefer to use simpler, non-persisted objects in unit tests.  Factory libraries may help here too by returning just the attributes as a hash or dictionary; e.g. factory_girl’s attributes_for method.

gulp Minus Plugin

photo by mattmariebache, https://www.flickr.com/photos/mattmariebache/4593569358

photo by mattmariebache, https://www.flickr.com/photos/mattmariebache/4593569358

I covered the potential simplicity of gulp plugins in my previous post (refresher: gulp, the hot new JavaScript build system, enables writing an asynchronous, streaming plugin in just a couple dozen lines).  On the other hand, gulp’s philosophy leads to a pretty lengthy list of strong recommendations for plugins, including: don’t write a plugin if it can be reasonably avoided.

Whereas the configuration-centric Gruntfile seems to require writing a plugin for any and every tool, the code-centric gulpfile makes it easy to call a tool directly from a task.  There are two major benefits to skipping the plugin:

  1. Plugins should be simple, but simplicity conflicts with supporting end users’ various needs
  2. Plugins hide tool interactions, and non-standard usage may involve time digging into their internals (more…)

Two Things I Missed Functional Testing with WebTest

Ian Bicking’s WebTest is a helpful module for running functional tests against a WSGI app. You may already be using it, it’s the suggested way to test Pyramid apps, TurboGears and Google App Engine, and (although I have no experience with this) you can use it with Django (apparently this is a good idea) and Flask. It’s not very complicated but in my haste to get things done I overlooked a couple of its nicest features.