(Third in the "Radically Cross Platform" series of posts; see previous posts about Xamarin, Memory Management, and Serialization.)
This is supposed to be about spritesheets, aka texture atlases, which are raster (bitmap, or pixel based) collections of multiple smaller raster images into individual, larger raster images.
Raster, because everything your app presents visually is a raster image. Your video adapter and monitor deal in raster pixels.
What I would rather talk about, however is:
I love vector graphics. This has been the case ever since when I zoomed in on my Windows Paintbrush artwork as a kid and saw the ugliness of pixelation. I knew that didn’t happen with vector based metafiles or TrueType fonts. I appreciated the efficiency of using concise mathematical curves to define a shape that can be rendered on screen.
Since then I’ve enjoyed bumping into vector graphics from time to time in my programming career. This included the DXF format for rendering technical drawings; WMF and EMF for clip art style images; PostScript based vector graphics in PDF files; and in a real sense, much of the graphics drawing code I wrote myself with GDI or Quartz. Platforms like Flash and Silverlight, and GUI libraries like WPF held promise of a bright, purely vector based future of GUI development.
Well, that bright future with vectors may be out there, but for widely cross platform mobile development, my conclusion is that it sure ain’t here yet. The closest you can get for 2D graphical apps is use of 3D models rendered in a flat, 2D projection. And that seems like it would be a lot of work for a questionably good result at best. So I wistfully concluded it wasn’t worth attempting right now.
How GPUs Work
Let’s take a brief look at the services offered by graphics acceleration. Even if you use a high level toolset, it’s useful to know what’s happening under the hood, because you want to work with your GPU instead of against it. While GPUs don’t support everything, what they do is parallel in the extreme, and we have to set things up to take advantage of that parallelism.
The fundamental operation of the GPU is to draw triangles, using texture taken from a raster image. You can change all kinds of things about the triangles, such as the size, rotation, bitmap texture placement, and color tinting.
My game engine is built on the XNA-compatible MonoGame framework and uses the SpriteBatch class, or rather a derivative which I heavily customized and optimized from the MonoGame source. My code (at least most of it) isn’t aware that it’s dealing with triangles; it thinks it’s drawing rectangular bitmaps. But it gets converted to triangles at a lower level.
The lowest level drawing code in my source is a function named "DrawUserIndexedPrimitives" that passes along all my (reformatted) drawing commands as a monolithic array of these heavily annotated triangles. This function is invoked for every rendered frame, with all the triangles required, all posted at once. Thus for 30 FPS, this function would be invoked and the screen redrawn 30 times a second.
For the GPU to "draw a bunch of triangles" in parallel, you absolutely must limit your number of source texture bitmaps. In fact, the GPU’s drawing pipeline is happiest if it has a single texture bitmap for all the rendered triangles. Hence the idea of a spritesheet — a single, raster or bitmap image that puts all the many textures you want to use in rows and columns, or better yet, in an even tighter packing.
It was interesting to see Apple’s recently unveiled Sprite Kit library providing first class support for dealing with efficient, spritesheet based graphics on iOS and OS X. Sprite Kit duplicates much of the functionality of XNA/Monogame, so it wasn’t particularly useful for my project, but it’s another cool tool in the box to be aware of.
Defining My Requirements
For my graphical app engine and designer tool, I wanted:
- Accelerated 2D graphics rendering targeting OpenGL and DirectX.
- Vector graphic source artwork.
- Jointed 2D animation, kind of like with Flash.
- Automatic optimization of graphics for each scene or page.
If you’ve been following the discussion above, you can see that my engine’s design tool needed to automatically generate a spritesheet unique to and optimized for each scene. That’s the only way to really optimize rendering performance via the GPU’s parallelism. In other words, it had to figure out what images are used on each scene and pack all those images together in one spritesheet for the scene.
Even though most of the source for my engine and designer are C#, I wrote a Python script for this purpose, just because this kind of task hits a sweet spot for me with the Python language and the PIL (Python Imaging Library). Here’s a link to a similar script. Mine has a few additions, such as:
- Outputting spritesheets with lengths and widths that are a power of two, as required by some GPUs.
- Performing lossy compression of the PNG output (“lossy PNG” sounds like a contradiction, doesn’t it? But see pngquant and example site TinyPNG.com ).
- Premultiplying alpha, if required by the target platform.
Back to Vector Graphics
I decided to standardize on Inkscape for my source image format. Inkscape, if you’re not familiar with it, is a free and open source vector graphics editor equivalent to Adobe Illustrator, and particularly capitalizes on the SVG file format. It also supports flexible command line export options, which is what I needed to integrate with my app engine’s designer.
My designer walks a directory of my source SVG artwork for an app, and lets me select from the files (and contained layers) when adding a graphical sprite to my title. It intelligently caches both thumbnails and layers exported at specific sizes to speed up the process of generating a preview or optimized spritesheet when required.
Note: One weird quirk that came up during this exporting was that Inkscape’s command line exports by layer ID, which was just a unique numbering not very convenient to use. So my designer has to parse the XML first to correlate the more meaningful layer label with the layer ID before it can export what I want.
The Export Process
So now I had all the parts I needed to generate spritesheets:
- Based on my visual designer, automatically populate a list of vector SVG artwork layers and required sizes.
- Export using Inkscape’s command line into separate PNG files.
- Run my Python script to optimally pack the spritesheet.
The output of the spritesheet builder script consists of two things: a PNG format spritesheet, either premultiplied or not, as required by the target platform; and a metadata file ready for binary serialization as it’s merged into the blob for the overall title in my proprietary packaging format (which would make a decent subject for a separate post).
When my app engine’s runtime is loading a new scene, it needs to query the saved metadata associated with the scene and current language to determine what resources to load. There will be an associated spritesheet image (just one!) and a file describing the contents, i.e. letting you lookup the image rectangle in the spritesheet by name.
It’s Easy! (See Terms and Conditions)
What I described above is basically what my app engine’s designer does, but there are a few extras I implemented:
- Including bitmap font characters as part of the spritesheet. If you display text on the screen that’s not part of your spritesheet, then guess what — the GPU is having to deal with more than one source texture, and it’s not optimal. So I made my designer automatically collect all the dependencies on displayed font styles and used characters (note: only the used characters in each style) to merge into my spritesheet.
- Exporting separate spritesheets by language. Really, language group, because I considered it wasteful to allocate one spritesheet per language. So all the Latin languages like English, Spanish, and French get their resources combined in one spritesheet, and all the Cyrillic varieties, etc. This is the kind of thing you can play with to find a good balance.
- Similarly to the “language groups” concept, exporting by groups of scenes instead of individual scenes, to reduce the app download size requirement. The trick is finding how to to group these so they have enough common graphics that they can feasibly share a spritesheet. Some day I may write some code to run offline to identify the optimal groupings, but I haven’t bothered to yet.
For bitmap fonts I used the excellent and free BMFont utility. BMFont lets you take any TrueType or OpenType font and export specified characters at a specified size and style. My Python spritesheet builder script then harvests required characters from the BMFont output and merges them into my own spritesheet. Of course, it also exports character metrics metadata including kerning tweaks and so forth. I use this in my own text rendering logic, but that’s definitely for another post, as this one’s getting too long.
To be clear, it is a lot of work building the kind of app engine that a game or other graphical app developer would actually want to use. I found it to be a rewarding undertaking, as I enjoyed understanding more about the underlying technology, and bringing together all these different parts to make my own resource pipeline. I’m planning more posts in the future discussing other aspects of this project; feel free to ask about anything you’re curious about in the comments section.