Asset Pipelines

All games use a variety of data assets that go to make up the final product. On a typical modern console game these will include textures, meshes, shaders and materials, animations, audio samples, level or map files, gameplay scripts, configuration files and more. Some of these assets will be fairly generic and authored using third party tools like Photoshop, 3ds max or Maya. Others will be very game specific and created using custom tools - a custom level editor for example.

An obvious and straightforward approach to handling fairly generic assets like textures is to use a third party tool to save the assets out in some standard file format like a .tga or .png and then to simply load these files using existing third party APIs and libraries. For more game specific assets one option is to define a reasonably flexible file format for the data, perhaps using XML, and then to write code to save the data from a custom tool and load the resulting file in the game. While this approach has some advantages in terms of simplicity, code reuse and flexibility, it also suffers from a number of limitations that mean it is generally not the best approach for a console game:

• Standard file formats for assets like textures are not necessarily designed primarily for fast loading and will not store data in exactly the format needed by the console hardware.
• A file format designed for exporting data from tools will have different and often conflicting requirements from a data format designed for fast loading on a console.
• Assets may benefit from fairly costly pre-processing (compression, vertex cache optimization, lighting precomputation, etc.) for optimal runtime performance while content creators will want quick exports and fast previews for speedy iteration.
• On a multi-platform title, it will be desirable to process assets differently to achieve optimal performance on the different platforms.
• Complex, flexible file formats often require quite a lot of code to load. Ideally it is preferable to keep the code that has to run on the console as small and simple as possible, pushing complexity to offline processing tools that only run on the PC.

In order to address these issues it is common for console titles to have an asset pipeline which introduces some extra stages into the process of getting assets from the tools that create them into the game. At the cost of a little extra complexity there are some big gains in flexibility and game performance. No two studios (or even projects) will have exactly the same asset pipeline but they generally share some common features and I will describe the features of a typical asset pipeline for a console title (many PC titles have asset pipelines as well but they generally require a subset of the functionality of a console pipeline).

Asset pipelines generally introduce at least one extra stage to the simple process described above of exporting a file from a tool and loading that same file in the game. Instead assets will be exported from the tools to a fairly 'fat' intermediate file format containing any information that is likely to ever be needed by the game, along with additional metadata describing the asset and perhaps other information like edit histories, comments, etc. This format should typically be designed to be flexible, versionable (ideally with forward and backward compatibility), debuggable (perhaps a human readable text format) and convenient for tools to work with. Efficiency in terms of storage space and parsing times are secondary concerns for this format.

The second stage of the asset pipeline is a batch build process to transform these fat source assets into the final format for loading in the game. For a multi platform title the build process will have multiple build targets and produce different final data for each platform. The final data format is designed primarily for small size, efficient loading and maximum runtime performance. There will often be an additional build step that bundles together multiple final assets into a package representing a level or chunk of a streaming environment.

The asset build process is analogous to the process of building an application from a collection of text C++ source files - the IDE plays the role of the content creation tool, the source code represents the flexible intermediate file format, the compiler is the batch build step that converts the intermediate format into an efficient platform specific binary format and the linker plays the same role as the packaging or bundling step, combining multiple binary assets into packages (.dlls or .exes). In fact I've worked with a pipeline that used the ELF file format (used for executables and libraries on many Unix systems) as the final asset format - I'll talk more about what's needed for efficient loading of a final data format and why ELF is a suitable choice in a future post. The similarity to code builds also means that tools commonly used for building code like Make, Ant/NAnt and Scons form an integral part of many asset pipelines. The large volumes of data and sometimes very expensive pre-processing steps mean that there are challenges unique to game asset pipelines that often require custom build tools as well.

Many asset pipelines will consist of a combination of build scripts, Python or Perl scripts and stand alone command line 'compilers' for particular asset types that perform different processing and optimization steps to produce final assets. An advantage of this kind of approach is that it is relatively easy to plug third party tools into the pipeline that perform tasks like texture compression, mip map filtering, mesh vertex cache optimization, shader compilation, etc. before feeding the results on to tools that convert data into the final game specific formats. An alternative approach to a collection of command line tools and scripts is to build an asset processing framework consisting of a main driving process and then custom plugins for performing particular processing steps. This alternative approach can be more efficient due to keeping assets in memory throughout the whole build process but may require more up front infrastructure development and may prove difficult to integrate with third party tools.

Whichever approach is used it is a good idea to have a build system that can take advantage of multiple machines to distribute the build. Asset build times can become a bottleneck for the whole team so it is important that the build process can be scaled by adding extra hardware if necessary. Another important consideration related to build times is how content creators can preview their assets on target - an important factor for iteration times. Some projects may have builds that are quick enough for artists and designers to run an incremental content build locally and quickly preview their work. If this process is slow though it may be a good idea to have a special preview build process that skips some of the expensive pre-processing to provide a reasonably accurate but less efficient asset for previewing. An alternative is to set up a build farm that artists can submit preview builds to - this is only really worthwhile if the turnaround is sufficiently fast.

A nice benefit of having a good asset pipeline in place is that it becomes relatively straightforward to make changes or optimizations to the final data format and do an automatic batch rebuild of all the assets. With a simple one stage pipeline (export the data, load the same file in the game) such changes, if they are possible at all, will generally require a batch re-export of all the assets in the game. A batch re-export will require tools that can be scripted to run in batch mode and a way to automatically get a list of all the source files for re-export, information that is available already in the build scripts with a proper asset pipeline.

I've described the basics of an asset pipeline but there are many interesting ways to extend the idea of asset pipelines in more sophisticated directions. One interesting avenue to pursue is to maintain an asset database that describes all the individual assets in the game and their relationships and dependencies. This can be used as the canonical description of what content is going into the game and so not only drive the asset builds but also easily answer questions like 'what game entities will be affected if I modify this texture/shader/material definition?', 'which textures are referenced in level 3?' or 'which assets are required for a demo featuring levels 1, 2 and 3?'. These kinds of questions have traditionally been difficult to answer for many games. Even the question of 'which assets are unreferenced?' is not always easy to answer with a traditional pipeline, often leading to wasted space on the disk.

There are also plenty of engineering challenges in developing a fully robust and scalable pipeline. Efficient incremental builds may require a system for efficiently checksumming files. A robust distributed build system is quite challenging to create. Catching content errors and delivering helpful and specific error messages requires careful engineering and is key to getting good productivity. Hopefully this post has presented a clear picture of the key issues involved in getting a basic asset pipeline in place, there's plenty more to be said on building a really solid asset pipeline however. In a future post I will talk more specifically about what the final data format should look like for efficient loading on a console.