Game Programming Lore

Console Technical Information Links

mattnewport — Tue, 25 Mar 2008 07:04:00 +0000

It's hard to find detailed technical information on consoles as much of it is covered by NDAs. There is some information in the public domain though and this collection of links contains a fair amount of information on the Xbox 360 and PS3.

Xbox 360

XNA Presentations on MSDN

Inside the Xbox 360 Part 1 and Part 2 on Ars Technica

Beyond3D - ATI Xenos: Xbox 360 Graphics Demystified

PS3

Introducing the IBM/Sony/Toshiba Cell Processor Part 1 and Part 2 on Ars Technica

CellPerformance.com

IBM Cell Broadband Engine Resource Center

Asset Pipelines

mattnewport — Tue, 18 Mar 2008 06:55:00 +0000

All games use a variety of data assets that go to make up the final product. On a typical modern console game these will include textures, meshes, shaders and materials, animations, audio samples, level or map files, gameplay scripts, configuration files and more. Some of these assets will be fairly generic and authored using third party tools like Photoshop, 3ds max or Maya. Others will be very game specific and created using custom tools - a custom level editor for example.

An obvious and straightforward approach to handling fairly generic assets like textures is to use a third party tool to save the assets out in some standard file format like a .tga or .png and then to simply load these files using existing third party APIs and libraries. For more game specific assets one option is to define a reasonably flexible file format for the data, perhaps using XML, and then to write code to save the data from a custom tool and load the resulting file in the game. While this approach has some advantages in terms of simplicity, code reuse and flexibility, it also suffers from a number of limitations that mean it is generally not the best approach for a console game:

Standard file formats for assets like textures are not necessarily designed primarily for fast loading and will not store data in exactly the format needed by the console hardware.
A file format designed for exporting data from tools will have different and often conflicting requirements from a data format designed for fast loading on a console.
Assets may benefit from fairly costly pre-processing (compression, vertex cache optimization, lighting precomputation, etc.) for optimal runtime performance while content creators will want quick exports and fast previews for speedy iteration.
On a multi-platform title, it will be desirable to process assets differently to achieve optimal performance on the different platforms.
Complex, flexible file formats often require quite a lot of code to load. Ideally it is preferable to keep the code that has to run on the console as small and simple as possible, pushing complexity to offline processing tools that only run on the PC.

In order to address these issues it is common for console titles to have an asset pipeline which introduces some extra stages into the process of getting assets from the tools that create them into the game. At the cost of a little extra complexity there are some big gains in flexibility and game performance. No two studios (or even projects) will have exactly the same asset pipeline but they generally share some common features and I will describe the features of a typical asset pipeline for a console title (many PC titles have asset pipelines as well but they generally require a subset of the functionality of a console pipeline).

Asset pipelines generally introduce at least one extra stage to the simple process described above of exporting a file from a tool and loading that same file in the game. Instead assets will be exported from the tools to a fairly 'fat' intermediate file format containing any information that is likely to ever be needed by the game, along with additional metadata describing the asset and perhaps other information like edit histories, comments, etc. This format should typically be designed to be flexible, versionable (ideally with forward and backward compatibility), debuggable (perhaps a human readable text format) and convenient for tools to work with. Efficiency in terms of storage space and parsing times are secondary concerns for this format.

The second stage of the asset pipeline is a batch build process to transform these fat source assets into the final format for loading in the game. For a multi platform title the build process will have multiple build targets and produce different final data for each platform. The final data format is designed primarily for small size, efficient loading and maximum runtime performance. There will often be an additional build step that bundles together multiple final assets into a package representing a level or chunk of a streaming environment.

The asset build process is analogous to the process of building an application from a collection of text C++ source files - the IDE plays the role of the content creation tool, the source code represents the flexible intermediate file format, the compiler is the batch build step that converts the intermediate format into an efficient platform specific binary format and the linker plays the same role as the packaging or bundling step, combining multiple binary assets into packages (.dlls or .exes). In fact I've worked with a pipeline that used the ELF file format (used for executables and libraries on many Unix systems) as the final asset format - I'll talk more about what's needed for efficient loading of a final data format and why ELF is a suitable choice in a future post. The similarity to code builds also means that tools commonly used for building code like Make, Ant/NAnt and Scons form an integral part of many asset pipelines. The large volumes of data and sometimes very expensive pre-processing steps mean that there are challenges unique to game asset pipelines that often require custom build tools as well.

Many asset pipelines will consist of a combination of build scripts, Python or Perl scripts and stand alone command line 'compilers' for particular asset types that perform different processing and optimization steps to produce final assets. An advantage of this kind of approach is that it is relatively easy to plug third party tools into the pipeline that perform tasks like texture compression, mip map filtering, mesh vertex cache optimization, shader compilation, etc. before feeding the results on to tools that convert data into the final game specific formats. An alternative approach to a collection of command line tools and scripts is to build an asset processing framework consisting of a main driving process and then custom plugins for performing particular processing steps. This alternative approach can be more efficient due to keeping assets in memory throughout the whole build process but may require more up front infrastructure development and may prove difficult to integrate with third party tools.

Whichever approach is used it is a good idea to have a build system that can take advantage of multiple machines to distribute the build. Asset build times can become a bottleneck for the whole team so it is important that the build process can be scaled by adding extra hardware if necessary. Another important consideration related to build times is how content creators can preview their assets on target - an important factor for iteration times. Some projects may have builds that are quick enough for artists and designers to run an incremental content build locally and quickly preview their work. If this process is slow though it may be a good idea to have a special preview build process that skips some of the expensive pre-processing to provide a reasonably accurate but less efficient asset for previewing. An alternative is to set up a build farm that artists can submit preview builds to - this is only really worthwhile if the turnaround is sufficiently fast.

A nice benefit of having a good asset pipeline in place is that it becomes relatively straightforward to make changes or optimizations to the final data format and do an automatic batch rebuild of all the assets. With a simple one stage pipeline (export the data, load the same file in the game) such changes, if they are possible at all, will generally require a batch re-export of all the assets in the game. A batch re-export will require tools that can be scripted to run in batch mode and a way to automatically get a list of all the source files for re-export, information that is available already in the build scripts with a proper asset pipeline.

I've described the basics of an asset pipeline but there are many interesting ways to extend the idea of asset pipelines in more sophisticated directions. One interesting avenue to pursue is to maintain an asset database that describes all the individual assets in the game and their relationships and dependencies. This can be used as the canonical description of what content is going into the game and so not only drive the asset builds but also easily answer questions like 'what game entities will be affected if I modify this texture/shader/material definition?', 'which textures are referenced in level 3?' or 'which assets are required for a demo featuring levels 1, 2 and 3?'. These kinds of questions have traditionally been difficult to answer for many games. Even the question of 'which assets are unreferenced?' is not always easy to answer with a traditional pipeline, often leading to wasted space on the disk.

There are also plenty of engineering challenges in developing a fully robust and scalable pipeline. Efficient incremental builds may require a system for efficiently checksumming files. A robust distributed build system is quite challenging to create. Catching content errors and delivering helpful and specific error messages requires careful engineering and is key to getting good productivity. Hopefully this post has presented a clear picture of the key issues involved in getting a basic asset pipeline in place, there's plenty more to be said on building a really solid asset pipeline however. In a future post I will talk more specifically about what the final data format should look like for efficient loading on a console.

Book Recommendations, General Programming

mattnewport — Mon, 17 Mar 2008 09:23:00 +0000

In my previous post I recommended some books on C++. I've also found some general programming books particularly useful and this post covers a few of my top picks. These books are all applicable to programming in any discipline, not just to games.

The Pragmatic Programmer

The Pragmatic Programmer: From Journeyman to Master by Andrew Hunt and David Thomas is all about the craft of programming and is full of advice about how effective and productive programmers approach the task of developing software. There's lots of invaluable tips here on how to be a more effective and productive developer.

Design Patterns

Design Patterns by 'The Gang of Four' introduced the idea of design patterns to a wide audience of developers and provides the definitive definitions of many of the most common design patterns. While some valid criticisims have been made of the idea of design patterns over the years, and the singleton design pattern presented in the book has been singled out for particular (and justified) criticism, it is still important to be aware of the major patterns and this book is still the best reference for them in my opinion.

Working Effectively With Legacy Code

Working Effectively With Legacy Code by Michael Feathers is full of valuable advice on an activity that most professional developers (including game developers) will spend a lot of time doing during their careers. While many developers would like nothing more than to start from scratch when starting a new project, that's not a very effective or realistic strategy in practice and working effectively with existing code bases is an important part of professional development.

In the games industry, developers will face many of the common challenges of working with legacy code - poor or non-existent documentation, code that has evolved over time with a lack of strong design direction, code that while fit for it's orginal purpose has over time been pushed to breaking point as it has had to meet new and unanticipated requirements and many other causes of developer headaches. Reading this book will put you in a better position when having to deal with these issues and more.

Book Recommendations, C++

mattnewport — Mon, 17 Mar 2008 08:25:00 +0000

The vast majority of commercial games developed today are written primarily in C++. While many games will use a scripting language for some game logic and other languages like Python and C# are often used for tools and in build and asset pipelines, the bulk of engine and game code is usually developed in C++. This means that an in depth knowledge of C++ is essential for any programmer wanting to work in the games industry. There are many good books on C++ on the market but this post lists the books I think are most useful for a game programmer wanting to improve their C++ knowledge.

Effective C++ and More Effective C++

Effective C++ and More Effective C++ by Scott Meyers are common fixtures in C++ book recommendations and with good reason. C++ has many gotchas to trip up the unwary and these two books highlight most of them and explain in a clear, concise and readable manner what the best practice is. If you want to avoid learning the hard way what some of the most common C++ mistakes are then you should read these two books.

C++ Common Knowledge

C++ Common Knowledge by Stephen Dewhurst covers similar material to the Effective C++ books but is worth reading in addition to those books because it offers an alternative presentation of some of the issues and some useful new advice.

C++ Coding Standards

C++ Coding Standards by Herb Sutter and Andrei Alexandrescu lays out a large number of recommendations for best practices when developing software using C++. It's a great basis for a coding standard for any project or team and even if your project already has a coding standard it contains enough useful information to be worth reading for any C++ developer. I have two copies of this so I always have a copy in reach at work and at home.

Effective STL

Effective STL is another addition to the Effective series by Scott Meyers and covers pitfalls and best practices when using the STL. The use of STL in games is still a matter of some debate in the industry and many teams, particularly those targeting consoles, do not allow the use of the STL in game code. Personally I think the STL has a place in game development provided programmers understand it and avoid common pitfalls. Much of the bad press the STL gets comes from bad usage by programmers who don't understand it very well. Read this book and avoid being one of those programmers who gives the STL a bad name.

The Design and Evolution of C++

In The Design and Evolution of C++ Bjarne Stroustrup, the creator of C++, talks about the history of the language and the thought processes and reasoning behind many of the features and design decisions that shaped the language we know today. If you've ever wondered why a particular aspect of C++ works the way it does, or wished part of the language was different, you'll probably find a well thought out explanation and justification here. Even design decisions that in retrospect seem like they might have been flawed usually had good reasons when they were made. Understanding the history and philosophy behind the language helps to gain a better understanding of the language as a whole and in addition the book is an interesting read in and of itself as a case study in software design.

Modern C++ Design

While I wouldn't necessarily recommend applying many of the coding techniques presented in Modern C++ Design by Andrei Alexandrescu in a production code base, I would recommend reading the book. I found it an interesting read and quite an eye opening experience as an insight into just how powerful and flexible (and potentially terrifying!) the C++ template system really is. Read it to understand some of the techniques of template meta programming, even if once you understand them you decide they're best avoided in most production code.

Differences between console and PC development, Part 2

mattnewport — Mon, 17 Mar 2008 06:24:00 +0000

Relatively stable and robust tool chains on PC vs. less mature tools on consoles

Compilers, debuggers and IDEs used for PC development are generally fairly mature and fully featured as they have been developed over many years and are used by large numbers of developers. Platform APIs and libraries are also usually fairly well tested and stable. By comparison, consoles generally have less mature tool chains and platform APIs, particularly early in their life cycle, and as a result developing on consoles has some problems that are not usually present on the PC.

PC developers are commonly advised to trust the compiler to do a good job of optimizing their code and to focus on algorithmic optimizations rather than low level optimizations. Compilers targeting x86 processors on the PC are mature and have many man years of work invested in their optimizers. Modern x86 processors are also quite sophisticated and use many techniques to get fairly good performance from code that is not optimally scheduled to hide latency or is branch heavy.

Console CPUs on the other hand are often non-x86 architectures and the compiler provided to developers will often not have had as many man years of work spent on its optimizer. Console CPUs are also often simpler than modern PC processors in order to keep costs down and are not as good at dealing with sub-optimal code. This means that trusting the compiler to do a good job of optimizing any code that's thrown its way will generally leave more performance on the table than it would on the PC. On recent consoles its usually possible to get fairly optimal performance without having to resort to writing assembly but only if code is written with a view to helping the compiler generate optimal code.

The compilers available to console developers, particularly early in the lifetime of a console, will generally have a particularly tough time dealing with more complex C++ constructs: features like templates, exception handling and virtual functions are often relatively more costly on a console than on a PC. This is partly a function of less mature compilers and partly a result of the relative performance of the console CPUs when dealing with some of the code constructs commonly used to implement these C++ language features. Historically it was not uncommon for the compilers available on consoles to not even support many C++ features, though more recent consoles usually have fairly complete support for the C++ language but with particular features having a higher than normal performance cost.

The relative lack of maturity of console tool chains compared to what is available on the PC also means that it is more common to find bugs in compilers, libraries and other parts of the tool chain. While on the PC it is very uncommon for an application level bug to trace back to a compiler bug and relatively uncommon for it to be due to a bug in common platform libraries, early in the lifetime of the console these are both possibilities that you have to consider when investigating any strange behaviour in your code. Again, compiler bugs are most likely to be encountered when using some of the more complex and less commonly used features of C++.

One bright spot in the tool situation on consoles is that developers usually have access to very comprehensive and sophisticated performance profiling tools, particularly once the console platform is established and has had a few years of development effort dedicated to its tool set. The tools for profiling CPU and graphics performance generally provide much more detail and much lower level performance information than is typically available to PC developers. While in many areas console tools are weaker than those available on the PC, this is one area where consoles generally have the edge.

Running games directly from read only media on console vs. hard drive on PC

PC games are almost always intended to be installed to the hard drive and load from this relatively fast storage device when played. Console titles on the other hand have traditionally been designed to run directly from the media they are shipped on - generally an optical disk on modern consoles. Some console games will use or occasionally even require a hard drive to improve load times but this is still the exception rather than the rule. Console optical drives generally have lower bandwidth (how fast sequential data can be read off the disk) and much slower seek times (how long it takes to move around the disk to find an item when reading non-sequential data) than a hard drive. The performance characteristics of optical media vs. a hard drive mean that console titles generally have to put a lot more effort into ensuring they load game data efficiently than PC titles in order to achieve acceptable load times.

The relatively slow seek times of console optical drives mean that it is important that game data should be arranged on the disk so that it is loaded in large sequential blocks with as few seeks as possible. Compression can be useful to speed up loading by reducing the amount of data that must be read as well, though the additional memory and CPU cost of decompressing the data on load must also be considered. Console titles will often have an automated pipeline for processing game data into an efficient format for loading. A common technique used when loading game data on consoles is to process the data into a form that can be loaded straight into a block of memory and used directly after a quick fixup process for any pointers. For games that have a streaming continous world rather than a level based progression there are additional challenges and complexities.

Details of some of the common techniques for efficient loading on consoles will be the topic of a future post.

Specialized hardware on consoles vs. more general purpose hardware on PC

Consoles usually have some fairly specialized hardware designed to make them a cost efficient system for their primary purpose of running games. PCs on the other hand are used for many different tasks and have more general purpose hardware reflecting their general purpose use. In order to get the most out of a console it is important to understand the specialized hardware and apply it effectively. This task is made easier by the detailed information on the hardware that is generally provided by the console manufacturers to developers but sometimes developers have to figure out a certain amount for themselves.

Modern games need to do quite a lot of heavy duty floating point number crunching for things like physics, 3D graphics and animation. Modern consoles have hardware designed to provide a large amount of raw processing power for floating point calculations but in order to provide this power at a reasonable cost they make design choices that require programmers to use this hardware in the right way to get good performance. Whether this number crunching power is provided as SIMD/vector extensions to the CPU instruction set, special purpose vector co-processors with their own unique instruction sets, through programmable units on the graphics processor or through some combination of these, it is generally accessed through non-standard extensions or APIs which developers must familiarize themselves with. While modern PCs also provide SIMD instruction sets and programmable GPUs with large amounts of floating point processing power, they are often less heavily used by PC games than by console titles.

Consoles often provide other specialized bits of hardware that can provide a significant performance boost when used appropriately. Things like specialized hardware for decompressing audio or texture data, special purpose fast memory for frame buffers or 'scratch buffers' for CPU calculations, special fast data paths between different processors or different memory areas, or custom CPU instructions geared towards specific operations common to games. Developers who know the details of the hardware available on the platform can use it to squeeze more out of the system than developers who don't utilize the unique features available. Sometimes clever use of specialized hardware can allow console developers to do things that are not possible on a PC unless it has general purpose hardware that is significantly more powerful than the general purpose hardware available on the console.

Cross compiling on consoles - targeting a different CPU architecture

Cross compiling is familiar to any developer who is building software on one system that is intended to run on a different system. PC developers are usually used to building and running their game on the same PC but when developing for consoles it is normal to develop and build the code on a regular PC and then transfer it to the console to run and debug it. This has some implications that can trip up developers who have not targeted a different architecture before.

One common source of difficulty for PC developers transitioning to console development is endianness differences between the host and target architectures. PCs are little endian but many consoles are big endian. This means that binary data created on the PC but intended for use on the console must be endian swapped at some point. This can entail quite a bit of work for a codebase that generates a lot of binary data and wasn't written to handle endianness differences.

Any code that makes non-portable assumptions about memory layouts can also cause problems when dealing with cross platform development. This is most often a problem when trying to port an existing PC codebase to consoles. Unsafe assumptions about data layouts and packing in classes or about details like vtable layouts or alignment requirements can cause code that worked on PC (but was actually relying on undefined behaviour) to blow up in nasty ways on other platforms.

Conclusion

I've tried to cover what I think are the most important differences between console and PC development but I've only skimmed the surface of many of these issues and future posts will delve into more detail on some of them. I've probably neglected to mention some other important differences as well but I hope this post will at least give a broad overview of some of the challenges to expect when moving from PC development to console development.

Differences between console and PC development, Part 1

mattnewport — Fri, 14 Mar 2008 06:37:00 +0000

The differences between console and PC development stem from the different characteristics of the platforms. Some of the key differences that have a direct impact on programmers developing for consoles include:

Fixed console hardware vs. endlessly variable PC configurations.
Minimal or no operating system on console vs. large and general purpose OS on PC.
Hard memory limits on consoles vs. disk backed virtual memory on PC.
Relatively stable and robust tool chains on PC vs. less mature tools on consoles.
Running games directly from read only media on console vs. hard drive on PC.
Specialized hardware on consoles vs. more general purpose hardware on PC.
Cross compiling on consoles - targeting a different CPU architecture.

I'll cover each of these areas in turn and discuss some of the implications of these differences on the way developers must approach the task of programming for the platforms.

Fixed console hardware vs. endlessly variable PC configurations

One of the most obvious and most important differences between console and PC development is that console hardware is fixed whereas PCs are very variable in what CPU, graphics hardware and memory they have. With console hardware being fixed and developers being covered by NDAs with the console manufacturers, developers will usually have much more direct hardware access and much more detailed information on the low level design of the hardware than on the PC.

Targeting a fixed, known hardware platform makes life easier for programmers in some respects. If the game runs well on the development kits then it should run just as well on every customer's box. This is in contrast to the PC where the different performance characteristics of different CPUs and graphics cards and the varying amounts and speeds of memory on different systems mean that developers must test many different configurations and provide options for performance scaling and sometimes entirely different code paths for different hardware.

On the other hand, in order to deliver the quality of experience consumers expect on a console, programmers must be comfortable with low level hardware details that developers on the PC don't have to deal with (even if sometimes they wish they could!). In order to compete with other titles on the market, console programmers will need intimate knowledge of the hardware at a level which is generally not even available to PC developers. Not every programmer on the team will need to be a low level hardware expert but a successful console team will need some programmers who are very knowledgeable about the hardware.

I'll have more to say about some of the specific technical implications of fixed console hardware in future posts.

Minimal or no operating system on console vs. large and general purpose OS on PC

Modern consoles generally have a minimal OS that provides low level system services but this OS is extremely cut down and special purpose compared to an OS on a PC. Older consoles often had no OS at all - everything was done through direct hardware access or through a minimal set of libraries provided by the console vendor.

The minimal nature of the OS on a console makes a game developer's life easier in many respects. On the PC a heavyweight, multi-tasking OS allows users to run multiple applications simultaneously which means that even discounting the variable hardware configurations discussed above, PC developers will never have full control over the performance of their game. At any time the OS can take resources away from the game for its own use or for the use of another running application. This is one of the reasons why PC games rarely attempt to target a fixed frame rate of 30 or 60 FPS the way many console games do.

On older consoles it was not uncommon for games to depend on very specific details of timing - something only practical when there is no risk of a multi-tasking OS interrupting at any time. As games have got larger and more complex and consoles have introduced OSs that may perform some background tasks not directly under the game's control, console games have come to rely less on very precise timing of sections of code. It is still much more feasible to aim for a fixed frame rate on a console than on a PC however.

Hard memory limits on consoles vs. disk backed virtual memory on PC

This is perhaps the most important difference between console and PC development and the one that typically causes the most pain for developers transitioning to console development from the PC. On the PC the only hard memory limit is determined by the amount of virtual address space available to an application. On 32 bit Windows this is generally 2GB and until recently few games had to worry about exceeding this limit unless they had a serious memory leak. Although running out of virtual address space is now a concern for some PC games on a 32 bit OS, with 64 bit OSs becoming common the 2GB limit increases to 4GB for 32 bit apps and a generous 8TB for 64 bit apps (8TB should be enough for anybody...). On a console on the other hand, the hard memory limit is just the amount of physical memory available on the system - this ranges from 32MB to 512MB for current consoles. A console game that doesn't take a much stricter approach to managing memory than is typical for a PC game will soon run into problems.

The only penalty for exceeding the amount of physical memory available to your game on PC is a performance penalty as memory accesses start page faulting and hitting the page file on the hard drive. While this performance penalty can be severe (as anyone who has experienced disk thrashing while playing a game on a PC can attest) it will not result in a crash. On a console on the other hand, once you've filled physical memory you simply won't be able to allocate any more until you free up some space yourself. A discussion of some of the approaches to managing memory on consoles is easily a post in itself and I will go into more detail on this topic in a future post.

In Part 2 of this post I will cover the rest of the differences listed above.

Introduction

mattnewport — Fri, 14 Mar 2008 06:02:00 +0000

There's plenty of information and resources on the web these days for people interested in pursuing a career in game programming but there are certain aspects of professional game development that are not as well covered. Topics relating to console development in particular are not as easy to find, partly because of the NDAs that cover many of the specifics of developing for particular platforms.

The primary purpose of this blog is to post articles on some of the aspects of professional game programming that I think are not as well known to those outside of the industry. This will include some of the technical considerations that are particularly relevant to console development, though I will obviously steer clear of revealing any platform specific information that is under NDA. I hope some of these articles will prove interesting to those who already have some familiarity with game development and perhaps are considering a career in game programming but haven't had access to much insider information.

Some of the topics I plan to cover in future posts include:

Some of the major differences between PC and console development.
Achieving fast load times on consoles.
Memory management on consoles.
The importance of asset pipelines.
Console specific optimization issues.
Working effectively on large multi-disciplinary teams.

I have gained a lot from freely shared technical information from other programmers over the years and I hope I can help some other aspiring developers through these posts.