Engine Development

Abstract

The purpose of this article is to provide context for what commercial game engine development means in practice. We focus on good practices and lessons learned to help the reader to better understand the real world challenges that engine developers typically encounter.

Introduction

At its core, a game engine is a collection of different systems and processes used to develop a video game. It contains both components that will be part of the final game and some components that are only used during development.

 

Some components that are typically regarded as must haves include

• renderer
• physics engine
• game world management system
• content build pipeline
• math library
• audio engine.

 

The above list is by no means an exhaustive list of all the components that a commercial game engine can be composed of. It does, however, provide some context on the typical feature set comprising such an engine.

What Expertise is Needed?

Most engine code is about organizing data, managing external API states and running different processes to update different kinds of datasets. Thus it is important for the developer of the engine to have a good understanding of algorithmic complexity and knowledge of a wide range of typical algorithms for data processing and management.

 

Another important topic to pay attention to is mathematics. Typically the math skills required when working on engine features revolve around trigonometry and linear algebra. In some cases it is also necessary to use more complex math concepts. However, these requirements are usually reserved for dedicated graphics engineers working on cutting edge renderer features.

The Process

Developing an engine is a complicated endeavour and requires careful management of the development process. Perhaps the most important thing to keep in mind is upholding high levels of efficient communication within the team and between the game and engine teams.

Serving the Game Team

The purpose of the engine is to allow game developers to make games. This immediately highlights the fact that the engine teams should serve the needs of the game teams. It is thus of utmost importance to emphasize communication between the engine team and the different game teams.

 

Features that are “cool” but don’t hold much value for the game teams using those features should not receive high priority. You should always evaluate the value that the specific feature provides and emphasize this when prioritizing your backlog.

Project Management Method

Engine development itself does not dictate any specific project management method. It is, however, important to choose a method that allows the teams to focus on a set of features without suffering from constant interruptions.

 

One such method, which allows safeguarding the team from interruptions, is Scrum. Scrum splits the development process into chunks of a couple of weeks, which allows developers to focus on a chosen feature set. Scrum does not come without its own challenges and it’s quite common place for teams to adopt a modified version of Scrum instead of the rather strict original one.

Code Reviews

In recent years we have seen pull requests and other code review methods becoming mainstream. The idea behind the concept of a pull request is to allow other team members or external parties to review a specific set of changes to the codebase before those changes are incorporated into the development version of the product. Major code hosting services, such as GitHub, already support pull requests as a built-in feature allowing teams to simply adopt the pattern.

 

Having multiple engineers review each other’s code facilitates information sharing and increases the bus factor. It also lowers the risk of trivial oversights finding their way into the codebase. Furthermore, the process builds social pressure to write better code because you will immediately have additional pairs of eyes going through it.

 

It is good practice to think of pull requests as always having at least two reviewers: someone else on the team and you yourself. This means that you should always go through the pull request when creating it to make sure you can explain each line of code if need be. This forces you to think through what you have written and make sure every change and addition has a purpose for being there.

Testing

Very few engineers find their passion in testing functionality. The reality, however, is that a non functioning system is as good as useless. This is especially true with game engines. If an engine feature does not work as expected it puts the game project developed using the engine at risk.

 

Custom in-house engines allow quicker turnaround times for fixing critical issues, but if issues keep constantly popping up this will create tensions between the game team and the engine team and slow down the progress of the project. The larger the team suffering from the issue the bigger the blow back.

Unit Testing

Unit tests are a good way to avoid regressing already fixed bugs. They can also be used as part of Test Driven Development where unit tests are developed at the same time as the feature itself. This provides an easy way to immediately test code that would be extremely difficult to test in the final environment it is used in. The added benefit is that these tests can be kept in place to make sure the behaviour does not change during refactoring, for instance.

Integration Testing

Many features are too difficult to test using unit tests. These include complex interactions between large scale systems. Testing each individual system is typically not enough and mocking is in many cases simply impractical. This can, for instance, be because of the complexity of the system interactions or because of the effects on performance that interfaces supporting mocking would induce.

 

In these cases it makes sense to create a system that allows scriptable test suites. Such as system should allow you to create test sequences where engine features are used on a higher level. These sequences can consist of events such as spawning effects and models, animating characters, playing sounds, etc.

 

Integration tests should be run frequently, preferably after each build. The results can be recorded and even compared to reference material to see if, for instance, the visual output has changed. Simple pixel by pixel comparisons are often hard because of the inherent imprecision of floating point math operations. Complex image recognition systems can be deployed to find defects, but simply storing the recordings of the tests can also be useful. For instance, if an artist notices that something is off, the recordings can be checked to find the revision that broke the feature.

Continuous Integration

Continuous integration allows development teams to constantly have access to the latest and greatest build of the engine. This is typically implemented with the help of a CI tool such as Jenkins that automatically triggers builds on build slaves when certain conditions are met. These conditions include changes being pushed into version control or maybe you simply want a build to happen every day at 3 AM.

 

Continuous integration typically includes build verification. This means running unit and integration tests when the build has finished. It can also mean running code validation tools to check the code against architectural guidelines. If any of these tests fail the build will typically be marked as a failure and the interested parties will be notified through e-mail or on a channel on an internal communication service, such as Slack.

Tooling

Tools are an integral part in developing video games. These come in many shapes and forms put their purpose should always be to make the team more productive in realizing their vision for the game. This means that tools development should be done in close collaboration with the game teams to make sure their needs are satisfied.

 

Tools don’t always have to mimic the functionality and user interface of mainstream engines such as Unity or Unreal Engine. For instance, for an endless runner it might make more sense to have a tool for defining prefabs and a separate parameterization tool. Instead of building the level in a what-you-see-is-what-you-get world builder tool you would simply use the parameterization tool to build the rules for spawning the game objects coupled with an open game window with hotloading support to immediately see the effects of the changes you make to the rule set.

Content Pipeline

It is typically impractical to use intermediate formats such as FBX of PNG in the actual game. The content pipeline is a toolset that allows processing these source assets to produce more optimized versions of the assets that can then be used by the runtime.

 

The pipeline can do different things depending on the assets in question. It can, for instance, perform image downsampling from original high resolution textures, generate pre-convoluted environment maps for physically based lighting, process the source FBX file to automatically extract or generate multiple levels of detail or simply convert an FBX file into the optimized runtime mesh.

 

Content builds are run often even when no content has actually changed. It is thus important to optimize the performance of the pipeline. A couple of ways this can be done:

• use faster algorithms,
• make sure you only touch changed content,
• parallelize building assets based on a dependency graph or implicit dependencies (all textures in parallel with meshes),
• use the GPU to perform expensive operations such as environment map pre-filtering.

 

The content build system should not require visual input/output. This allows it to be run on machines with no displays, such as servers on the cloud, so it can be part of the continuous integration process.

Architecture

The engine’s architecture defines the shape language of the engine code. A clean architecture is the key to making the engine approachable and maintainable. We already discussed pull requests for reviewing code changes, which are a good way to assure an existing architecture is followed. However, architectural changes and additions are more tricky because it’s often too late to make large scale at the pull request review time.

Documenting the Architecture

A good approach is to include engine architecture documentation side by side with the engine code. This documentation is updated when the code shape language is extended or modified and the pull request process is followed just as with normal code changes. This allows you to have a single workflow for approving architectural changes and more detailed code changes. It is recommended that these architecture changes are reviewed by a wider range of engineers to get their buy in on larger scale design changes.

 

It is also important to keep architecture documentation relatively simple and focused. Documentation has the tendency of going out of date if it is not constantly maintained. Furthermore, architecture documentation should mostly cover topics that are not prone to change.

Coding Conventions

It is important to document and follow common code guidelines. C# has a more commonly accepted set of guidelines, but C++ tends to be a wild west in terms of how code is written. It is thus beneficial for the team to come up with a common approach to code layout and naming conventions.

 

Another important aspect is the structure of the code itself. Because the engine forms a framework on top of which the game is built, it is important that the code is easy to understand and use. Good code that uses descriptive names for methods, variables and types also has the tendency of documenting itself.

 

Sometimes it is important to add information in the form of code commenting. The drawback of code commenting, however, is that it is not part of executed code and is easily left unchanged when the code itself has changed. This issue is emphasized by code review tools that tend to only show modified lines with some minimal context. Comments associated with changed code might thus not even appear in the pull request leading to reviewers missing the need to update them.

Programming Languages

C++ is still the most widely used language for developing game engines. It is a versatile and performant language, but it also provides you as a developer a multitude of ways to shoot yourself in the leg. This also emphasizes the importance of maintaining good coding conventions, which were discussed in the previous section.

 

In addition to C++ engines often also support some form of scripting language. Lua is a language that has been widely used over the years but in recent years it has been C# that has taken the limelight. For the most part this has been thanks to Unity. C# carries the benefit of being efficient enough to be used for complex algorithms, but it is also designed with productivity in mind.

Data Oriented Design

In Data Oriented Design code is understood as the process of transforming data from one form to another in an efficient manner. This approach fits engine development really well, because it embodies the understanding that applications are run on real world hardware with real world limitations.

 

The difference between Data Oriented Design and Object Oriented Design is that instead of having objects that communicate with each other through messages you simply have data that is transformed by executable code.

 

One example of Data Oriented Design is the concept of the Entity Component Systems. With this approach objects in the world are defined as a collection of components, which are linked together by an entity. A major optimization opportunity that this approach provides is that all component instances of a specific type can be stored in consecutive memory in order of access to facilitate better processor cache behaviour.

 

The component data can also be split based on how the data is used. For instance, a transform components might have the produced world matrix stored in a separate array, because that data is not accessed as often as the position and rotation of an individual transform.

Serialization

Serialization is required anywhere where in-memory data structures need to be stored on disc or transmitted over the wire. Serialization can generate binary data that is read with the help of a common contract. The generator and reader need to both know the format of the data they are passing to each other. Other types of serialization include generation of textual data such as XML and JSON. Textual formats are often slower to read than binary formats, but easier to debug.

 

Typical uses of serialization are game configuration files and communication between the client and the server. In both of these cases it is important to validate the data that the game is reading. Gracefully handling invalid data is important especially when tools are built to run in the game process. In this case, if error handling were ignored, the application could crash on invalid data and the user would lose all unsaved changes.

 

It is good practice to build validation support into the serialization framework itself. It should be possible to specify validation rules for individual field values that are applied when deserializing the data. You might also need to extend validation with a separate validation function that will also check whether certain combinations of field values are valid, but supporting validation of individual fields is usually enough.

 

Especially on mobile platforms it is important to put emphasis on the performance of serialization and deserialization. Deserialization of large data sets is typically performed at load time and many developers have noted that increases in load times have a significant negative effect on player retention.

 

Many engines also implement general purpose reflection. Reflection is the concept of using metadata generated from the executable code to access information such as data structure layouts as well as enumerating available functions. It typically also allows setting data structure values at run-time and even calling functions.

 

C# has a built-in reflection system but C++, for instance, does not have one. It is, however, relatively trivial to implement such a system on top of C++ using macros or code generation. There are some ready made systems as well, which provide a broad feature set out of the box.

Performance

Performance is an important topic in engine development because engines tend to handle large sets of data. The reader might also be familiar with the saying “premature optimization is the root of all evil.” However, I would encourage reading this article: https://ubiquity.acm.org/article.cfm?id=1513451, as gives a good explanation of why things are not that simple.

 

Overall, it is important to write generally well performing code. If a specific piece of code isn’t currently used on a hot path, this does not mean it never will. An innocent function that was originally implemented to be called here and there might accidentally find its way to being called multiple times per frame.

 

Another important point to remember is that on mobile platforms general code performance does matter. It is not only important to hit 60 frames per second, but it is also important how you hit it. The more instructions you have executing on the CPU the more energy you will consume. This in turn leads to worse battery life and heating. If the device heats up too much, it will start throttling resulting in the frame rate dropping. This effect is often not immediately visible and only manifests itself after the game has been running for some time.

 

It is generally important to choose the right algorithms for the task at hand. Due to the processor being efficient at accessing memory linearly it is often more efficient to use linear arrays of data for small data sets than using a more complicated collection such as a hash map. It is also good practice to keep key values, such as object identifiers as a separate array. This approach makes finding an object with its identifier really fast. An approach inherent in Data Oriented Design, which was covered above.

 

Threading allows processing data in parallel but it can also result in added overhead due to synchronization and complexity that can lead to bugs. Also, using too many threads on systems with only a couple of cores can also reduce performance due to the added thread context switching. Furthermore, you should consider splitting your data into blocks that individual threads can take ownership of. This reduces the need for synchronization. It is always important to verify that your assumptions of the application performance are correct by testing them on real hardware.

Dependencies

Modern game engines tend to be a mix of custom and 3rd party components. Like with any project, it is important to pick the most important fights instead of doing everything yourself. It is important, however, to carefully evaluate components before integrating them.

 

One important factor in this consideration is how much control you have on the dependency. If possible, it is beneficial to only integrate components that you can acquire the source code for. Especially when developing free-to-play games, which will hopefully be maintained for years to come, it is important to have the ability to modify the library if, for instance, official maintenance on it has ceased.

 

To avoid rewriting large parts of the engine in case you have to replace a component with another one, consider implementing an abstraction interface. This interface might not support all the functionality of the component but it can always be extended if needed. The interface should also be generic enough to be able to wrap other similar libraries. For instance, the renderer should use an abstraction interface built around the modern low-level graphics APIs such as DirectX 12, Vulkan and Metal. This allows the API to support the low level APIs and you to benefit from their optimizations while still maintaining support for more flexible interfaces such as OpenGL.

Summary

In this article we cover the expertise needed in developing a game engine, how the development process is managed to maintain a high quality bar and lessons learned regarding different areas of engine development.

 

We explain processes like code reviews that facilitate information sharing and spotting obvious issues with code, but also how they can be used to review engine architecture designs. Unit testing and integration testing are also covered. These methods minimize regressing already fixed issues and help spotting larger scale problems. We also touch upon continuous integration and how testing can be integrated in that process.

 

One of the most important topics in this article is the overall engine architecture. We cover general topics such as documenting the architecture and defining clear coding conventions. In addition to this we cover the topic of data oriented design, which is a very important design approach when developing engines, or any real-time applications for that matter. We also talk about serialization and why it plays a central role in engine development.

 

Engine development might sound daunting at first, but it is important to remember that not all engines are the same. A custom engine should always aim to provide value for the internal teams using it. They can also provide the flexibility that licensed engines cannot, which need to be more general purpose to allow as many teams as possible to use them. It is also important to remember that not all components have to, or should not, be custom implementations. Modern engines tend to integrate a wide variety of 3rd party components mixed with custom implementations.