This document describes the basic resource management system in the Bitsquid engine – how a resource gets from a file on an artist’s computer and into a running game. It describes, in general terms, how resources are exported, compiled and loaded into the game.
This document does not discuss topics that are specific to only certain types of resources, such as how to use the Maya exporter or what format to use when exporting normal maps. These things will be discussed in special artist how-to-documents.
To understand how resources work in the Bitsquid engine it is important to first be aware of some basic concepts.
A resource in the Bitsquid engine passes through three distinct stages:
The stages of the content pipeline
Artist Data are the original source files that the artists work with in Maya, Max, Motion Builder, Photoshop, etc. From these content tools, the data is exported by the artists to an Intermediate format. The intermediate format is designed for stability, compatibility and flexibility rather than for maximum efficiency. For graphics, we use Collada and DDS as the intermediate formats. In-house content tools, such as editors, read and write the intermediate formats directly.
Before the game can be run, the intermediate data must be compiled to a Runtime format. The runtime format is platform specific and designed for maximum efficiency on a specific target platform.
Usually, the Intermediate Data is the data that gets stored in the Subversion/Git/Mercurial/Perforce repository for a game. The Artist Data is kept on a backup server (or perhaps checked into a separate source database). The Runtime Data is usually only kept locally, on each user machine, as it can be regenerated from the Intermediate Data automatically, whenever it is needed.
The reason for having the Intermediate Data is that it provides a layer of independence which makes it much easier to change the other two ends of the pipe. We can use any content tool we like (even custom written ones), as long as it can output data in the intermediate format. And since the compile process only depends on the intermediate data, we don’t need to have access to different versions of MAX and Maya and a lot of strange exporters to compile the data.
In the other end, we can change the runtime format in whatever way we like as we try out different optimizations. All we have to do is to change the compiler so that it produces data in the new format. The intermediate files are automatically recompiled whenever the runtime format changes. This flexibility makes it a lot easier to add features, experiment with ideas and optimize the engine.
Note
The Bitsquid engine does not care how the Artist Data is organized, managed and backed up. That is of course something that is can be very important to a game project, but it is not something that concerns the Bitsquid engine. It only works with the Intermediate and the Runtime data.
All the intermediate data for a game should be organized under a single root folder on the disk. We will refer to this as the source folder for the game or the project root. We will also refer to the intermediate data as source data.
You can organize the source folder and its subfolders in any way you like. For example, you could organize it by type and put all the game’s textures in a textures folder. Or, you could organize it by the game objects and put the textures that belong to the player’s car in vehicles/cars/player_car. The engine doesn’t care what strategy you use. Pick whatever fits your project and processes best.
Project layout for a very small test project
A resource in the Bitsquid engine is uniquely identified by a type and a name. The type of a resource is its extension. And the name of the resource is its full path from the project root, without the extension.
So the file found at units/characters/player.unit in the source folder has the name units/characters/player and the type unit.
Note
Note that the name is a name, not a path. The name must match perfectly. The character units/characters/player can only be referred to with that name, not as units\characters\player or units/characters/../characters/player. And the name is always the full path to the file, no matter what directory the referring document lies in.
(In the runtime format, the full names of the files are not kept, instead they are hashed to a more efficient format)
Sometimes you may want to have several versions of the same resource for different locales. For example, a piece of dialog may be different in English and in French.
In the Bitsquid engine this is achieved through file properties. A file can have a set of string properties associated with it, such as en or fr, small or large. In the running game a property preference order can be set up. For example, the game may be set up to first look for a (en, large) resource, then for an (en) resource, then for a (large) resource and finally for a () resource.
In the source data, properties are specified with extensions that are added before the final extension (which specifies the type of the file). Like this:
| Path | Type | Name | Properties |
|---|---|---|---|
| sound/phrases/hello.en.wave | wave | sound/phrases/hello | en |
| sound/phrases/hello.fr.wave | wave | sound/phrases/hello | fr |
You may wish that a controller image should look different on the X360 than on the PS3 for example. This is achieved by specifying the platform in the file properties like this.
| Path | Type | Name | Properties |
|---|---|---|---|
| controller/buttons/select.en.ps3.texture | texture | controller/buttons/select | en, ps3 |
Note
Note that the property preference order does not include the platform since this is fully automatic. A platform specific version always overrides a platform independent version and platform specific files for other platforms than the current running will not exist in the runtime data.
When you run the compilation step, the data in the source folder is compiled and put into the runtime folder.
Compiling data
During a normal compile there is a 1-1 correspondence between the source data and the runtime data. I.e., each source file is compiled to a separate runtime file. The runtime folder stores the compiled files as regular files named by increasing numbers: 0, 1, 2, 3 ...
If the data has been compiled before, only the files that were changed since the last compile are recompiled. This means that an incremental compile is usually a fast procedure.
The loading time for a game that has been compiled in this way is not optimal, because to load the game the system has to open a lot of individual files which involves a lot of hard-drive or DVD seeking. To optimize the loading times, a special build called a bundle has to be made. In a bundle, all the data needed to load the game is put in a single file, in the order that the data will be read, so that the entire game can be read without any disk seeks.
Bundling gives optimal load times and the bundled game exactly matches the format that will be delivered to the end customers. However, making a bundle is a much slower procedure than doing incremental changes, so it is not something you do if you want to make frequent changes to the data.
Note
The patching system has not yet been written.
Almost all data in the source folder uses a generic text-based configuration file format which is a slightly extended and simplified version of JSON . The exceptions are scripts (which are plain lua files), raw textures (stored as uncompressed DDS), meshes (stored as COLLADA files) and animations (also stored as COLLADA files).
A typical file in the Simplified JSON (or SJSON) format can look like this:
// The script that should be started when the application runs.
boot_script = "boot"
// The port on which the console server runs.
console_port = 14030
// Settings for the win32 platform
win32 = {
// Sets the affinity mask for QueryPerformanceCounter()
query_performance_counter_affinity_mask = 0
}
render_config = "core/rendering/renderer"
SJSON has been modified from JSON to have a slightly cleaner look, be friendlier to hand-editing and support comments. The changes from regular JSON are:
For debugging purposes it is easy to inspect and modify the SJSON files in a regular text editor.
The Bitsquid engine loads data in resource packages. A resource package is a set of resources that are used together. In a typical setup, there is one resource package for each level in the game, and that resource package contains all the units, graphics, animations, textures, etc used in that level. But that is not strictly necessary; you are free to arrange the resources into packages in any way you like.
A resource package is specified by a .package file that specifies all the resources that the package should load:
unit = [ "units/testbox/testbox" "units/cave_podroom/cave_podroom" "units/player/player" "units/drone/drone" "units/tesselation_test/tesselation_test_01" ] lua = [ "lua/game" "lua/freeflight" "lua/thread" "core/engine_utility" ]
Unit dependencies, such as materials, textures, animations, etc are automatically made part of the package.
Resource packages can be loaded in the background, while the game is running. The script can do regular checks to see how far the loading has progressed, and when it has completed, the script can bring the resources in the package online and start using them.
The resource packages can be used both for a traditional setup with distinct levels separated by loading screens or for a streaming solution with no loading times.
In a setup with distinct levels and loading screens the game first loads the resource package for the loading screen game world. This should be a very small package so that the loading screen can be displayed almost instantly, but note that the loading screen is just a regular game world like any other. You can have sound, animations, 3D graphics, mini games or whatever you like in the loading screen. So in fact, it is more suitable to call it a loading world.
The loading world is displayed and updated, while the main level is being loaded. When the loading has completed, the loading world is hidden and the main level is displayed.
If the loading world uses very little memory, you may want to keep its resources in memory at all times, so that it can be displayed instantly between levels. This would give you a data flow like:
If the loading world is more expensive, you can load and unload it between levels:
For more flexibility, you can combine this basic setup with optional packages. For example, if the player can choose between two main characters, you can put them in separate resource packages and only load the one that the user has selected.
In a streaming solution each resource package would be a suitable “chunk” for streaming.
For example in a 1-dimensional streaming game world setup as below:
A possible setup for one-dimensional streaming
Each zone would be separate resource package and the game would keep two zones in memory at all times, starting with Zone A and Zone B. When the player moved into the shaded area in the figure the game would unload the resources for Zone A and instead load the resources for Zone C.
In an event based streaming system, the game might keep the level geometry in memory at all times and instead stream in the resources for specific events (a boss fight, a harvest festival, etc) as they occur.
You can also imagine combinations of 1-, 2- or 3-dimensional world streaming together with event streaming, all depending on how much complexity you are willing to accept in managing the loading and unloading of resources. As long as you can divide the streaming up into suitable chunks, the Bitsquid engine does not care what method you use.
Note
The Bitsquid engine does not advocate any particular type of streaming. It is up to each game project to decide what data to stream, how to stream it, when data should be loaded and unloaded and how the system memory should be divided between loaded chunks and streaming buffers. In addition, processes and tools must be set up so that artists and level designers can work efficiently with the streaming model that has been chosen.
These are a lot of technical decisions to make, but they are outside the scope of the engine.
No special considerations are needed to work with the game source data. You can move, delete and rename the files freely. When you export data from artist tools, save it in any suitable location in the source folder. You can open and edit the files in the source folder using the supplied content tools or standard text.
The Win32 engine executable is used to compile the data for all platforms. To compile the data you start the executable with some special command line options: engine_win32_development.exe -data-dir DATA -source-dir SOURCE -compile
When the data has compiled the result looks like:
Sample runtime data
The folder 0 holds the compiled files (named 0, 1, 2, 3…). The other files contain information about the compile process.
If there are any errors during the compile procedure, they will be printed to the command window. You need to fix them and recompile before running the game.
To run the game you just put the executable in your runtime directory and double click it. If you want to you can also specify on the command line to use another runtime data directory:
engine_win32_development.exe -data-dir DATA
If you run the game with the –compile flag and the –continue flag, the engine will first do an incremental compile of the data and then it will start up the game:
engine_win32_development.exe -data-dir DATA -source-dir SOURCE –compile -continue
Most resources in the game can be hot-reloaded without restarting the engine. This is done through the console command reload:
reload TYPE [NAME]
If you specify both a type and a name for reload, that specific resource will be reloaded. If you only specify a type, all resources of that type will be reloaded.
Note that if you change a resource, you must recompile it before you reload it to see the changes you have made. So you first need to run the compile command to compile the resource, and then the reload command to reload it. Most Bitsquid tools have this functionality built in, so that you can both recompile and reload with a single hot key. For example, in the console, you can press F5 to recompile and reload the Lua scripts.
From the script, the main interaction with the Resource Manager is to queue packages for download on the background thread (ResourcePackage.load), wait for the download to complete (ResourcePackage.has_loaded) and then bring the packages online (ResourcePackage.flush). And finally to unload the packages when you are completely done with them (ResourcePackage.unload).
When loading a package, you can either flush() it immediately, which will stall the application until the resources are online:
ResourcePackage.load(loading_screen_package) ResourcePackage.flush(loading_screen_package)
You typically only want to do that for the initial package containing the loading screen. For other packages, you want to update and animate the loading screen while polling to check if the package has been loaded:
ResourcePackage.load(level_package) ... function update() if ResourcePackage.has_loaded(level_package) then ResourcePackage.flush(level_package) -- Create the level ... end
Before unloading a package, you should make sure that the resources of the package are no longer in use. This means destroying any units that were spawned from the package, etc.
Whenever you use a resource in the script, for example when you spawn a unit:
World.spawn_unit(world, “units/test/box”)
The loaded packages will be check for that resource. If the resource is not found in the loaded packages you will get an error message. It is the responsibility of the gameplay programmer to make sure that all needed resources are loaded and that resources are unloaded when they are no longer in use.
This section describes the implementation details of the resource management.
The basic operation of the Data Compiler is to look at each file in the source folder, invoke a compiler on it (based on its type) and write the result of the compile to the runtime data folder.
The compile process
A compiler for a particular type is just a function that takes the source file as input and produces a binary blob as output, the result of the compilation. The binary blobs are stored in the runtime directory in files with increasing numbers: 0, 1, 2, 3 … For each blob, the compiler also stores an entry in the data base index that maps the name, type and properties of the resource to the file number where the data is stored:
(unit, units/characters/player) --> 17
This index is later used when resources need to be loaded from disk.
To save space the full strings “unit” and “units/characters/player” are not stored in the index. Instead they are hashed with a murmur hash to a 64 bit value, the ResourceID.
In addition to the source file, a compiler function can also read any other file in the source tree. This can be useful for reading supporting files. For example, when a unit test is compiled test.unit contains the main unit data, while the model source is found in test.bsi and the physics definitions in test.physics. In addition, it can be used to load compile settings, such as the compression rate for animations.
The Data Compiler keeps track of all the files that a compiler reads when a certain output file is being built and stores those as dependencies of the output file. This allows the compiler to do fast incremental compiles. When an incremental compile is made, the compiler checks the modification dates of all the dependencies of all the compiled files. If none of the dependencies have changed since the last compile, that file is skipped. This means that only the source files that have changed, or that have dependencies that have changed will be recompiled.
In addition to dependency tracking, each compiler also has a version number that is changed whenever the runtime data format for that compiler changes. A change in the version number will trigger a recompile of all the data of that type.
The loaded game resources are kept track of by the Resource Manager. The resource manager keeps a simple map:
(type, name) -> void *
Again, the type and name are not stored as full strings, but as ResourceIDs to save memory. To find a resource, we just look it up in the map, based on type and name, and then cast the void * to the known type of the resource.
Depending on the resource, the void * could be a flat memory block with the data of the resource, a complicated structure with lots of objects and internal references or a handle to a relocatable object.
All resource loading is done on a separate thread by the Resource Loader class. The Resource Loader keeps a simple queue of requests (basically type, name pairs) and loads the binary data for each request in turn. The Resource Manager queues the requests with the Resource Loader and monitors it to see when the loading has completed. Once loading has completed, the resource gets inserted in the Resource Manager’s map.
Each resource type can define up to five callback functions that define how it interacts with resource loading (most resources only define load and destroy):
A Resource Package is simply a collection of resources: (type, name)-pairs. When a Resource Package is loaded, it queues all its resources for loading with the Resource Manager and when it unloads, it tells the Resource Manager to unload all its files.
Resource Packages are mainly a way of grouping resources into more manageable chunks for loading and unloading. They also make it easy to speed up loading times. Since a resource package is read as a single unit, we can put all its files after each other in a binary blob and reed them without having to seek.