dave news

Game programming

[NYI] Stuff

The File Manager

This document outlines my design for a game file manager (or asset manager). The file manager is the library in a game which is responsible for finding and loading all of the data files that the game uses. A file manager is a lot like a file system; it must be able to quickly locate and retrieve data spread out among multiple sources and multiple directories.

I don't think this design is anything revolutionary; I wouldn't be surprised if pretty much every game out there has a system that's basically functionally equivalent to this. The basic problems associated with file and asset management are pretty well-known and well-understood.

File names

The primary design decision to make when starting out to build a file manager is how the file manager will identify files. You basically have a choice between requiring filenames to be globally unique throughout the project or using relative paths to compartmentalize files. For smaller sets of data, requiring globally-unique filenames isn't a bad choice. It has some advantages:

  • It can be more efficient to find any given file because the lengths of the strings being compared are usually smaller. Checking for "bodytex2.bmp" will be faster than looking for "models\small\body\bodytex2.bmp".
  • It's easier to detect and eliminate duplicate assets. If you're keeping your data set size down, you don't want duplicate files to accidentally creep in, and detecting them is pretty simple since it's all just based on filenames.

But for most games it's generally better to use relative paths. It allows the artists to group files however they want; it eliminates the possibility of two artists accidentally using the same name for different assets; and overall file management and maintenance is much easier. I prefer to use relative paths.

Package files

The next problem to solve is how to package up the large sets of files that most games will need. It's not really an option to just install all your files all over the user's hard drive, because many file systems (at least under Windows) don't deal very well with large numbers of small files. So you have to use some kind of "package" file which contains all of your individual files. Given that, you have a choice to make: do you go with a well-known package file format (like .ZIP or .RAR files) or roll your own? The advantages of using, say, .ZIP files is that the file-format is well-known, the tools are widely available, and you can often use public, free source code to manipulate these files in your file manager.

The advantages of rolling your own are little more subtle. One is potentially being able to discourage people from snooping through your data files; if you want to protect your data from people, you can hide or encrypt it as you put it into your package file. This is never a very reliable method, since a programmer with enough time on his or her hands can easily break whatever scheme you use, but it might discourage casual snooping. Another reason why you might want to define your own package file format is that you can better control the process of getting your files ready to ship. You might want to have a process which checks for duplicate files, for example, or which sorts files according to access patterns within the game. This is much easier to enforce if you control the packaging tool.

Although in the past I've used .ZIP files for this, I think this time I'll try using my own packaging system. I'm not very worried about compression; disk and memory are cheap and getting cheaper. And if I need to get some compression going, it's not too hard to write your own. Plus, having more control over the order and grouping of resources within my package files gives me the opportunity to try out some optimizations, such as grouping or ordering files by usage patterns.

By the way, one thing to think about if you're allowing users to create mods or plugins for your game is that if you use a custom package file format, your packaging tool will have to be robust enough for end-users. One easy way around this is to support .ZIP files in addition to whatever your custom package file format is.

Mods and Plugins

All games are user-modifyable now; it's just a fact of life. Supporting mods with your file manager can be fairly easy; all you really need to do is be able to load from new package files that users add. Morrowind uses a system where at startup users can pick which mods are active for that gameplay session, which works well. We'll put our system for doing that into the app itself; the app will tell the file manager which packages to load.

A note about mods in multiplayer games: Multiplayer games can make supporting mods much more complicated. You'd like to be able to prevent cheating, and you'd like to be able to support server-side mods that clients don't have to download (like, say, a new gameplay mode that scores kills differently). This design does not directly address those issues, but they're not hard to add. Cheating by modifying data files can be discouraged by CRCing package files or their contents. Mods can be marked as server-side or client-side, and your game logic can restrict what kinds of files can appear in which kinds of mods.

File Types

Often, types of resources can be represented by more than one kind of file. For example, you might have an audio system that supports .WAV or .MP3 files, or you might allow both .BMP and .TGA textures. I'd like to try to build support for this into the file manager, as follows. I'm considering a system where you can define an extensible list of file types and the extensions, in priority order, that can satisfy that file type. I'm not sure how well it will work, since you have to do different things when loading a .WAV vs. an .MP3 anyway, but it might be more efficient to only call into the file manager once rather than have the defaulting happen outside the file manager.

Similarly, I might want to try having a system where you can specify a list of path prefixes to try for each file type. So for example you can say that when looking for audio files, look under the "audio\" directory. This way, any given file search only has to look in a smaller set of subdirectories, which will improve file manager performance. The down side, of course, is that it restricts how your artists can arrange files, but you can't make an omelette without pissing off artists, so there you go.

The Package File Build Process

I want to write a tool to manage creation of the package files. We'll want this tool to solve a couple different problems, including optimizing package layout based on file load patterns and removing multiple copies of files.

Whenever you load a particular level or area or map in your game, you will often load the same set of files in the same order. For example, you might load your level geometry and then load the wall textures for the player's starting room, or you might run through all the entities in the level loading a script for each one. It should be pretty simple to optimize for this kind of access pattern by running an instrumented build which records a list of file accesses; using that list of file accesses to organize files within the package file; and adding a little support within the file manager to optimize looking for the next file in the package.

If you have a file structure where files of the same name can appear at different places in your directory structure, it's possible to end up with duplicate files. In fact, sometimes you want to encourage it. A situation that comes up often is that artists will take texture files, say, from another part of the project to use in their models. If somebody changes the original texture, all models will be affected, which is usually not what you want. The solution is to have the artists copy any files they use into a new directory for each model. This gets you a few benefits: not only is the art protected from accidental change of shared textures, but if models are cut from the game, all associated textures can be easily found and removed. But what do you do about all the file duplication? Simple, at least in theory: when you build your package file, you CRC or hash every file and only pack files with a given name and hash once. In order to fix up the missing references, I'll introduce the idea of "symlinks" -- entries that you retain in the package file that simply point to other entries.

For my own use, I definitely want the package file builder to be a simple, bulletproof command-line app. If I were going to release it to a mod community, I'd probably have to think about writing a nice friendly GUI version, but supporting .ZIP files for mods finesses that problem away.

Other Features

This last section basically just lists, in no particular order, other random features that we will want or need in our file manager.

If you've got a level-based game, it would be nice to only have the current level's files available for searching, in order to reduce file find times.

It might be nice to have a quick way to check for the existence of a given resource. For example, you might have a texture-quality system where you want to load 32-bit textures if they're available but fall back to 16-bit textures otherwise. This can be done using the combination of a file-naming convention and a quick way to check for the existence of a file (alternatively, a find-files method that takes a list of filename options in priority order would work too). It might also be cool to cache file-find misses too. For example, if you've got an entity/object system where each entity might have a script file associated with it, you might do a lot of trying to find files that don't exist. The package file could actually have a (hopefully short) list of nonexistant files in it to optimize those checks away.

I'll want the file manager to be well-instrumented. In particular, I'll want to be able to record all file accesses that occur during a run, and maybe where the files came from. If I have the ability to insert files in sorted order into the package file, it'd be smart to have the file manager's output be in the same format as the build tool's input. It would also be nice to get an idea about disk and memory usage by resource/file type.

Dealing with files that aren't found is an interesting issue. Sometimes it's benign, like if you're looking for a custom script file that need not exist. Sometimes it's not, like when you look for a needed texture file and can't find it -- your model shows up with a missing texture, which looks pretty bad. How to handle this case is sort of a matter of opinion, but I prefer to just log any missing files while running. You can make the argument that it's better to quit noisily or pop up a dialog box or something, especially if you have modders who may not think to check the log file before releasing their mod.

For some files, you want to open them all at once and just get a buffer; this is the case for small files like textures or sound clips. For other files, you want to be able to stream them so that you're not taking up too much memory; this might be the case for larger sound clips used as background music or for full level files, which could be many megs. For now I'm just going to support retrieving buffers for files, but it would be interesting to look into retrieving streams for some files instead. Obviously the user of the file manager would have to decide to request one or the other form of data and know how to consume it appropriately. Clearly, if I ever get around to writing a large game like an RPG, this will have to change.

Game links:


File Manager

Resource Manager


Solitaire spec