     |
 |
 |
 | | | | |
 |
|||
 |
||||||
          |
 |
 |
||||
|
The Game Asset Pipeline: The Make Tool Probably the single most commonly used dependency-based build tool is make, a utility originally developed for Unix systems but later ported to just about every modern operating system. Make is available in every Unix distribution, and there are many Windows ports available, including direct ports of the Unix versions, and variants that are supplied with most compilers. Make's original purpose was to assist with compiling source code, but it is built in a very generic manner, allowing the invocation of virtually any command line tool as part of the process of converting input files into output files. As such a generic tool, make is not able to use dependency information from the asset files themselves, and instead relies on an external file, known as a descriptor file that specifies both the dependencies between inputs and outputs, and the processing steps that should be performed on them. Make uses file timestamps to determine if a file is up-to-date, by comparing the last modifications of the source and destination files for each operation. Descriptor File Syntax Make's descriptor files are stored in a text file (typically called "makefile"), which is comprised of a series of rules, each of which defines the information needed to build a specific output file. The syntax is very simple: the name of the output file is supplied first, followed by a colon, and then a list of the prerequisites for that file. For example:
specifies that the textures.bin output file depends on the two .TGA files listed. Therefore, if any of those files have a timestamp newer than that of textures.bin (or it simply does not exist), it will be rebuilt. The commands to build the file are specified immediately after the rule, preceded with a tab character to distinguish them.
In this case, the command is simply executed "as is," and specifies directly the files to be operated on. However, make supports macros that can be used to allow rules to operate more easily on lists of files - for example:
In this example, the list of input textures is defined as a macro, which is then subsequently referenced where it is needed rather than supplying the items explicitly. Macros are defined by supplying a macro name, followed by either "=" or ":=" and then the contents. If a variable is defined with "=", then it is a recursively expanded variable. Any reference to other variables will be kept intact in the macro and expanded every time it is used. If, on the other hand, ":=" is used, then it is a simply expanded variable. In this case, references to other variables will be expanded at the time the variable is defined, and the results stored instead. For example:
At
this point, LEVEL1TEX
will contain "body.tga face.tga grass.tga bluesky. As seen in the previous examples, to reference a macro, simply surround the list name in brackets and prefix it with a $ sign (in other words, "$(<macro name>"). There are also some built-in macros (as well as more defined from the host machines' environment, such as the path to installed compiler tools), and a class of macros known as "automatic variables." These are automatically set up every time a command is executed, and contain information such as the target filename and the list of modified dependencies. For a full list of these, see the make documentation. Make also contains various functions that can be referenced in a similar way to variables, and that similarly insert their results into the rule. These can be used to perform many useful tasks such as string manipulation and wildcard expansion (again, a full list can be found in the make documentation). The rules in the definition file describe how to actually build the files referenced, but they will not actually cause anything to happen unless make has a reason to build the file. This will only occur if it is either explicitly asked to (by the user typing "make texture.bin," for example), or the file appears as a prerequisite in another rule that it needs to build (which, in turn, must have either been explicitly specified or invoked from a third rule). In order to provide a convenient way to specify "top-level" rules that build a number of files, make supports phony targets. A phony target is a file that does not actually exist (and will never be created), but is always considered to be out-of-date. This can be used to write a rule solely for the purpose of triggering other rules, for example:
The ".PHONY" declaration defines that the target alltextures should be considered phony; in fact, even without this the rule would still operate normally. However, if for some reason a file called alltextures happened to exist on the disc, and it was newer than the source files (textures.bin and texture2.bin), then the rule would be considered to be up-to-date and skipped. Marking it as phony simply ensure that this can never happen. In this case, the phony rule tells make that when it is asked to build alltextures, it should build the textures.bin and textures2.bin targets (because they are specified as dependencies). This rule can then be invoked by issuing the command "make alltextures" from the command line, or as a dependency of another rule, for example, a rule that makes all of the resources for the game. In addition, asking make to build the special target "all" causes it to build all of the top-level targets in the file (that is, all targets that are not prerequisites of another target). Pattern Rules As make was originally designed for processing and compiling source code, the early versions of the tool required every input file to be explicitly specified somewhere in the input rules. This is generally fine for programs, as the number of source files is usually relatively small, and additions are infrequent. However, this is not generally the case with game assets, and therefore maintaining a file that must contain the name of every single asset in the game soon becomes very unwieldy. Fortunately, later version of make introduced a feature known as pattern rules. Pattern rules are a form of implicit rule (that is, a rule that operates on an entire class of targets, rather than an explicitly specified list) that allow a rule to be defined that is executed on every target whose name matches a specified string pattern. This way, rules that operate on specific types of assets can be easily built. Pattern rules follow exactly the same syntax as normal rules, except that the % character is used to indicate "one or more arbitrary characters" in the names specified. For example:
This rule enables any target with a .tex extension to be built from a corresponding .tga file. The $@ and $< entries are automatic variables that correspond to the name of the target file and the source file for the rule, respectively. For example, if the target file grass.tex exists, then this command would expand to "converttexture grass.tga grass.tex." It should be noted that, like all other make rules, this does not actually perform any actions unless another rule references a file matching it. Wild Cards Therefore, what is needed as the logical companion to pattern rules is some mechanism for specifying groups of files as the prerequisites of a target without actually listing them. This can be very easily achieved through the use of wildcards, for example:
This rule causes all of the files with a .tex extension in the current directory to be built (using whatever rules are available to do so, such as the pattern rule given above) when the alltextures target is referenced. However, this rule will only update existing files so that if a corresponding .tex file for the asset does not exist, it will not be built. Note also that while wildcards will be automatically expanded if they appear in a target or dependency list, in a variable declaration they must be explicitly expanded by wrapping them in the built-in wildcard function, "$(wildcard $.tex)," for example. This is where the string manipulation features of make come in handy. Since what is actually required is not a list of the output files that exist, but a list of the output files that should exist, we can build that list by taking the list of input files and changing the extensions; we know that in this case, every .tga file should generate a corresponding .tex file. This can be done with the following rule:
This rule uses the patsubst function, which performs a pattern substitution. The first argument is the pattern to match (with, as before, % indicating any sequence of one or more characters), the second is the replacement pattern, and the third argument is the input data, in this case, taken from the list of .tga files generated by the wildcard function. This pattern substitution has the effect of creating a list of the target files, by stripping the .tga extension and replacing it with .tex. The creation of this list is placed in a variable definition to improve performance. Since the variable is defined as being simply expanded, the wildcard and pattern substitution operators are only evaluated once, and then the resulting list is stored for re-use. Overriding Rules With this, it is possible to build make files that take source assets of different types from various locations, and build them as required without having to provide explicit rules for every single asset. However, there are often cases where it is desirable to be able to do just that, for example, when there is one texture that looks poor under the default compression settings, or if a special-case is needed to handle the player's character model differently from other NPCs. Fortunately, make provides a very convenient mechanism for doing this. When searching for a rule to build a specific target, make will always use a rule that explicitly names that target if one is available, only examining implicit and pattern rules if none is found. Thus, even though a rule exists that specifies how to build .tga files into .tex files, if another rule is written with a target of player.tex, it will be used to build that file rather than the more general rule. If the same target is explicitly specified by two rules, make will generate an error. Advantages and Limitations of Make Make is a very powerful tool, and the description given here only covers a relatively small fraction of the available functionality. Make is very widely used, and has been tested on many large scale projects. There is even (albeit somewhat primitive) functionality included for running multiple tasks in parallel to improve performance on multi-CPU machines. The descriptor file syntax is somewhat arcane at first sight, but is quite readable and can be easily read and edited by both humans and other applications. In particular, it can be very useful to use external tools to generate portions of these, as a mechanism for encoding dependency information from asset files. Make works very well on Unix systems, where just about any conceivable task can be achieved through shell scripts or other command line tools. However, on Windows systems, less basic functionality is available to command line programs. In practice, though, this is a relatively minor hurdle. Most of the important tools for asset processing can be command line driven (or must be written in-house), and the other "glue" utilities can be fairly simply replaced or rewritten. The main disadvantages of using make are that it only checks for file modification through the file timestamps, and adding support for more complex dependencies (such as those based on asset contents) can be complex, and require many custom tools to build additional dependency information in a format make can understand. Also, make has no native support for integrating with asset management systems, it works strictly from a local filing system. Therefore, for most purposes some form of external program will be needed to handle the task of getting asset updates from the database and invoking make when required to perform the processing tasks.
|
||||||||||||||||||||||||||||||||||||||||||||||||
Features
|
|
|