Shader authoring/Compile Pipeline

The pipeline to actually go from .cpp and .fxc files to a working and compiling shader is quite complex and has a history of causing problems. This document will aim to describe all of the steps of the Valve shader compiling pipeline to assist user understanding, aid trouble shooting efforts, and serve as a quick reference guide.

The stages of the pipeline are listed in order, and each entry will describe the command line format and the purpose of the script. Command line arguments place [] around paths/user variable inputs and the placeholder names will include any file type extensions if they should be included.

buildshaders.bat

Command Line

buildshaders.bat [shaderproj] -game [moddir] -source [srcdir]

shaderproj : The name of a text file (with no extension) containing the names of the shader files to be compiled, one per line. moddir : The mod directory located in <steamdir>/steamapps/sourcemods/ srcdir : The directory containing the source, Ex: C:\MyMod\src

Functions

1. Checks for and creates the folders shaders\fxc, shaders\vsh, and shaders\psh.

2. Deletes the old filelist.txt and filestocopy.txt files.

3. Calls devtools/bin/updateshaders.pl to generate a makefile titled makefile.shaderproj where shaderproj is the same as the command line argument.

4. Runs the makefile with nmake.exe.

5.1 Removes duplicates and alphabetizes entries in filestocopy.txt through some neat command piping and uniq.pl. The sorted/unique version is stored in uniquefilestocopy.txt.

5.2 Repeats the above with filelist.txt, storing the result in uniquefilelist.txt.

5.3 Moves uniquefilelist.txt to filelist.txt.

6. Calls runvmpi.pl with the arguments -nompi -game [moddir].

7. Copies the files generated by runvmpi.pl from stdshaders/shaders to moddir/shaders.

updateshaders.pl

Command Line

perl updateshaders.pl [shaderproj] -source [sourcedir] -xbox -nv3x -shaderoutdir [shaderoutdir]

shaderproj : Same as above. sourcedir : shaderoutdir : xbox : nv3x :

Function

Parses the shader files listed in [shaderproj] generating a makefile and checking dependencies.

1. Opens [shaderproj] and scans for file names and adds everything with .fxc, .vsh, and .psh files to the worklist.

2. Creates makefile.[shaderproj] and adds a default build target that depends on all of the shaders in the worklist.

3. Goes through the worklist, adding build targets for each shader.

3.1 For each shader, generates a dependency list and checks for circular dependencies.

makefile.[shaderproj]

Command Line

nmake makefile.[shaderproj]

shaderproj : Same as above.

Function

Calls various preparatory perl scripts. It's rather self explanatory if you open it up.

For those unfamiliar with makefiles, they are simply a series of targets, dependencies, and build commands. Each target is simply a label for the build instructions, and before building a target, it first builds all of it's dependencies. The target default will be built if no other target is specified. In this case, the default target has no build instructions, but references every other target (excluding clean) as a dependency.

uniq.pl

Removes duplicates from a sorted list passed in on the command line.

*_prep.pl

This refers to three different .pl files, where * is replaced by fxc, vsh, or, psh. The command line args and details aren't really important, but the general purpose is to set up all the behind the scenes stuff to allow faked dynamic compiling.

runvmpi.pl

Function

All this does is set up the command-line options for and call shadercompile.exe. Since that's not a script it's much harder to say what exactly it does, but I think it's safe to say it's the exe responsible for actually compiling the shaders.

the VMPI in the title refers to "Valve Message Passing Interface" which allow multiple computers to be used to compile a shader for example

as of now its unknow if this works for shader compiler