VMPI stands for "Valve Message Passing Interface". It was a previously unreleased tool by Valve used for distributed compiles of Source Engine maps. The tool has since then been quietly released around November of 2007 in an update that also Included Orange Box tools for mapping. Both the Episode One and the OB compile tools can use VMPI for distributed map compiles, though the OB compile tool text help has not been updated to show the new command list.
The most popular reason for Valve not releasing VMPI until late 2007 was because it required a MySQL server for tracking network statistics across the master and worker machines. The new VMPI does not require a MySQL server and therefore doesn't have a license restriction for using external code and programs.
How it Works
The new VMPI is significantly different than the old VMPI in the way it works. The most notable difference is that worker machines do not run a system service to "announce" their presence on the network. This means that worker machines are not automatically detected anymore and must be manually started in polling mode before the master starts a compile. It also means that a local copy of Steam must be installed on the worker machines with the Source SDK and at least one Source engine game installed. (previously VMPI uploaded the required files and binaries to a network directory in which all of the worker machines ran from.)
To let it be known before you start, the Orangebox version of VMPI is broken (and has been broken for over a year now) and will not compile a map properly. The problem is known to reside in the code dealing with ambient leaf lighting in VRAD. VVIS on Orangebox is unaffected, and Ep1 VMPI works without any issues.
Before you start a VMPI compile, you need to have at least two machines (one master and one worker.) You also need a copy of Steam and the Source SDK (along with one Source based game) for a compile to work. The worker machine(s) must also have at least Windows 2000 installed (Windows 95-series OSs are not supported).
- To start a master machine in Hammer, all you need to do is add the "-mpi" flag to the compile parameters of both VVIS and VRAD. VBSP cannot be run in VMPI mode, and really doesn't need to be. If you use the command console (cmd.exe) or a batch compiler, it works basically the same as all you need to do is add the flag on. You may also want to add "-low" also so your machine doesn't lock up while compiling a map.
- Starting a worker machine is somewhat different. First, you need to select the right version of compile tools for the job. Ep1 compile tools are not VMPI compatible with Orangebox tools and vice versa. After you figure that out, open a command console (cmd.exe) and navigate to the directory with the compile tools. You can generally use these shortcuts in the command console to get there quicker:
Once you're there, take a look at the master compile process and see which tool that it is currently running. Start VVIS on the worker for VVIS on the master, and VRAD for VRAD. An example to start the worker machine is below:
vvis.exe -mpi_worker <master IP Address> -mpi_port <port> -low
vrad.exe -mpi_worker <master IP Address> -mpi_port <port> -low
If you get an error about missing gameinfo.txt, add this additional flag to tell it where gameinfo.txt is:
-game "%sourcesdk%\..\half-life 2\hl2"
There are several additional options you can set on the compile tools which show things like stats and the total work done. Here is a complete list of the VMPI related commands on the compile tools:
Enable VMPI mode on the master machine.
Used on the worker machine to connect to the master machine. Put the IP Address of the master machine after the command. (Ex: -mpi_worker 192.168.1.100)
Used on the master and worker machines to change the default port number to something else. (Ex: -mpi_Port 2946) The default port VMPI uses is 23311.
Used on the master machine to show a graphical output of the work that's being done. If you have ever used the Windows 9x defragmentation utility, this will look similar to that.
- Grey Blocks - Work units not yet sent.
- Green Blocks - Completed work units.
- Light Green Blocks - Work units assigned to master.
- Blue Blocks - Work units assigned to worker machines.
- Red Blocks - Mentioned as "sent work", usually not seen.
Blue and light green blocks are more specifically single CPUs on a worker and master. If a worker or master has multiple CPUs or CPUs with multiple cores, it will be represented by additional dots. Dual cores will be two, quads will be 4, etc.
Here is an example of 18 processors working on compiling a single map (16 worker CPUs + 2 master.) What's great about the current VMPI is that if you have the correct ports forwarded on your router, you can have machines all over the world help you compile your map. A caveat in this is that both the master and worker machines need decent internet connections with lots of bandwidth, or the workers can time out while downloading resources from the master and slow the compile down.
Used on a worker machine to keep polling a specified master until the master starts a compile. If not used before a master machine starts a compile, the worker will time out and display an "MPI_Init failed" error.
Used on a worker machine to auto-restart polling mode after a compile is finished. This command is buggy and doesn't always work, so don't rely on it. Must be used in conjunction with "-mpi_Retry" if you don't want to keep getting the "MPI_Init failed" message.
Used on the master machine to enable a debug menu during compiles ("-mpi_Graphics" automatically enables it.) To access the menu, press "D" on the keyboard during a compile. This command also seems to be buggy as I have never gotten it to work. --The HAVOK (talk) 04:50, 11 November 2016 (UTC)The program needs to be run with administrative privileges for this to work.
Shows the statistics of the workers used in the compile and how much work they have completed. Statistics are shown after the compile is completed.
Used on the master to delay the compile until a keystroke is pressed. Useful to let workers connect before the master starts the compile.
Used on the master to set the maximum amount of workers allowed in the job.
Used on the master machine to spawn a local worker on the master machine. Only used for testing.
Throttles the rate at which files can be downloaded from the master machine to any given worker machine.
Level of debug output by VMPI. Either 0, 1 or 2.
Don't process work units locally (on the master.) Useful if you only want the master to track compiled data instead of processing work units.
Over the Internet
If you intend to compile over the internet, you must make sure that the end where the master resides has proper port forwarding through its router (if any exist). The default is port 23311 on both TCP and UDP. If you intend to use a non-standard port, make sure that the port is also forwarded on both TCP and UDP.
Garry's Mod workaround
In order to get VMPI working on Garry's Mod's tools, you must create 2 blank text files in the BIN folder called:
If you don't create those 2 files, when you start the VMPI master, it just won't load up and will crash.
Garry's mod's vrad VMPI works perfect, however, if you don't utilize the flag -mpi_AutoRestart, you must restart the workers when compiling HDR lighting.
Also, the only way to get VMPI to work for Garry's Mod's Tools is creating a bat file/shortcut to vvis.exe and vrad.exe with all the flags you need. (don't use third party software like VBCT)
Adding -vmpi to the CS:GO vvis or vrad will display an error:
Initializing VMPI... VMPI running in SDK mode but incorrect SDK install detected (error 0).
You can circumvent this by patching the dll file to jump over the corresponding if statement. This trick will work for vvis_dll.dll but not vrad_dll.dll, which probably inherited its brokenness from earlier versions of Source. You can download a pre-patched version here. (Mirror: CM2.Network). Please take notice of the disclaimer and back up your original vvis_dll.dll and vrad_dll.dll before you replace them.
Additional CS:GO VMPI specific commands
These are additional optional (some hidden) arguments found in the CSGO vvis_dll.dll:
Calculate a CRC for shuffled work unit arrays in the SDK work unit distributor.
Automatically launches vmpi_job_watch.exe on the job.
Similar to -mpi_AutoLocalWorker, but the automatically-spawned worker's console window is hidden.
Don't set worker thread priorities to idle.
Delay and group some of the worker packets instead of sending immediately.
Enables the use of a database to store compile statistics.
Enables the workers storing all of their text output into the stats database.
Warning - this should only be used for testing and with 1 worker!
Don't timeout VMPI sockets. Used for testing.
Force VMPI to run in SDK mode.
Use the default work unit distributor. Optimized for high numbers of workers, higher numbers of work units, and lower latency. Note that this will automatically be used in non-SDK distributions. --Walki: Tested & Broken unfortunatly
Use the SDK work unit distributor. Optimized for low numbers of workers and higher latency. Note that this will automatically be used in SDK distributions.
VMPI performance appears to be pretty bad compared to the amount of CPUs you have to invest. Take these tests with a pinch of salt as there could have been the following performance influences: wine (to run VVIS on Linux) and VPS (multiple vCores splitting the same physical core).
- 10 x 4 vCores (1.80GHz) (VMPI, ~100% Usage): 1 hour, 12 minutes, 3 seconds elapsed (Figure 2)
- 4 physical cores (3.4GHz) (8 threads, ~100% Usage) + 4x4 vCores (VMPI): 47 minutes, 1 second elapsed
As a comparison here is a pure local compile:
- 4 physical Cores (8 threads, ~100% Usage): 52 minutes, 53 seconds elapsed
The tests show that a combination of dedicated high frequency and numerous low frequency VPS CPUs are the optimal way to go if you want to use VMPI to it's best extend. During the end of the compile process some CPUs will work on the same world unit (WU) which makes it likely that a high frequency CPU will finish this task first thus not leaving all the other cpus idling like in a 32 threads compile.
SENC (Source Extended Net Compiler) provides a GUI for VMPI compiles.
Additionally, its worker comes with a fake (but not in an illegal sense!) Steam installation, allowing you to run workers on whatever machine you please.
(Site has been down for a long time. Mirrored: http://mirror.cazzaserver.com/senc/)