VMPI: Difference between revisions

From Valve Developer Community
Jump to navigation Jump to search
(redone)
Line 1: Line 1:
'''VMPI''' stands for "'''V'''alve '''M'''essage '''P'''assing '''I'''nterface". It was a previously unreleased tool by Valve used for distributed compiles of Source Engine maps. The tool has since then been silently released around November of 2007 in an update that also Included Orange Box tools for mapping. Both the Episode One and the OB compile tools can use VMPI for distributed map compiles, though the OB compile tool text help has not been updated to show the new command list.
'''VMPI''' (Valve Message Passing Interface) is a tool that distributes map compiles over multiple computers, either on a LAN or the Internet. Every computer runs VMPI in Valve's offices, even the receptionist's, allowing production-quality maps to be spat out in a matter of seconds.


The most popular reason for Valve not releasing VMPI until late 2007 was because it required a MySQL server for tracking network statistics across the master and worker machines. The new VMPI does not require a MySQL server and therefore doesn't have a license restriction for using external code and programs.
VMPI was released to the public with [http://store.steampowered.com/news/1294/ the Orange Box SDK update]. It is thought that the tool had been withheld up to that point because of a dependence on [[wikipedia:MySQL|MySQL]] that has now been eliminated.


== How it Works ==
VRAD and VVIS can be distributed. VBSP cannot (and doesn't really need to be).
The new VMPI is significantly different than the old VMPI in the way it works. The most notable difference is that worker machines do not run a system service to "announce" their presence on the network. This means that worker machines are not automatically detected anymore and must be manually started in polling mode before the master starts a compile. It also means that a local copy of Steam must be installed on the worker machines with the Source SDK and at least one Source engine game installed. (previously VMPI uploaded the required files and binaries to a network directory in which all of the worker machines ran from.)


== The Master Machine ==
== Usage ==
To initiate VMPI mode in the compile tools, all you need to do is add the "-mpi" flag to both the vvis.exe and vrad.exe startup options (vbsp.exe does not support VMPI mode and really doesn't need to.) When the compile tools are started in VMPI mode, it will open up a port on the local machine (usually 23311, but can be changed) and will allow worker machines to connect and it will distribute the work. In VMPI mode, unlike the older VMPI, the master machine will contribute to the compile (the old VMPI just used the master machine to store the final compile results, therefore if no workers connected, the map wouldn't compile)


There are several additional options you can set on the compile tools which show things like stats and the total work done. Here is a complete list of the VMPI related commands on the compile tools:
Each distributed compile is organised by a master machine, to which worker machines connect. Each machine must have the game/mod being compiled for installed.


* '''-mpi'''
*To create a master, start VRAD or VVIS with <code>-mpi</code>; port 23311 will open to allow workers to connect.
Enable VMPI mode on the master machine.
*To create a worker, start with <code>mpi_worker <master IP:port></code>


* '''-mpi_Worker'''
This will get the job done, but you may also want to consider the following:
Used on the VMPI worker to connect to the master machine. Put the IP Address of the master machine after the command. (Ex: -mpi_worker 192.168.1.100)


* '''-mpi_Port'''
=== Master ===
Used on the master machine to change the default port number to something else. (Ex: -mpi_Port 2946) The default port VMPI uses is 23311.


* '''-mpi_Graphics'''
;<code>-mpi_port</code>
[[Image:Vmpi.jpg|350px|thumb|right|VMPI graphical output example.]]
:Change the master's connection port. Default is 23311.
Used on the master machine to show a graphical output of the work that's being done. If you have ever used the Windows 9x defragmentation utility, this will look similar to that. (Gray blocks = work unit not sent yet, Green = completed work unit, Light Green = work unit being done by master, Blue = work unit being done by a worker machine. There is a mention of a "Red" block as being "sent work" but I have never seen it. The Blue block more specifically is a single CPU on a worker machine. So if a worker machine had 4 processors, it would be represented by 4 blue dots.
;<code>-mpi_timingwait</code>
:Delays the compile until a keystroke is pressed. Useful for letting workers connect.
;<code>-mpi_graphics</code>
:[[Image:Vmpi.jpg|thumb|VMPI's graphical output]]
:Shows a graphical output of the work that's being done.
:*'''Gray''': work unit not sent yet
:*'''Green''': completed work unit
:*'''Light green''': work unit being done by master
:*'''Blue''': work unit being done by a worker machine.
:Each block represents a single core on a machine, so a quad-core processor will have four.
;<code>-mpi_trackevents</code>
:Enables a debug menu during compiles (<code>-mpi_graphics</code> automatically enables it.) To access the menu, press "D" on the keyboard during a compile. This command also seems to be buggy as I have never gotten it to work.
;<code>-mpi_showdistributeworkstats</code>
:Shows the statistics of the workers used in the compile and how much work they have completed. Statistics are shown after the compile is completed.
;<code>-mpi_workercount</code>
:Sets the maximum amount of workers allowed in the job.
;<code>-mpi_autolocalworker</code>
:Spawns a local worker on the master machine. Only useful for testing.
;<code>-mpi_nomasterworkerthreads</code>
:Prevents the master from processing its own work units.


Here is an example of 16 processors working on compiling a single map. What's great about the new VMPI is that if you have the correct ports open on your router, you can have machines from all over the world contribute to compiling your map which this image represents. The only downside is that you will want the remote worker to have a low network latency or the worker may time out or slow the compile down. You also need a decent internet upload speed on your end to prevent remote workers from timing out while downloading resources.
=== Worker ===


* '''-mpi_Retry'''
;<code>-mpi_retry</code>
Used on a worker machine to keep polling a specified master until the master starts a compile. If not used before a master machine starts a compile, the worker will time out and display an "MPI_Init failed" error.
:Keeps polling a specified master until it starts a compile. If not used before a master machine starts a compile, the worker will time out and display an "MPI_Init failed" error.
;<code>-mpi_autorestart</code>
:Auto-restarts polling mode after a compile is finished. This command is buggy and doesn't always work, so don't rely on it. Must be used with "-mpi_Retry" if you don't want to keep getting the "MPI_Init failed" message.


* '''-mpi_AutoRestart'''
=== Either / unknown ===
Used on a worker machine to auto-restart polling mode after a compile is finished. This command is buggy and doesn't always work, so don't rely on it. Must be used with "-mpi_Retry" if you don't want to keep getting the "MPI_Init failed" message.


* '''-mpi_TrackEvents'''
;<code>-mpi_filetransmitrate</code>
Used on the master machine to enable a debug menu during compiles ("-mpi_Graphics" automatically enables it.) To access the menu, press "D" on the keyboard during a compile. This command also seems to be buggy as I have never gotten it to work.
:Throttles file transmission rate (kB/s)
;<code>-mpi_verbose <0-2></code>
:Level of debug output.


* '''-mpi_ShowDistributeWorkStats'''
== Remote workers ==
Shows the statistics of the workers used in the compile and how much work they have completed. Statistics are shown after the compile is completed.


* '''-mpi_TimingWait'''
VMPI workers can connect over the internet. However, high-latency or low-bandwidth connections (including upload speed) can cause timeouts.
Used on the master to delay the compile until a keystroke is pressed. Useful to let workers connect before the master starts the compile.


* '''-mpi_WorkerCount'''
Bear in mind that because data flows both ways both the master ''and'' the worker require an open and forwarded port, TCP and UDP.
Used on the master to set the maximum amount of workers allowed in the job.


* '''-mpi_AutoLocalWorker'''
== SNEC ==
Used on the master machine to spawn a local worker on the master machine. Only used for testing.


* '''-mpi_FileTransmitRate'''
[http://mapping-tutorials.de/david/SENC.zip SENC] (Source Extended Net Compiler) provides a GUI for VMPI compiles.
VMPI file transmission rate in Kilobytes/second.


* '''-mpi_Verbose'''
Additionally, its worker comes with a fake (but not in an illegal sense!) Steam installation, allowing you to run workers on whatever machine you please.
Level of debug output by VMPI. Either 0, 1 or 2.
 
* '''-mpi_NoMasterWorkerThreads'''
Don't process work units locally (on the master.) Only used by SDK work unit distributer.
 
== The VMPI worker machine ==
To start a VMPI worker machine you must possess both Steam and the Source SDK, and at least one Source game must be installed. There is a way to get around installing a Source game though. All you have to do is copy the game directory folder over with just gameinfo.txt in it while keeping the directory structure intact. The basing startup line to start a worker is (<code>tool.exe</code> being either <code>vvis</code> or <code>vrad</code>):
 
tool.exe -game "gamepath\gamedir" -mpi_Worker <Master IP> -mpi_Port <Master Port>
 
Unless you're running a non-standard VMPI port, you don't need to specify the port number. You also may want to tack "-low" on the end of that string so that the worker machine isn't bogged down and only uses spare CPU idle time. If you want to add a worker to a compile that's not on the local network, you will need to be sure that port 23311 (or whatever port is used by the compile) is forwarded on your router and on the master's router (since data goes both ways) and that both the TCP and UDP protocol are forwarded.


[[Category:Tools]]
[[Category:Tools]]
[[Category:Level Design]]
[[Category:Level Design]]
[[Category:Glossary]]
[[Category:Glossary]]
== (Un)official tools ==
[[Image:M config th.jpg|140px|thumb|right|Configuration Page of SENC]]
The 'Net-Compiler' GUI tool is no longer available.
A new tool, called 'SENC' (Source Extended Net Compiler) is available for download.
It is much easier to handle (because it configure itself) and supports the EP2 engine. The tool consists of two programs, the master and the worker. The worker is running without an extra steam installation, so that you can use it on every windows machine in your network.
This tool is released under Creative Commons, so feel free to distribute.
Download: [http://mapping-tutorials.de/david/SENC.zip HERE]

Revision as of 09:04, 5 December 2008

VMPI (Valve Message Passing Interface) is a tool that distributes map compiles over multiple computers, either on a LAN or the Internet. Every computer runs VMPI in Valve's offices, even the receptionist's, allowing production-quality maps to be spat out in a matter of seconds.

VMPI was released to the public with the Orange Box SDK update. It is thought that the tool had been withheld up to that point because of a dependence on MySQL that has now been eliminated.

VRAD and VVIS can be distributed. VBSP cannot (and doesn't really need to be).

Usage

Each distributed compile is organised by a master machine, to which worker machines connect. Each machine must have the game/mod being compiled for installed.

  • To create a master, start VRAD or VVIS with -mpi; port 23311 will open to allow workers to connect.
  • To create a worker, start with mpi_worker <master IP:port>

This will get the job done, but you may also want to consider the following:

Master

-mpi_port
Change the master's connection port. Default is 23311.
-mpi_timingwait
Delays the compile until a keystroke is pressed. Useful for letting workers connect.
-mpi_graphics
VMPI's graphical output
Shows a graphical output of the work that's being done.
  • Gray: work unit not sent yet
  • Green: completed work unit
  • Light green: work unit being done by master
  • Blue: work unit being done by a worker machine.
Each block represents a single core on a machine, so a quad-core processor will have four.
-mpi_trackevents
Enables a debug menu during compiles (-mpi_graphics automatically enables it.) To access the menu, press "D" on the keyboard during a compile. This command also seems to be buggy as I have never gotten it to work.
-mpi_showdistributeworkstats
Shows the statistics of the workers used in the compile and how much work they have completed. Statistics are shown after the compile is completed.
-mpi_workercount
Sets the maximum amount of workers allowed in the job.
-mpi_autolocalworker
Spawns a local worker on the master machine. Only useful for testing.
-mpi_nomasterworkerthreads
Prevents the master from processing its own work units.

Worker

-mpi_retry
Keeps polling a specified master until it starts a compile. If not used before a master machine starts a compile, the worker will time out and display an "MPI_Init failed" error.
-mpi_autorestart
Auto-restarts polling mode after a compile is finished. This command is buggy and doesn't always work, so don't rely on it. Must be used with "-mpi_Retry" if you don't want to keep getting the "MPI_Init failed" message.

Either / unknown

-mpi_filetransmitrate
Throttles file transmission rate (kB/s)
-mpi_verbose <0-2>
Level of debug output.

Remote workers

VMPI workers can connect over the internet. However, high-latency or low-bandwidth connections (including upload speed) can cause timeouts.

Bear in mind that because data flows both ways both the master and the worker require an open and forwarded port, TCP and UDP.

SNEC

SENC (Source Extended Net Compiler) provides a GUI for VMPI compiles.

Additionally, its worker comes with a fake (but not in an illegal sense!) Steam installation, allowing you to run workers on whatever machine you please.

Retrieved from "https://developer.valvesoftware.com/w/index.php?title=VMPI&oldid=87210"