Talk:VMPI

Is there any chance of the glue software for VMPI being released for we, the proletariat?

It's just that I got the good-old netvis working yesterday on a map for the original Half-Life, and spread over three machines and four processors, it was stunningly fast. Obviously a few hours later, after the single-machine hlrad had done its work, I discovered that I'd compiled the wrong map. —Cargo Cult _{(info, talk)} 04:31, 14 Jun 2006 (PDT)

Look hard enough around the Internet and you'll find this tool and some company custom compilers. Great stuff.

Pirhana and TomEdwards, stop messing with the page

Stop trashing up the page with unverified claims, wrong usage and incorrect information.

"Every computer runs VMPI in Valve's offices, even the receptionist's, allowing production-quality maps to be spat out in a matter of seconds." Unless you have a source for that, don't post it.

The usage you posted is also incorrect and skewed from the original article I wrote. If you want to clean the article up, by all means do so, but don't trash it up again. The page has been reverted to its old state.GiGaBiTe 18:21, 17 Jan 2009 (PST)

This isn't Wikipedia, but there is one policy of theirs that I'd like to apply here: that you don't run in, revert loads of hard work, then post an angry message about how terrible it's all been on the talk page (unless the changes are obviously vandalism). You instead discuss your problems.

My problems with the page as I found it were overly lengthy paragraphs that don't actually help anyone use the tools, a lack of any distinction between commands that belong to the server and commands that belong to the client, and minor formatting issues. What are your specific problems with the page as it was before you arrived today? --TomEdwards 12:50, 18 Jan 2009 (PST)

Incidentally, the receptionist anecdote comes from an interview/feature done with/at Valve. I can't find it right now. --TomEdwards 12:53, 18 Jan 2009 (PST)

I fire your comment right back at you. You made mass edits to the page without "discussing your problems with the page" which is the first reason I'm angry. The second is that key parts of the article were deleted and replaced with rubbish that was completely irrelevant and wrong. Even though the paragraphs were long and technical, they were explaining how it worked. If you want to be a hypocrite, please do so elsewhere.

I took the time to remove the jargon and formatted it properly so it shouldn't be obfuscated again, because that's how VMPI works.GiGaBiTe 02:49, 19 Jan 2009 (PST)

I didn't have anyone to discuss with. I also wasn't reverting to an earlier version. Do you actually have any specific problems with the page? --TomEdwards 19:01, 26 January 2009 (UTC)

I don't have any problems with the page, seeing as I was the one that originally wrote the article, and did all of the research on it. I do, however have a problem with people like you deleting and obfuscating the article while having no idea what you're doing. You clearly don't know anything about VMPI, and therefore shouldn't be changing things you know nothing about. If you want to do such things, go on Wikipedia and edit some article you know nothing about; Replace the entire thing with "lul i dun know wat dis iz" and see how long you last.

Though from your last comment, you seem to be just trolling around, in which case ignore all of the above and get lost.GiGaBiTe 08:58, 27 January 2009 (UTC)

Running workers without games installed

Perhaps this would work with VVIS, but VRAD needs to read from texture and model files to fulfil its duties. I don't see how a worker could produce good data without them. --TomEdwards 08:08, 5 Dec 2008 (PST)

   VVIS + VRAD send the content to the workers before they start working.

The worker machines download all textures, models, etc. from the master before they start working on the compile. The only file that isn't downloaded is lights.rad, which needs to be on the worker machine if the map has any texture based lights. if the worker starts compiling without lights.rad, you'll end up with strange checkerboarded lighting GiGaBiTe 05:03, 14 February 2009 (UTC)

In a nutshell...

So basicly, VMPI uses resources from an entire network of computers for each compile that runs it, making the compile faster and easier, right? --JeffMOD 03:46, 19 Jan 2009 (PST)

Make no mistake, VMPI does make maps take less time to compile, but it isn't a crutch for crappy mapping as some people mistakenly believe. VMPI isn't going to make carving a 42 sided cone out of a 64 sided sphere work any better than one machine would do it.GiGaBiTe 07:15, 5 February 2009 (UTC)

I doubt if someone was stupid enough to try that, that they'd know about this ;) --JeffMOD 21:30, 30 June 2009 (UTC)

An example please?

I'm building TF2 maps and the compile time is killing My productivity. I have multiple win2k3 machines sitting idle and would like to put them to work. Can someone give a CLEAR example of how to use VMPI across a local network?

Also, about the caveat on the page, does VMPI work for OrangeBox yet? --Obliterous 14:36, 22 September 2009 (PST)

---

--VMPI is still broken in OrangeBox. You can still use VVIS based VMPI, but with func_viscluster and proper optimization, you really don't need it. The problem with VRAD VMPI is that it starts sending ridiculous packet sizes over the network (we're talking 80-90 MB per packet) which are malformed because they're so large and cause the worker machines to crash. Unless Valve releases the source to the VMPI lib, or fixes the problem themselves, there's really no way to fix it. GiGaBiTe 03:15, 21 December 2009 (UTC)

Clarification of "Caveats"

Now, I may not be a longterm user and editor who made many legit edits like TomEdwards, but I guess I can stand being accused of being a troll anyway ;) (or unintelligent, or an idiot, or whatever else not appropriate for a wiki discussion page such as this)... (tl;dr, be polite)

• Anyways, stupid question: Is VMPI in its entirety broken in OB, or is just VRAD broken?

The paragraph starts out with "Orangebox version of VMPI is broken", and then states "VVIS on Orange Box is unaffected." Color me confused. I'm guessing the paragraph is simply stating "VMPI for VRAD is broke for the Orange Box, but works for Episode One. VVIS works for both Episode One and Orange Box.

• Stupider question: Is it possible to export (at least the lights and geometry) of the map to Episode One, and "copy" the collection of luxels with exponents to an Orange Box map?

If it is, then it is worth mentioning in the wiki along with a howto.

• What about the Left 4 Dead SDK?

Does it come with VMPI? Does it work? I don't have L4D yet so I wouldn't know if it is obvious.

QUINTIX256 02:44, 23 December 2009 (UTC)

Someone on a forum I frequent recommended that I try enabling LAA on the VRAD executable to see if it fixes VRAD crashing on the OB engine and it seems to have worked. I'll need to do more testing to see whether it actually makes VRAD work properly.

In the mean time VRAD on OB doesn't work in VMPI mode without crashing, though VVIS will work fine in VMPI mode. Using VVIS in VMPI mode on the OB engine is rather pointless though because func_viscluster fixes the insane portals problem.

GiGaBiTe 11:46, 23 December 2009 (UTC)

VMPI MySQL??! (Alien Swarm)

After looking Through the two files "dependency_info_vrad.txt" "dependency_info_vvis.txt", in the AlienSwarm Bin directory, both list dynamic link libraries "libmysql.dll" and "mysql_wrapper.dll" that don't exist. This could Possibly explain why vmpi doesn't work or work correctly, or what could be missing from the other sdks/authoring tools builds. This may also mean that VMPI is still dependant on MySQL for network compiles, but I personally wouldn't know 100% for sure. If anyone has any harder information to this please post.

"dependency_info_vrad.txt"

vrad.exe shaderapiempty.dll materialsystem.dll vrad_dll.dll tier0.dll vstdlib.dll optional mysql_wrapper.dll libmysql.dll vphysics.dll dbinfo_vrad.txt

dependency_info_vvis.txt

vvis.exe vvis_dll.dll tier0.dll vstdlib.dll optional mysql_wrapper.dll libmysql.dll dbinfo_vvis.txt --Kwp17pitts 00:52, 25 December 2011 (PST)

Those files are very familiar to me. The 2003 Source leak had a version of VMPI that used a MySQL database to track compile statistics. It looks like Alien Swarm might have the original implementation of VMPI (which doesn't make sense since the original compile tools are incompatible with the current engine.)

The SQL statistics part was very buggy and rarely ever worked properly, which might be one reason Valve decided to trash that iteration of VMPI. Another likely reason is that Valve probably didn't have the rights or licenses to distribute MySQL libs (which is why they're missing.) If you can figure out what version of MySQL VMPI wants (I recall something in the 4.0x branch), you can copy the required libs over and see if you can make it work.

The text files you speak of (dependency_info_vvis/vrad.txt) tell VMPI what dependencies that the worker machines will need to successfully complete a network compile. Basically you have to make a public folder somewhere on the network where all machines participating in the compile can access, and the files listed in the text files are uploaded to that folder for the worker machines to use. Like the SQL stats, this process is also really buggy and difficult to get working. It also makes it impossible to have workers on the WAN (unless you have a VPN or something.) GiGaBiTe 23:22, 18 September 2012 (PDT)

VMPI info dump

• VRAD is broken due to sending "oversized malformed packets"

Now that the vrad and vvis source code is publicly available we can see that this is not the case for the default distributor. VRAD is super heavy on the network, but the actual reason nobody got VRAD MPI working is because of an issue with the VMPI_ProcessLeafAmbient function. VRAD over MPI with the default distributor works in all games up until the ThreadComputeLeafAmbient step, with the problem seemingly being related to work units not being properly assigned to threads on the worker machine

EDIT: The SDK distributor is unfortunately broken in TF2. We managed to get it working in TF2 by creating all the necessary files/configuring everything to pass the if statement, and not only does VRAD fail to compile in SDK mode but VVIS as well. VRAD does not even print an error message and crashes immediately.

Functional VMPI VRAD in TF2

Building VisLeafs

Functional VMPI VRAD Worker in TF2

VVIS code line 974

VRAD code line 2757

Massive thanks to trigger_hurt (living legend) for diving into the vrad/vvis code to figure this out, we knew something was not right with the theory why VRAD crashes when I noticed the worker spews invalid launch argument errors, it does not just crash with no error.

• VMPI requires TCP and UDP ports

There are zero mentions to UDP in the source code for either vvis or vrad in relation to mpi, and plenty of hits for TCP. I have a working VMPI setup over the internet that only has TCP ports open. This potentially would allow for an IP tunneling service like ngrok to connect with worker machines without sending your real IP address.

EDIT: It seems like there is a UDP wrapper in iphelpers.h, however the point stands that the actual compile process is TCP only.

• Worker setup info

-mpi_Port is only used on the master, worker ports are formatted completely differently. This info was just straight up wrong.

• CSGO specfic arguments

-mpi_Local and -mpi_SDKMode are both valid launch params with TF2 tools, the latter will need to be patched similar to CSGO to work, untested yet.

• Performance testing section

This is a pretty wack VMPI setup, using a high core count low power cpu vps over the internet, VMPI setups distribute the workload evenly across workers, and works best over LAN. These performance testing results can be improved significantly.