Fixing VPK mounting for older Source SDK Bases

Background

When Valve rolled out the Steampipe update and the existing titles were updated to use the VPK file format, Source 2006 and Source 2007 were supplied with "depot" VPKs, but were left unable to properly mount them. This meant that old mods using the Bases were broken, and that mods using 2007 Base weren't able to reference Half-Life 2: Episode Two content, unless it was directly copied into the mod's folder; this is despite the fact that the "depot" VPKs included those files (but were switched around; see below).

Apparently, this wasn't meant to happen, and both SDK Base 2006 and SDK Base 2007 can mount their VPKs with a specific fix.

Overview

The gmod9.com blog details an extensive research into this issue and provides a working solution.

It explains that the Steam2 wrapper, meant to handle VPK mounting for the old games, requires a missing file, steam_vpks.vdf, which is not shipped in any known product and whose structure needed to be reverse engineered.

Elaboration on the Steam2 wrapper 

The following text is copied from the original blog.

The Steam2 wrapper is a steam.dll replacement that acts kind-of as a proxy between the new post-Steampipe Steam and the game. It's the thing that handles the VPK mounting and makes the VGUI friends dialog still work. Note that the version of Steam2 wrapper inside the main Steam directory isn't the one that the old Source games use, instead they come with an old build of it dated Wed Feb 26 2014. The Steam2 wrapper uses an internal Valve library called vpklib to open VPK files and provide it to the game's virtual file system, the library is also used in newer Source Engine builds for the filesystem module and vpk.exe.

There are three known methods to fix it, starting from the easiest one:

Fixing with a custom steam_vpks.vdf

The file in question needs to list relative paths to the depot VPKs. The spewfileio parameter defines if additional debug info is being spewed; usually you can leave it at false.

The following example lists depot VPKs for Base 2006:

"steam_vpks"
{
	"spewfileio" "true"
	"mount"
	{
		"vpk" "vpks/depot_206"
		"vpk" "vpks/depot_207"
		"vpk" "vpks/depot_208"
		"vpk" "vpks/depot_212"
		"vpk" "vpks/depot_213"
		"vpk" "vpks/depot_215"
		"vpk" "vpks/depot_308"
		"vpk" "vpks/depot_309"
		"vpk" "vpks/depot_381"
		"vpk" "vpks/depot_421"
		"vpk" "vpks/depot_422"
		"vpk" "vpks/depot_423"
		// other vpks can be mounted here as well
	}
}

When placed in the relevant SDK Base's root folder (next to hl2.exe), it allows the Base to load its depot VPKs and, in turn, allows a mod for that Base to properly initiate and reference content. This works without needing to change the mod's gameinfo.txt or moving it to another Base.

A simple way to check it is to try and load an old mod for this Base without the file present (or renamed to anything other than steam_vpks.vdf). The mod should crash and report being Unable to load manifest file (typically scripts/surfaceproperties_manifest.txt or something similar); another frequent error is Can't find background image materials/console/startup_loading.vtf.

Now place the file, steam_vpks.vdf, next to the Base's hl2.exe. The mod should properly initiate.

Keep in mind that steam_vpks.vdf disables the automatic dependency enumeration and mounting by taking a different code path, so you have to include the base VPKs in the steam_vpks.vdf file.

Todo: Elaborate. Does just listing all the depots is enough?

Fixing with mount_vpk function

A relatively simple alternative. Calling mount_vpk with the path to a depot, relative to the Base's main directory, should get the target VPK at that path mounted.

Fixing with SteamMountFilesystem

According to the blog, this isn't the best way, as it requires using specific paths to the depots -

<game_path>/vpks/depot_<depot_id>_dir.vpk

Nevertheless, calling this function in the same manner as filesystem_steam.dll does it, should work.

Additional caveats

The Bases' depot VPKs follow a slightly different filing structure compared to all others. Their tree looks like root\<game name>\<content folders> (materials, models, sound, etc), whereas normal VPKs look like root\<content folders>.

This means that even with the fix, they won't properly mount "traditional" VPKs shipped with the games - the resulting path to files would be treated as incorrect due to a 'missing' folder in it.

It also means that while you can pack the mod's content into a VPK and reference that VPK in steam_vpks.vdf, you have to add a 'fake' game directory into the path:

root\fake_game_dir\maps
root\fake_game_dir\models
root_fake_game_dir\materials
etc...

and referencing that folder in the mod's gameinfo.txt's SearchPaths:

Game  fake_game_dir

To make matters worse, SDK Base 2007 does not include files from Episode Two. Those VPKs - depot_421, depot_422, depot 423 - are all shipped with SDK Base 2006 by mistake, even though 2006 is supposed to "reflect" Episode One.

A crude, but working solution requires having both SDK Base 2006 and 2007 installed at the same directory (same \Steam\steamapps\common\) and referencing 2006's depots by their relative paths, i. e.:

"vpk" "./../Source SDK Base/vpks/depot_421"
"vpk" "./../Source SDK Base/vpks/depot_422"
"vpk" "./../Source SDK Base/vpks/depot_423"

A sidenote about the VPK files coming with the game and the vpk(lib) version used 

The following text is copied from the original blog.

I deem the VPK files and the vpklib version built into the steam.dll a very specific one. The VPK files coming with all Valve games and licensed games that utilize this mounting system are special, specifically it's pretty much a valid VPK2 file except for the checksums at the end. The tree and chunk checksums have been replaced by something that looks like corruption, which might've been caused by the checksums being uninitialized memory aligned next to each other. Here's how it looks like in the depot_215_dir.vpk file from Source SDK Base 2006. This pattern can also be seen in the depot_307_dir.vpk file from Source SDK Base 2007.

    {
        "PersonaName"       "martin

Here is another pattern, this one is found in depot_309_dir.vpk and depot_308_dir.vpk from Source SDK Base 2007.

    {
        "0"     "0100000065a3e27eb

As to what the actual cause of the corruption is, I can only speculate. The VPK files might've been made by some internal vpk.exe build, which licensees would also have, because this can be observed in both Valve and licensed games as I have stated before.

Credits

All info on researching and documenting this is taken from the gmod9.com blog.

The original credits include:

mv - for providing BGT and The Ship VPK files.

Nafrayu - for trying to figure this out with me long ago
DarkOK - for reviewing this

… and everyone else that I've forgotten.