Fixing VPK mounting for older Source SDK Bases

Background

When Valve rolled out the Steampipe update and the existing titles were updated to use the VPK file format, Source SDK Base and Source SDK Base were supplied with "depot" VPKs, but were left unable to properly mount them. Old mods using the Bases were broken, and 2007 Base mods weren't able to reference Half-Life 2: Episode Two content, unless it was directly copied into the mod's folder; this is despite the fact that the depot VPKs included those files (but were switched around; see below).

Apparently, this wasn't meant to happen, and both SDK Base 2006 and SDK Base 2007 can mount their VPKs with a specific fix.

Overview

The gmod9.com blog details an extensive research into this issue and provides a working solution.

It explains that the Steam2 wrapper, meant to handle VPK mounting for the old games, requires a missing file, steam_vpks.vdf, which is not shipped in any known product and whose structure needed to be reverse engineered.

Elaboration on the Steam2 wrapper 

The following text is copied from the original blog.

The Steam2 wrapper is a steam.dll replacement that acts kind-of as a proxy between the new post-Steampipe Steam and the game. It's the thing that handles the VPK mounting and makes the VGUI friends dialog still work. Note that the version of Steam2 wrapper inside the main Steam directory isn't the one that the old Source games use, instead they come with an old build of it dated Wed Feb 26 2014. The Steam2 wrapper uses an internal Valve library called vpklib to open VPK files and provide it to the game's virtual file system, the library is also used in newer Source Engine builds for the filesystem module and vpk.exe.

There are three known methods to fix it, starting from the easiest one:

Fixing with a custom steam_vpks.vdf

The file in question needs to list relative paths to the depot VPKs.

Create it in the Base's root (e. g. <steam_dir>/steamapps/common/Source SDK Base/ for '06, <steam_dir>/steamapps/common/Source SDK Base 2007/ for '07). You can use Notepad++ (or basic Windows' Notepad) to edit it.

The following example lists depot VPKs for Base 2006:

"steam_vpks"
{
	"spewfileio" "true"
	"mount"
	{
		"vpk" "vpks/depot_206"
		"vpk" "vpks/depot_207"
		"vpk" "vpks/depot_208"
		"vpk" "vpks/depot_212"
		"vpk" "vpks/depot_213"
		"vpk" "vpks/depot_215"
		"vpk" "vpks/depot_308"
		"vpk" "vpks/depot_309"
		"vpk" "vpks/depot_381"
		"vpk" "vpks/depot_421"
		"vpk" "vpks/depot_422"
		"vpk" "vpks/depot_423"
		// other vpks can be mounted here as well
	}
}

• The "vpk" keys point at files relative to the Base's root folder (as you can see the /vpks/ folder right inside the root).
• The spewfileio parameter defines if additional debug info is being spewed; usually you can leave it at false.

This file allows the Base to load its depot VPKs and, in turn, allows a mod for that Base to properly initiate and reference content. This works without needing to change the mod's gameinfo.txt or moving it to another Base.

Note:Keep in mind that steam_vpks.vdf disables the automatic dependency enumeration and mounting by taking a different code path, so you have to include the base VPKs in the steam_vpks.vdf file.

Todo: Elaborate. Is just listing all the depots enough?

Note:You can use steam_vpks.vdf to manually specify any number of additional VPKs inside it (within vpklib's maximum), so long as you use a path respective of the Base's install path. Without this file, Steam.dll automatically determines what IDs to mount, then finds the respective depot VPK. At this point, it won't load any additional VPKs. With the file, the wrapper instead mounts whatever is listed in the file.

Testing

A simple way to check it is to try and load an old mod for this Base without the file present (or renamed to anything other than steam_vpks.vdf). The mod should crash and report being Unable to load manifest file (typically scripts/surfaceproperties_manifest.txt or something similar); another frequent error is Can't find background image materials/console/startup_loading.vtf.

Now place the file you created, steam_vpks.vdf, next to the Base's hl2.exe . The mod should now properly initiate.

Fixing with mount_vpk function

A relatively simple alternative in your mod's code. Calling mount_vpk with the path to a depot, relative to the Base's main directory, should get the target VPK at that path mounted.

Todo: Elaborate. Is it the same relative paths as in steam_vpks.vdf? And where is the function supposed to be called?

Fixing with SteamMountFilesystem

According to the blog, this isn't the best way, as it requires using specific paths to the depots -

<game_path>/vpks/depot_<depot_id>_dir.vpk

Nevertheless, calling this function in the same manner as filesystem_steam.dll does it, should work.

Additional caveats

Difference in paths

The Bases' depot VPKs follow a slightly different filing structure compared to all others. Their tree looks like <root>/<game name>/<content folders> (materials, models, sound, etc), whereas normal VPKs look like <root>/<content folders>.

Thus, even with the fix, they won't properly mount "traditional" VPKs shipped with the games - the resulting path to files would be treated as incorrect due to a 'missing' folder in it.

It also means that while you can pack the mod's content into a VPK and reference that VPK in steam_vpks.vdf, you have to add a 'fake' game directory into the path:

root\fake_game_dir\maps
root\fake_game_dir\models
root_fake_game_dir\materials
etc...

and reference that folder in the mod's gameinfo.txt's SearchPaths:

Game  fake_game_dir

Misplaced VPKs

To make matters worse, SDK Base 2007 does not include files from Episode Two. Those VPKs - depot_421, depot_422, depot 423 - are all shipped with SDK Base 2006 by mistake, even though 2006 is supposed to "reflect" Episode One.

A crude, but working solution requires having both SDK Base 2006 and 2007 installed at the same directory (same /<steam_dir>/steamapps/common/) and referencing 2006's depots by their relative paths, i. e.:

"vpk" "./../Source SDK Base/vpks/depot_421"
"vpk" "./../Source SDK Base/vpks/depot_422"
"vpk" "./../Source SDK Base/vpks/depot_423"

Suspected corruption of the VPKs

A sidenote about the VPK files coming with the game and the vpk(lib) version used 

The following text is copied from the original blog.

I deem the VPK files and the vpklib version built into the steam.dll a very specific one. The VPK files coming with all Valve games and licensed games that utilize this mounting system are special, specifically it's pretty much a valid VPK2 file except for the checksums at the end. The tree and chunk checksums have been replaced by something that looks like corruption, which might've been caused by the checksums being uninitialized memory aligned next to each other. Here's how it looks like in the depot_215_dir.vpk file from Source SDK Base 2006. This pattern can also be seen in the depot_307_dir.vpk file from Source SDK Base 2007.

    {
        "PersonaName"       "martin

Here is another pattern, this one is found in depot_309_dir.vpk and depot_308_dir.vpk from Source SDK Base 2007.

    {
        "0"     "0100000065a3e27eb

As to what the actual cause of the corruption is, I can only speculate. The VPK files might've been made by some internal vpk.exe build, which licensees would also have, because this can be observed in both Valve and licensed games as I have stated before.

Base 2007 doesn't draw menus

Neither Base 2007 itself nor mods on it will display the main menu, console panel, HUD or other interfaces. They appear as completely blank and have to be navigated 'blindly' (main menu buttons work and produce button sounds, but you can't see anything; the console accepts can input).

The cause of it appears to come from mounting depot_317'.vpk. The reason for it is unknown; when unpacked, its contents don't appear to be causing the same issues (if any). It may be some kind of a corruption.

Fix:Excluding depot_317.vpk from the listing appears to be enough.

Credits

All info on researching and documenting this is taken from the gmod9.com blog.

The original credits include:

mv - for providing BGT and The Ship VPK files.

Nafrayu - for trying to figure this out with me long ago
DarkOK - for reviewing this

… and everyone else that I've forgotten.

Thanks to Jai "Choccy" Fox for helping me with the article and the subject.