View source for VPK (file format)

{{LanguageBar}}

'''VPK''' ("'''V'''alve '''P'''ac'''K'''") is a package format used by post-[[GCF]] {{source|4}} engine games to store content, as well as in {{Source2|4}}.

{{confusion|{{vtmb|2}} uses a different package format with the same file extension.
{{todo|Document "Vampire PacK" files on their own page.}} }}

== Conception ==

Prior to {{l4d|4}} and {{src13|4}}, aswell as prior to backporting VPK to older Source engine branch after [[SteamPipe]] update, typical Source engine games stored their content in [[.gcf]] files.  Executable files, modifiable files (e.g. configuration files) and custom content were copied and stored locally on the user's hard drive.  Possibly brought on by [https://web.archive.org/web/20170915143435/http://nemesis.thewavelength.net/index.php?c=216 poor performance], the [[NCF]] file format was introduced and all game content was copied entirely to the hard drive.  This, however, introduced a new problem.  Source engine materials and models are stored in thousands of small files and it would be expensive to continuously open and close these files.  The solution was the conception of the VPK file format which is used to store Left 4 Dead materials, models and particles in a handful of files which can be quickly accessed.

== Features ==

=== Preload Data ===

In order to efficiently access small or critical files, the beginning of each file can optionally be stored in the VPK directory.  In practice, this seems to be limited to the first 1000 bytes of Source engine materials ([[VMT]] files) which are typically only a few hundred bytes in size.

=== Multiple Archives ===

Previous Source engine games that had been distributed by the more advanced GCF file format had the luxury of internally fragmenting new and updated files.  This meant that new and updated files could be efficiently downloaded and saved with minimal bandwidth and disk IO.  Because the new VPK files are independent of distribution (Left 4 Dead is distributed by NCF files and Steam knows nothing of the VPK file format), their content is split up over multiple archives that seem to be limited to about 32 MB in size.  Because of this, when a file in a specific file is updated, only the VPK archive that contains the file needs to be updated.  Additionally, new files can be downloaded to their own individual archives.  This is why most of the newer archives are small in size; their contents are limited the the files added in a single update.

== Versions ==

; 1
: {{as|4}}
: {{dota2|4}} (original release)
: {{l4d|4}}
: {{l4d2|4}}
: {{portal2|4}}
: {{sfm|4}}
: {{p3|4}}
; 2
: {{csgo|4}}
: {{css|4}}
: {{dods|4}}
: {{gmod|4}}
: {{hls|4}}
: {{hl2|4}}
: {{hl2dm|4}}
: {{portal|4}}
: {{tf2|4}}
: Depot VPK files for {{src06|4}}, {{src07|4}} and {{src09|4}} games
: All {{src2|4}} games

; Third-Party (unknown version)
: {{Titanfall|4}}

== File Format ==

A VPK package is actually spread out over multiple files sharing the same extension.  The directory is stored in a specific file called <name>_dir.vpk and the content is spread over several additional archive files called <name>_*.vpk (where * is the zero based archive index).  Consequentially, there are two file formats:

=== Directory ===

==== Header ====

===== VPK 1 =====
Originally, the VPK file had no header or identifier.  This changed when the [http://store.steampowered.com/news/2620/ June 25, 2009 Left 4 Dead update] was released adding support for third party campaigns.  VPK directory files created after this date have the following header:

<source lang=cpp>struct VPKHeader_v1
{
	const unsigned int Signature = 0x55aa1234;
	const unsigned int Version = 1;

	// The size, in bytes, of the directory tree
	unsigned int TreeSize;
};
</source>

If the file data is stored in the same file as the directory, its offset is <code>(sizeof(VPKHeader_v1) + TreeSize)</code>.

===== VPK 2 =====

<source lang=cpp>struct VPKHeader_v2
{
	const unsigned int Signature = 0x55aa1234;
	const unsigned int Version = 2;

	// The size, in bytes, of the directory tree
	unsigned int TreeSize;

	// How many bytes of file content are stored in this VPK file (0 in CSGO)
	unsigned int FileDataSectionSize;

	// The size, in bytes, of the section containing MD5 checksums for external archive content
	unsigned int ArchiveMD5SectionSize;

	// The size, in bytes, of the section containing MD5 checksums for content in this file (should always be 48)
	unsigned int OtherMD5SectionSize;

	// The size, in bytes, of the section containing the public key and signature. This is either 0 (CSGO & The Ship) or 296 (HL2, HL2:DM, HL2:EP1, HL2:EP2, HL2:LC, TF2, DOD:S & CS:S)
	unsigned int SignatureSectionSize;
};
</source>

If the file data is stored in the same file as the directory, its offset is <code>(sizeof(VPKHeader_v2) + TreeSize)</code>.

==== Tree ====

The format of the directory tree is a little unorthodox.  It consists of a tree three levels deep that seems to be structured for file size.  The first level of the tree consists of file extensions (e.g. ''vmt'', ''vtf'' and ''mdl''), the second level consists of directory paths (e.g. ''materials/brick'', ''materials/decals/asphalt'' and ''models/infected''), and the third level consists of file names, file information and preload data.  Each tree node begins with a null terminated ASCII string and empty strings are used to signify the end of a parent node.  Pseudo-code to read the directory might look something like:

 ReadString(file)
 	string = ""
 	while true
 		char = ReadChar(file)
 		if char = null
 			return string
 		string = string + char

 ReadDirectory(file)
 	while true
 		extension = ReadString(file)
 		if extension = ""
 			break
 		while true
 			path = ReadString(file)
 			if path = ""
 				break
 			while true
 				filename = ReadString(file)
 				if filename = ""
 					break
 				ReadFileInformationAndPreloadData(file)

A nonexistent extension (in example/file), path (in example.txt), or filename is represented by a single space.

Immediately following the null terminator for the filename is the following structure:

<source lang=cpp>
struct VPKDirectoryEntry
{
	unsigned int CRC; // A 32bit CRC of the file's data.
	unsigned short PreloadBytes; // The number of bytes contained in the index file.

	// A zero based index of the archive this file's data is contained in.
	// If 0x7fff, the data follows the directory.
	unsigned short ArchiveIndex;

	// If ArchiveIndex is 0x7fff, the offset of the file data relative to the end of the directory (see the header for more details).
	// Otherwise, the offset of the data from the start of the specified archive.
	unsigned int EntryOffset;

	// If zero, the entire file is stored in the preload data.
	// Otherwise, the number of bytes stored starting at EntryOffset.
	unsigned int EntryLength;

	const unsigned short Terminator = 0xffff;
};
</source>

If a file contains preload data, the preload data immediately follows the above structure.  The entire size of a file is PreloadBytes + EntryLength.

==== Footer ====

VPK2 adds a footer section that contains extra CRC data for [[Pure Servers|pure mode]] "so the dedicated servers do not need to compute them at startup but can be checked with the command <code>sv_pure_checkvpk</code>".

==== VPK 2 Sections ====

===== File data =====
This can be read like a separate VPK archive embedded in the directory file (see EntryOffset comment in VPKDirectoryEntry).

===== Archive MD5 checksums =====
This section is an array of these:
<source lang=cpp>
struct VPK_ArchiveMD5SectionEntry
{
	unsigned int ArchiveIndex;
	unsigned int StartingOffset; // where to start reading bytes
	unsigned int Count; // how many bytes to check
	char MD5Checksum[16]; // expected checksum
}
</source>
Since <code>sizeof(VPK_MD5SectionEntry)</code> is 28, the section size must be a multiple of 28.

Valve's VPK tool refers to these as cache line hashes. They can be checked against the file content with the checkhash command.

===== Other MD5 checksums =====
<source lang=cpp>
struct VPK_OtherMD5Section
{
	char TreeChecksum[16];
	char ArchiveMD5SectionChecksum[16];
	char WholeFileChecksum[16];
}
</source>

===== Public Key & Signature =====
<source lang=cpp>
struct VPK_SignatureSection
{
	unsigned int PublicKeySize; // always seen as 160 (0xA0) bytes
	char PublicKey[PublicKeySize];

	unsigned int SignatureSize; // always seen as 128 (0x80) bytes
	char Signature[SignatureSize];
}
</source>

This section can be viewed with Valve's VPK tool's dumpsig command.

Games with this section (296 bytes):
: {{css|4}}
: {{dods|4}}
: {{hls|4}}
: {{hl2|4}}
: {{hl2dm|4}}
: {{hl2ep1|4}}
: {{hl2ep2|4}}
: {{hl2lc|4}}
: {{portal|4}}
: {{tf2|4}}

Games lacking this section:
: {{csgo|4}}
: {{ship|4}} (Multiplayer)
: Depot VPK files ({{src06|4}}, {{Src07|4}} and {{Src09|4}} games)

''All VPKs tested were from the English Language''

=== Archive ===

VPK Archives store raw file data.  They have no identifying header and know nothing of their contents.  Though not necessary, the raw file data is typically tightly packed.

Archives are named by the main VPK name with the archive number appended. (e.g <code>tf2_misc_'''050'''.vpk</code>).

{{todo|Rewrite and merge Notes section to the format section}}

=== Notes ===
{{cleanup}}
Valve apparently added skipping to the specifications. Someone found out this when trying to write their VPK parser in C#. This should be merged with the format area later, and reworded.

Valve uses nulls to signify if skipping is used. On a normal entry, it uses 2 nulls, and is followed by the format above. However, there are cases where there are only one, or no nulls at the start, and this means that some level of skipping is used.

If there are 2 nulls, no skipping is used, and the extension, path, and name are read as usual.
If there is 1 null, the extension is skipped (It's the same extension as the last read entry), and then the path and name are read as usual.
If there are no nulls, the extension and path are the same as the last entry (Skipped), and only the name is read.

This system has only been observed in VPK1.

== VPK readers ==
* {{vrf|4}} (v1/v2) — Can also create VPK files from a folder.
* {{vPKEdit|4}} (v1/v2) - Can also create new VPKs of any version, plus add & remove files.
* {{gcfscape|4}} (v1/v2)
* [https://blog.gib.me/2008/11/14/left4dead-vpk-extraction-tools-now-with-ui/ Rick's VPK Tool] (v1)
* [[VPK File Format/vpk2 reader.py]] (v2)
* [https://www.17buddies.net/17b2/Tools.html#RubHlb HLBox17b] (v2)
* [https://github.com/ata4/jvpk jvpk] (v1/v2)
* [https://dev.cra0kalo.com/?p=82 Titanfall VPK Tool] (v3)
* [https://github.com/ValvePython/vpk VPK Python module] (v1/v2)
* [https://pastebin.com/xGEhsMRu Common Lisp VPK Library] — Can only read VPK files.

==See also==
* [[L4D Campaign Add-on Tutorial]] (VPK authoring.)
* [[vpk.exe]]
* [[GCFScape]] (VPK file viewer.)
* [[HLLib]] (Example VPK source code.)
* [[Fixing VPK mounting for older Source SDK Bases]] (for {{src06|2}} & {{src07|2}})
* [[GCF]] - Package format which VPK replaces
* [[PAK]] - Package format used by {{idtech2|2}} and {{goldsrc|2}}

{{source topicon}}{{source 2 topicon}}<!--Use more specific if possible--><!-- more specific how?-->

[[Category:File formats|vpk]]