Difference between revisions of "Using Source Control with the Source SDK"

From Valve Developer Community
Jump to: navigation, search
(Overview)
(Overview)
Line 1: Line 1:
 
[[Category:Programming]]
 
[[Category:Programming]]
 
===Overview===
 
===Overview===
 
test
 
  
 
This document describes how to use source control with the Source SDK. By using a source control system with the SDK, you will have the ability to easily integrate the latest optimizations and fixes from Valve into your mod's code. In addition, you get all the benefits of source control like revision history, backups, multiuser support, branches, and labels.  
 
This document describes how to use source control with the Source SDK. By using a source control system with the SDK, you will have the ability to easily integrate the latest optimizations and fixes from Valve into your mod's code. In addition, you get all the benefits of source control like revision history, backups, multiuser support, branches, and labels.  

Revision as of 19:35, 28 June 2005

Overview

This document describes how to use source control with the Source SDK. By using a source control system with the SDK, you will have the ability to easily integrate the latest optimizations and fixes from Valve into your mod's code. In addition, you get all the benefits of source control like revision history, backups, multiuser support, branches, and labels.

Note: This document refers to Perforce as the source control system because that is what Valve uses internally for all of its source (and content). There is a free 2-user version of Perforce that can be downloaded on the Perforce web site. Most other source control systems (like CVS) should be able to support the process that we describe here.

The high-level steps that you want to use to keep your code in sync with updates from Valve are.

  1. Create a codeline (codeline A) that contains the latest code from Valve.
  2. Create a codeline (codeline B) that contains your modified version of the code - you work out of this codeline.
  3. Integrate the latest changes from Valve into your code.

Source control flowchart.gif

Integrating Source Code Changes

The high-level steps to integrate the latest source changes from Valve into your existing source code are:

  1. Open all files in Codeline A for edit, copy all the latest Valve source into it, and revert unchanged files.
  2. Submit all the changes in Codeline A.
  3. Submit any pending changes in Codeline B.
  4. Integrate from Codeline A to Codeline B. At this point, any changes that happened to Valve's source code since the latest time codeline A was integrated into Codeline B will be integrated into your source code.
  5. Check the differences in Codeline B to make sure you want the changes in your code.
  6. Submit the change. At this point, you are up to date with the latest code from Valve.

Step By Step

This section walks through the entire process of setting up your mod to use source control.


Prerequisites


If you already have installed the SDK source code and modified it, and you want to use the process described in this document, then you must follow the steps in this section before proceeding.

  1. First, backup the source code that you have.
  2. Double check that you just backed up your source code.
  3. Next, you must get yourself in sync with the latest version of Valve's SDK code. This must be done manually by clicking Create a Mod from the SDK launcher, installing the source code somewhere, and merging your own code into it by hand or vice-versa. Tools like Beyond Compare can make this process easier, and this should be the last time you ever have to do this process painfully.


Download and install Perforce

To download Perforce, go to www.perforce.com, go to the downloads page, and get the most recent version. Note that you want to download the Core Perforce Windows Installer because that includes the Perforce server as well as the front end.

  1. Run the installer, and choose "Administrator - Typical" as the type of install. You can install to any directory, but this document will assume you've installed it into C:\Perforce. Click Next.
  2. Accept the default options at the next screen and click Next.
  3. Click Install.
  4. Perforce may mention that installation won't be complete until the system is restarted. If it mentions this, restart your system before proceeding.


Creating Your First Client Workspace

In this section, you will create a client workspace that will hold Valve's source code only.

Briefly, a client workspace describes where to place a codeline on a user's hard drive. It is similar to mapping a network drive - Perforce has its own internal file system that has a root called //depot (and you're going to place Valve's source code in //depot/ValveSDKCode), and the client workspace is what tells Perforce to store the contents of //depot/ValveSDKCode on your hard drive in C:\ValveSDKCode.

For example, as described above, you will have a codeline called "Codeline A", which will hold Valve's latest source code. This codeline exists inside Perforce and cannot be edited directly. To edit it, each user must have a copy of the codeline on his or her hard drive. They edit that copy, then Submit their changes to Perforce, in effect saying, "Take the source code on my hard drive and put it up on the Perforce server so other users can get my changes".

For more details on client workspaces, visit the Perforce documentation here.

  • The first time you run Perforce, it will bring up the Client Workspace Wizard dialog. We need to enter some advanced options, so click Cancel.
  • Select the ClientSpec menu and choose New.
  • At the next dialog, enter ValveSDKCode and click OK.

Source control clientspec name.gif

  • At the next dialog, make these changes (and leave the other fields as they are):
  • Set Root to C:\ValveSDKCode (or wherever you want to put Valve's source code on your machine - you will never edit the code in here directly).
  • Add ValveSDKCode after the //depot in the View section, so it reads:

//depot/ValveSDKCode/... //ValveSDKCode/...

  • Click Update.
  • A dialog will come up asking if you want to Sync To Head (i.e. copy the latest files from the codeline to your hard drive). Since there are no files in the codeline yet, click Dont Sync.

Source control clientspec.gif

Populating Codeline A

In the previous step, we created a client workspace to hold Valve's source code. It is still empty - we haven't added any files to the Perforce server, so now we want to add all of Valve's source code to the codeline (in //depot/ValveSDKCode).

  • Run the Source SDK and choose Create a Mod.
  • Choose the Source code only option and click Next.

Source control create mod.gif

  • Enter in the directory name you chose above to hold the Valve code (C:\ValveSDKCode).
  • Click Next.
  • All the latest source code will be copied into C:\ValveSDKCode.

Source control mod info.gif

  • Now, to add all the files, open a command prompt and cd to C:\ValveSDKCode.
  • To tell Perforce which client workspace you're adding the files to, type:

set p4client=ValveSDKCode

  • Then type the following command to add all the files to the depot:

for /R %i in (*.*) do @p4 add "%i"

  • You will see Perforce adding all the files. It will be scrolling lines like this:

//depot/ValveSDKCode/dlls/nav_generate.cpp#1 - opened for add

Source control command prompt.gif

  • At this point, we've told Perforce that we want to add all the Valve SDK code to Codeline A. All that remains is to submit the change to the Perforce server.
  • Alt+TAB back to Perforce and hit F5. Under Pending Perforce Changelists, you should see an item called Default.
  • Right-click on the Default item and choose Submit.

Source control perforce submit.gif

  • In the dialog that comes up, the only thing you need to change is the Description field.
  • Fill in the Descrpition field as shown in the image below, and click the Submit button.

Source control perforce submit2.gif


To recap: At this point, we have created Codeline A and populated it with the most latest version of Valve's SDK code. From now on, the only time Codeline A will ever be modified is when you want to integrate a newer version of Valve's SDK code into your mod's codebase.

We need two more things to complete the source control setup:

  1. We need a codeline for your mod's source code to live in (Codeline B).
  2. We need Perforce to know that Codeline A and Codeline B are related, so when you update Codeline A with Valve's latest SDK code, Perforce will do all the dirty work of merging the changes into your mod's code (in Codeline B).

Populating Codeline B

Populating Codeline B is going to be similar to populating Codeline A, with a few key differences. The main difference is that we're going to branch the code in Codeline A into Codeline B. This means we're making a copy of Codeline A, but Perforce is going to remember that the files in the two codelines are linked, so it can easily merge changes between the two copies of the files (which is the main reason we want to use source control in the first place).

  • In Perforce, click the ClientSpec menu and choose New.
  • Enter LocalModCode in the dialog and click OK.

Source control clientspec name local mod.gif

  • At the next dialog, make these changes (and leave the other fields as they are):
  1. Set Root to C:\LocalModCode. Important: This must be a directory that does not currently exist on your hard drive. If you want to store and edit your source code somewhere else (and you probably will), you can change this later - we'll show you how.
  2. Add LocalModCode after the //depot in the View section, so it reads:

//depot/LocalModCode/... //LocalModCode/...

  • Click Update.


  • A dialog will come up asking if you want to Sync To Head (i.e. copy the latest files from the codeline to your hard drive). Since there are no files in the codeline yet, click Dont Sync.

Source control clientspec local mod.gif

  • Click on the BranchSpec menu and choose New.
  • At the next dialog, make these changes (and leave the other fields as they are):
  1. Set Branch to valve_to_mod.
  2. Change the View section to read like this:

//depot/ValveSDKCode/... //depot/LocalModCode/... </blockquote>

  • Click Update.
  • We just told Perforce that we want to map the files from Codeline A (//depot/ValveSDKCode) into Codeline B (//depot/LocalModCode). All we have to do now is branch the files over.

Source control branchspec.gif

  • Hit F10 to flip back to the ClientSpec view.
  • On the left-hand side of the Perforce window, you should see a tree control with //depot at the root. Right-click on //depot, and choose Integrate Using, then choose Branchspec in the popup menu.

Source control integrate branchspec.gif

  • The valve_to_mod BranchSpec that we just created should be selected.
  • Click Next.

Source control perforce branches.gif

  • In this dialog, the only thing you need to change is the radio buttons next to the File Specifications - Integrate area in the middle.
  • Check the To button, as shown in the image below.
  • Click Finish.
  • Perforce will spit out a lot of messages like this:

//depot/LocalModCode/dlls/particle_light.h#1 - branch/sync from //depot/ValveSDKCode/dlls/particle_light.h#1

  • When Perforce is done branching all the files from the previous step, you can proceed to the next step.

Source control final integrate.gif

  • At this point, Perforce has a big list of all the files it wants to branch from Codeline A into Codeline B. All that remains is to Submit the change.
  • Hit F6 to bring up the Change List pane in Perforce.
  • In the right-hand panel, there will be a tree control that says Pending Changelists, and right under there, Default.
  • Right-click on Default, and choose Submit.

Source control about to submit branched code.gif

  • In this dialog, enter anything you want into the Description field and click Submit.

Source control submit branched code.gif

At this point, you have Codeline A and Codeline B setup on the Perforce server.

Copying Your Code Into Codeline B

If, before embarking on this document, you already had source code for a mod you were working on, you should follow the instructions in this section. Otherwise (i.e. if you are starting a new set of mod source code right now), you should skip over this section and start reading at "Moving Codeline B's Directory" below.

  • In Perforce, hit F10 to go to your ClientSpecs list.
  • If the LocalModCode ClientSpec doesn't have a green arrow next to it (i.e. it is not the active ClientSpec), right-click on it and choose Switch to LocalModCode.

Source control switch to local mod code.gif

  • On the left-hand side, right-click on //depot and choose Open For Edit.
  • Answer Yes to the dialog asking if you want to proceed with the edit.

Source control open for edit.gif

  • Now, either in Windows explorer or in a command prompt, copy the source tree from your mod into C:\LocalModCode. For example, if you installed your mod into C:\MyMod, the command line would be:

xcopy C:\MyMod\src C:\LocalModCode /e

  • We also need to make sure that any extra files files in your mod's code are added to Perforce.
  • Open a command prompt and cd to C:\LocalModCode.
  • To tell Perforce which client workspace you're adding the files to, type:

set p4client=LocalModCode

  • Then type the following command to add all the files to the depot:

for /R %i in (*.*) do @p4 add "%i"

  • Perforce will be scrolling lines like this:

//depot/LocalModCode/vgui2/controls/controls.cpp - can't add existing file

  • This just means that Perforce already has most of the files on its server, so now we'll check to see if there were any new ones.


Source control add files to localmodcode.gif

  • Hit F6 to bring up the 'Change List' pane in Perforce.
  • In the right-hand panel, there will be a tree control that says Pending Changelists, and right under there, Default.
  • Right-click on Default, and choose Revert Unchanged Files
  • Perforce will work for a few minutes gathering the file list.

Source control revert unchanged.gif

  • When this dialog appears listing the files to Revert, click Revert Files.

Source control revert unchanged final.gif

  • Hit F6 to bring up the 'Change List' area in Perforce.
  • In the right-hand pane, there will be a tree control that says Pending Changelists, and right under there, Default.
  • Right-click on Default, and choose Submit

Source control about to submit branched code.gif

  • In this dialog, enter anything you want into the Description field, then click Submit.

Source control submit branched code.gif


Moving Codeline B's Directory

At this point, the Perforce server has your mod's latest code in Codeline B. The only problem now is that you've got Codeline B in C:\LocalModCode, and you may want it elsewhere (like in C:\MyMod\src if you installed your mod into C:\MyMod). To move where Codeline B is stored on your hard drive, follow the steps below.

First, if you want Codeline B to sit in the same directory where you had your source code previously (if you were already working on a mod), then go rename that directory to something else for now. Note: You should have already copied your source code into Codeline B in the "Copying Your Code Into Codeline B" section, so we're just going to tell Perforce that we want Codeline B's code in the directory you copied from.

For example, if you already were working on a mod, and its source code was in C:\MyMod\src, then in the "Copying Your Code Into Codeline B" section, you would have copied from C:\MyMod\src into C:\LocalModCode. Now, you would rename the C:\MyMod\src directory to something else, and we'll tell Perforce to put Codeline B's source code in C:\MyMod\src.

  • In Perforce, hit F10 to go to your ClientSpecs list.
  • If the LocalModCode ClientSpec doesn't have a green arrow next to it (i.e. it is not the active ClientSpec), right-click on it and choose Switch to LocalModCode.

Source control switch to local mod code.gif

  • Right-click on LocalModCode and choose the Edit LocalModCode option from the popup menu.

Source control edit local mod spec.gif

  • In this dialog, edit the Root field to point to where you want Codeline B's source code on your hard drive. Note: C:\MyMod\src is just an example. What you enter here depends on where you want to work on your mod.

Source control change local mod dir.gif

  • Click Yes in this dialog.

Source control confirm change root.gif

  • Now we need to have Perforce copy the latest source code for Codeline B back into the directory you just specified, so you can edit it.
  • In the left-hand pane, there should be a tree control with //depot at the root.
  • Right-click on it, go to Sync, then select the Force sync to Head Revision option from the sub-menu.
  • Perforce will now copy all the source code into the directory you specified earlier.

Source control force sync.gif


Editing Your Source Code In Perforce

At this point, you should be all set to continue with development of your mod. The main thing to be aware of is that Perforce needs to know about any changes to the source code you are making. What you do is build a changelist of the files that you've edited, and you Submit the changelist to Perforce. For a complete description of how to use Perforce, see the Perforce Documentation.


Editing Files

  • In Perforce, hit F10 to go to your ClientSpecs list.
  • If the LocalModCode ClientSpec doesn't have a green arrow next to it (i.e. it is not the active ClientSpec),

right-click on it and choose Switch to LocalModCode.

Source control switch to local mod code.gif

  • On the left-hand side, there should be a tree control with //depot at the root.
  • Navigate the tree control by clicking the "+" next to folders, to the file you want to edit. Then right-click the file and choose Open for Edit.
  • Now you can edit the file on your local hard drive (but don't forget to submit the changelist when you are done).

Source control open file for edit.gif


Adding Files

This section describes how to add a new source code file to Perforce.

  • In Perforce, hit F10 to go to your ClientSpecs list.
  • If the LocalModCode ClientSpec doesn't have a green arrow next to it (i.e. it is not the active ClientSpec), right-click on it and choose Switch to LocalModCode.

Source control switch to local mod code.gif

  • Go to the File menu and choose Add to Source Control.
  • Find the file you want to add and double-click on it.
  • At the following dialog, make sure that the files listed are the files you want to add, then click the Add Files button.

Source control add file.gif


Submitting Changelists

As you add and edit files, they sit in a temporary area waiting to be Submitted to Perforce. The nice thing about this is that you can revert your changes at any time, in case you decide to back up and start over on them. Once you have a set of changes that you do want to commit to Perforce (which will give you the ability to restore that version and to make it available to other users), follow the steps below to submit the change.

  • In Perforce, hit F10 to switch the right pane to your ClientSpecs list.
  • If the LocalModCode ClientSpec doesn't have a green arrow next to it (ie: it is not the active ClientSpec), right-click on it and choose Switch to LocalModCode.

Source control switch to local mod code.gif

  • Hit F6 to bring up the Change List pane in Perforce.
  • In the right-hand pane, there will be a tree control that says Pending Changelists, and right under there, Default.
  • Right-click on Default, and choose Submit.

Source control about to submit branched code.gif

  • In this dialog, enter anything you want into the Description field, and then click Submit.

Source control submit branched code.gif


Integrating SDK Code Into Your Mod

This section assumes you've setup the codelines as described above. When new code from Valve comes out, you can use the steps here to merge it into your code. First, make sure that you have submitted any changelists in your LocalModCode client workspace.

  • In Perforce, hit F10 to go to your ClientSpecs list.
  • If the ValveSDKCode ClientSpec doesn't have a green arrow next to it (ie: it is not the active ClientSpec), right-click on it and choose Switch to ValveSDKCode.

Source control switch to valvesdkcode.gif

  • On the left-hand side, right-click on //depot and choose Open For Edit.
  • Answer Yes to the dialog asking if you want to proceed with the edit.

Source control open for edit.gif

  • Run the Source SDK and choose Create a Mod.
  • Choose the Source code only option and click Next.

Source control create mod.gif

  • Enter in the directory name you chose above to hold the Valve code (C:\ValveSDKCode).
  • Click Next.
  • All the latest source code will be copied into the directory (C:\ValveSDKCode, in this example).

Source control mod info.gif

  • We don't want to merge in the .vcproj files, because they are difficult to merge. It's easier to just add and remove any changes in your LocalModCode codeline.
  • Open a command prompt and cd into the C:\ValveSDKCode directory.
  • Next type:

set p4client=ValveSDKCode

  • And then:

for /R %i in (*.vcproj) do @p4 revert "%i"

Source control revert vcproj.gif

  • Hit F6 to bring up the 'Change List' area in Perforce.
  • In the right-hand pane, there will be a tree control that says Pending Changelists, and right under there, Default.
  • Right-click on Default, and choose Revert Unchanged Files.
  • Perforce will work for a few minutes.

Source control revert unchanged.gif

  • When this dialog appears, click Revert Files.

Source control revert unchanged final.gif

  • In the right-hand panel, there will be a tree control that says Pending Changelists, and right under there, Default.
  • Right-click on Default, and choose Submit.

Source control about to submit branched code.gif

  • In this dialog, enter anything you want into the Description field and click Submit.
  • At this point, Perforce now has the latest Valve code in Codeline A (ValveSDKCode). What remains to be done is to tell Perforce to integrate the changes that just happened in Codeline A into Codeline B. Perforce will automatically detect which files changed, and it will move those changes into your MOD's code.

Source control submit branched code.gif

  • In Perforce, hit F10 to go to your ClientSpecs list.
  • If the LocalModCode ClientSpec doesn't have a green arrow next to it (i.e. it is not the active ClientSpec), right-click on it and choose Switch to LocalModCode.

Source control switch to local mod code.gif

  • On the left-hand pane of the Perforce window, you should see a tree control with //depot at the root. Right-click on //depot, and choose Integrate Using, then choose Branchspec in the sub-menu.

Source control integrate branchspec.gif

  • The valve_to_mod BranchSpec that we just created should be selected.
  • Click Next.

Source control perforce branches.gif

  • In this dialog, the only thing you need to change is the radio buttons next to the File Specifications - Integrate area in the middle.
  • Select the To button, as shown in the image to the right.
  • Click Finish.

Source control final integrate.gif

  • Hit F6 to bring up the changelists pane on the right.
  • There should be a tree control that says, Pending Changelists (client 'LocalModCode') with a bunch of files in the tree (when you expand the tree).
  • Right-click on any file and choose Resolve, then Auto resolve... from the sub-menu.

Source control auto resolve.gif

  • In this dialog, the only thing you need to change is to make sure the All open files requiring resolution option is checked.
  • Then click the OK button.

Source control auto resolve dialog.gif

  • Now, look over all the files in the tree control, and if there are any with a yellow exclamation point next to them, it means that they couldn't be automatically resolved by Perforce (see example in this image). This means that changes you have made in your LocalModCode codeline are conflicting with changes Valve has made, and you will have to resolve them manually. If there are no yellow exclamation points, then you can skip to the last step of this section where you submit the change.

Source control yellow exclamation.gif

  • First, if there are any library (.LIB) files with yellow exclamation marks, you want to use Valve's version of the .LIB file.
  • Select all the .LIB files with yellow exclamation marks.
  • Right-click, and choose Resolve, then Auto Resolve from the sub-menu.

Source control resolve libs.gif

  • In the dialog that comes up, make sure that the Accept Theirs option is selected.
  • Click the OK button.

Source control accept theirs.gif

  • For all other files (ones that aren't .LIB files), you must resolve them manually.
  • For each file, right-click on it and choose Resolve, then Interactively from the sub-menu.
  • Perforce will bring up a 3-way merge application that lets you merge Valve's changes into your code.

Source control resolve interactively.gif

  • Click on the Run Merge Utility button on this dialog.

Source control resolve interactively dialog.gif

  • Here is a brief description of how to use Perforce's default 3-way merge utility.
  • The top-left window pane contains Valve's version of the file.
  • The top-right window pane has your version of the file.
  • The bottom window pane has the merged version of the file.
  • Use the green button in the toolbar on the top to go to each difference.
  • At each difference, pick which of the top windows you want to keep the source from, and double-click on the section of code that you want to move into the bottom window.
  • You can also right-click in the bottom window to edit the source code directly.

Source control interactive merge window.gif

  • Now, finally, you can submit the merged version.
  • In the right-hand panel, there will be a tree control that says Pending Changelists, and right under there, Default.
  • Right-click on Default, and choose Submit.

Source control about to submit branched code.gif

  • In this dialog, enter anything you want into the Description field and click Submit.

Source control submit branched code.gif

At this point, you should do a fixup pass on the code to fix any compile errors that may have been introduced by the code merge. You could also do this pass right before submitting the merged code changelist, but it tends to be easier to make changes starting with an empty changelist rather than one full of code merges.


Further Investigations

We highly recommend that you read the Perforce Beginner's FAQ and look at the Perforce Documentation if necessary, and get acquainted with:

  • Codelines
  • Client workspaces
  • Changelists
  • Branches
  • Integrations


Working with a source control introduces some concepts that take getting used to, but it pays dividends. It saves time, helps teams work together on the same product, and provides access to the whole history of a project.

Keep in mind that, while this document has a lot of steps, this stuff is all trivial once you're familiar with Perforce. Once you know how this all works, all these integrations and merges can be done in a matter of minutes. We recommend that you jump back to the "overview" section of this document to review the high-level process that is involved, and see how all the detailed steps fit into that simple high-level process.

Internally, Valve uses source control in a similar fashion to that described in this document. The main difference is that ALL files for Valve games go in Perforce, including content source .TGA artwork, .VMT files, executables, scripts, documents, etc. We recommend doing this while developing a mod as well, because you can instantly restore any version of the mod that ever existed.

Valve also uses a multitude of codelines (whereas there are just two in this document). For example, we have a main codeline that about 20 programmers submit code into. Oftentimes, programmers need to stay sheltered from other people's changes (which may happen as many as 20 times per day), so each programmer has their own set of personal codelines that are branched off the main codeline. This way, they can choose when they want to integrate all the changes from other programmers into their personal codeline (just like you now have the choice about when you want to integrate Valve's source code into your mod's source code).

Happy coding!