Head Tracking using the FaceAPI

This tutorial explains in detail how to incorporate the non-commercial version of the FaceAPI head-tracker (v3.2.6) into the Source 2007. Once completed, the reader will be able to use the offset (x, y,z) and orientation (pitch, yaw, roll) of the players head as a form of input.

It is worth noting that this tutorial does not cover facial feature tracking/integration, as this feature of the FaceAPI tracker is only available in the commercial version.

Setting up the Project

Install FaceAPI

Download and install the non-commercial version of the faceAPI head tracker. In doing this you will need to agree to Seeing Machine's non-commercial license, which includes the following points of interest:

1. Usage is prohibited in any commercial environment. If you are earning a salary, wages, commission, or some other in-kind payment for working with faceAPI, then you are a commercial user and are not entitled to download or use the non-commercial license.
2. You may only release applications built using the non-commercial edition of faceAPI if you also release the source-code for your application, including the faceAPI license agreement in unmodified form.
3. You acknowledge that the Non-Commercial edition of faceAPI comes with no technical support or warranties of any kind.

Two alternatives to using the non-commercial FaceAPI is to either purchase a FaceAPI licence or to use a different tracking program such as FreeTrack.

Create a Source Mod

Use the My First Mod instructions to create a Source 2007 single player mod. Do not create the project as a Source 2009 mod as the Source SDK Base 2009 has not been released yet.

Rearrange the Files

Some of the FaceAPI files need to be moved into the newly created Source mod directory. If FACEAPI represents the directory that the FaceAPI was installed to (typically C:\Program Files (x86)\SeeingMachines) and MODBASE represents your Source mod's install directory (typically C:\Program Files (x86)\Steam\steamapps\SourceMods\your_mod_name) you will need to move:

1. FACEAPI/FaceTrackingAPI_NC 3.2\API\include to MODBASE/src/game/client/faceapi/include
2. FACEAPI/FaceTrackingAPI_NC 3.2\API\bin to MODBASE/bin/
3. MODBASE/bin/smft32.exp, MODBASE/bin/smft32.lib and MODBASE/bin/smft32.ref to MODBASE/src/game/client/faceapi/lib

Add Some Extra Files

Create the files face_api.cpp, face_api.h, lock.h and mutex.h with the content listed below and add them to your MODBASE/src/game/client/faceapi/ folder.

face_api.h:

#ifndef FACEAPI_H
#define FACEAPI_H

#include "sm_api.h"

struct FaceAPIData
{
	float	h_pitch;
	float	h_yaw;
	float	h_roll;
	float	h_depth;
	float	h_width;
	float	h_height;
	float	h_confidence;
};

class FaceAPI
{
public:
	FaceAPI();
	void Init( void );
	void Shutdown();
	void SetData(smEngineHeadPoseData head_pose);
	FaceAPIData GetData();
private:
	smEngineHandle engine_handle;
	FaceAPIData m_data;
};

#endif FACEAPI_H

Contents of face_api.cpp:

#include "cbase.h"
#include "face_api.h"
#include "lock.h"

#define THROW_ON_ERROR(x) \
{ \
    smReturnCode result = (x); \
	if(result != 0) DevMsg("error code: %d\n", result); \
}

using namespace std;

faceapi::Mutex	g_mutex;
FaceAPI*		_faceapi;

// Callback function for head-pose
void STDCALL receiveHeadPose(void *, smEngineHeadPoseData head_pose, smCameraVideoFrame video_frame)
{
	_faceapi->SetData(head_pose);
}

void FaceAPI::SetData(smEngineHeadPoseData head_pose)
{
	faceapi::Lock Lock(g_mutex);
    
	m_data.h_height		= head_pose.head_pos.y;
	m_data.h_width		= head_pose.head_pos.x;
	m_data.h_depth		= head_pose.head_pos.z;
	m_data.h_yaw		= head_pose.head_rot.y_rads;
	m_data.h_pitch		= head_pose.head_rot.x_rads;
	m_data.h_roll		= head_pose.head_rot.z_rads;
	m_data.h_confidence	= head_pose.confidence;
}

FaceAPIData FaceAPI::GetData()
{
	faceapi::Lock Lock(g_mutex);
	FaceAPIData copy = m_data;
	return copy;
}

FaceAPI::FaceAPI( void )
{
	// don't initialise anything here as it would be too early
	_faceapi = this;
}

void FaceAPI::Init( void )
{
	//THROW_ON_ERROR(smAPIVersion(&major, &minor, &maintenance));

	// Determine if non-commercial restrictions apply
	const bool non_commercial_license = smAPINonCommercialLicense() == SM_API_TRUE;

	// Initialize the API
	THROW_ON_ERROR(smAPIInit());

	// Register the category of camera
	//THROW_ON_ERROR(smCameraRegisterType(SM_API_CAMERA_TYPE_PTGREY));
	THROW_ON_ERROR(smCameraRegisterType(SM_API_CAMERA_TYPE_WDM));

	// Create a new Head-Tracker engine that uses the camera
	THROW_ON_ERROR(smEngineCreate(SM_API_ENGINE_LATEST_HEAD_TRACKER,&engine_handle));

	// Check license for particular engine version (always ok for non-commercial license)
	const bool engine_licensed = smEngineIsLicensed(engine_handle) == SM_API_OK;

	// Hook up callbacks to receive output data from engine.
	// These functions will return errors if the engine is not licensed.
	if (engine_licensed) 
	{
		THROW_ON_ERROR(smHTRegisterHeadPoseCallback(engine_handle, 0, receiveHeadPose));
	} 
	else 
	{
		DevMsg("Engine is not licensed, cannot obtain any output data.");
	}

	//THROW_ON_ERROR(smHTSetTrackingRanges(engine_handle, 0.15, 1.5));

	// Start tracking
	THROW_ON_ERROR(smEngineStart(engine_handle));
}

void FaceAPI::Shutdown() 
{
	// Destroy engine
	THROW_ON_ERROR(smAPIQuit());
}

mutex.h (taken from Seeing Machine's sample code):

#ifndef SM_API_TESTAPPCONSOLE_MUTEX_H
#define SM_API_TESTAPPCONSOLE_MUTEX_H

#include "Windows.h"
#include <stdexcept>

namespace faceapi
{
	// A very simple mutex class for sample code purposes. 
	// It is recommended that you use the boost threads library.
	class Mutex
	{
	public:
		Mutex()
		{
			if (!InitializeCriticalSectionAndSpinCount(&_cs,0x80000400)) 
			{
				throw std::runtime_error("Failed to initialize Mutex");
			}
		}
		~Mutex()
		{
			DeleteCriticalSection(&_cs);
		}
		void lock() const
		{
			EnterCriticalSection(&_cs); 
		}
		void unlock() const
		{
			LeaveCriticalSection(&_cs); 
		}
	private:
		// Noncopyable
		Mutex(const Mutex &);
		Mutex &operator=(const Mutex &);
	private:
		mutable CRITICAL_SECTION _cs;
	};
}
#endif

lock.h (taken from Seeing Machine's sample code):

#ifndef SM_API_TESTAPPCONSOLE_LOCK_H
#define SM_API_TESTAPPCONSOLE_LOCK_H

#include "mutex.h"

namespace faceapi
{
	// A very simple scoped-lock class for sample code purposes. 
	// It is recommended that you use the boost threads library.
	class Lock
	{
	public:
		Lock(const Mutex &mutex): _mutex(mutex)
		{
			_mutex.lock();
		}
		~Lock()
		{
			_mutex.unlock();
		}
	private:
		// Noncopyable
		Lock(const Lock &);
		Lock &operator=(const Lock &);
	private:
		const Mutex &_mutex;
	};
}
#endif

Tweak the gameinfo.txt

Open your mod's gameinfo.txt file (MODBASE/gameinfo.txt) and change the SteamAppId from 420 to 218 so that it reads:

SteamAppId				218 		// GCF for Episode 2

Project Settings

Open Game_Episodic-2005.sln (MODBASE/src/Game_Episodic-2005.sln) in Visual Studio. Make the following changes:

1. Under Build > Configuration set both projects to Release
2. Under the Client Episode's Properties:
   1. Add ./faceapi/include to the C/C++ > General > Additional Include Directories
   2. Add ./faceapi/lib/smft32.lib to Linker > Input > Additional Dependencies
3. Add the face_api.cpp, face_api.h, lock.h and mutex.h files to the Client Episode project by either choosing to Add an existing file or by dragging the files into the project

Code Changes

You will need to make several changes to the code to allow it to compile.

viewrender.cpp changes

Alter the file so that it reads:

	#define GetObject GetObject				// ADD THIS LINE
	ClientWorldListInfo_t *pResult = gm_Pool.GetObject();

jobthread.h changes

Alter the file so that it reads:

	#if defined( Yield )					// ADD THIS LINE
		#undef Yield					// ADD THIS LINE
	#endif							// ADD THIS LINE
	virtual void Yield( unsigned timeout ) = 0;

protected_things.h changes

Find and delete/comment-out the following lines:

	#if defined( EnterCriticalSection )
		#undef EnterCriticalSection
	#endif
	#define EnterCriticalSection EnterCriticalSection__USE_VCR_MODE

platform.h changes

Find and delete/comment-out the following line:

	typedef signed char int8;

viewrender.h changes

Alter the top of the file so that it reads:

#include "view_shared.h"
#include "faceapi/face_api.h"			// ADD THIS LINE

Alter the CViewRender declaration so it reads as:

	int				m_BuildWorldListsNumber;
	FaceAPI				m_FaceAPI;				// ADD THIS LINE

view.cpp changes

Alter the top of the file so that it reads:

#include "ScreenSpaceEffects.h"
#include <vector>		// ADD THIS LINE

Somewhere near the top of the file add the line:

	ConVar headSmoothing("headSmoothing", "10", FCVAR_ARCHIVE, "", true, 1, false, 1);

Alter CViewRender::Init( void ) so that it reads:

	AngleVectors( angles, &m_vecLastFacing );
	m_FaceAPI.Init();					// ADD THIS LINE

Alter CViewRender::Shutdown( void ) so that it reads:

	tempents->Shutdown();
	m_FaceAPI.Shutdown();					// ADD THIS LINE

As an example of how to actually use the head position data, in CViewRender::Render( vrect_t *rect ) after the line:

	m_View.m_flAspectRatio	= ( engineAspectRatio > 0.0f ) ? engineAspectRatio : ( (float)m_View.width / (float)m_View.height );

add:

	// a simple use of the head tracking to control the roll of the view
	FaceAPIData headpos = m_FaceAPI.GetData();
	static std::vector<float> average;
	average.push_back(headpos.h_roll);
	while(average.size() > headSmoothing.GetInt())
		average.erase(average.begin());
	float avg = 0;
	for(int i = 0; i < average.size(); i++)
		avg += average.at(i);
	avg /= average.size();
	m_View.angles[ROLL] -= avg * 57.2958;

Testing it

Compile the mod using Build > Build Solution. Whilst it is compiling, you should use the time to find a map to test your game with. You can either choose to create a map or copy one from an existing title using GCFScape, noting that you cannot distribute these latter levels with your mod.

Once compiled launch the game. From within the game, open the console (~) and type map your_map_name to load your chosen map. You then alter the smoothness/responsiveness of the head tracking technique by opening the console again and typing headSmoothing n, where a larger n will cause the technique to be smoother, yet less responsive.

Comments

The use of the head data in this tutorial can be improved in several ways. Some changes worth considering include:

1. Averaging over a fixed sample size does not take into account the computer speed. As such, on a faster computer this fixed size will represent a smaller period of time as compared to when run on a slower machine. To improve this, the sample window should be timestamp based. It also preferably to use a running sum to make the averaging process more efficient.
2. As head-tracking relies on fuzzy logic, it is not safe to assume a neutral position, as this position will vary from one player to the next. To improve this, a dynamic neutral position should be computed and subtracted from the incoming data. One approach for computing this neutral position is to compute the average head position over all time.

Once you have completed this tutorial you may wish to improve on how it is integrated