Recognize and Track Faces

Use an instance of the ObjectTracker class to detect, recognize, and track faces in a video stream.

The video stream may originate from a camera or a video file. The object tracker allows you to either process the video stream in real time or as fast as possible. The former mode would be typically used for a video stream that originates from a camera, while the later mode can be used to process the content of a video file at a speed faster than the normal playback speed of the video file.

Initialize the ArgusKit Framework

You must initialize the ArgusKit framework before you call any of its APIs. You can do this by calling the ARKInit() function at the beginning of your application.

Note: You need to call this function only once per application session.

Create the Video Player

The video player is used to playback video from a file or stream. The video frames are then passed into the object tracker for detection and recognition. The video player is created with a URL, and various callbacks are initialized to obtain information about the video state. The most important callback for tracking is the didCreateVideoFrame which is called when a new video frame is available that can be passed to the object tracker.

// Create the video player
ARKVideoPlayerCallbacks callbacks0;
callbacks0.context = NULL;
callbacks0.didCompletePreroll = video_player_preroll_completed_callback;
callbacks0.didChangePauseState = video_player_pause_changed_callback;
callbacks0.didCreateVideoFrame = video_player_frame_callback;
callbacks0.didEncounterError = video_player_error_callback;
callbacks0.didReachEndOfStream = video_player_end_of_stream_callback;
 
gPlayerRef = ARKVideoPlayerCreate(url, &callbacks0);

Create the Object Tracker

The first step is to create an object tracker instance. You do this by creating an instance of an object tracker configuration object that stores all the configuration information an object tracker needs to do its work. Most of the configuration information has sensible default values, but the following information must be explicitly provided:

The following code snippet shows how to set up the object tracker configuration and how to initiate the object tracker:

// Select the appropriate cloud environment. Eg. PROD
const char* environName = get_feature_string_value(argc, argv, "--environment");
char environBuffer[16];
strcpy(environBuffer, "com.real.");
strcat(environBuffer, (environName) ? environName : "PROD");
ARKEnvironmentRef envRef = ARKEnvironmentCopyNamed(environBuffer);
 
// Specify the cloud environment log-in credentials
ARKUserRef userRef = NULL;
 
const char* userName = get_feature_string_value(argc, argv, "--user");
const char* userPassword = get_feature_string_value(argc, argv, "--password");
const char* directory = get_feature_string_value(argc, argv, "--directory");
if (userName && userPassword) {
    userRef = ARKUserCreate(userName, userPassword);
 
    // Set the user directory in which the cloud should store its data
    ARKUserSetDirectory(userRef, (directory) ? directory : "test");
}
 
// Create the object tracker configuration
ARKObjectTrackerConfigurationRef configRef = ARKObjectTrackerConfigurationCreate();
ARKObjectTrackerConfigurationSetObject(configRef, kARKObjectTrackerConfigurationKey_Environment, envRef);
ARKObjectTrackerConfigurationSetObject(configRef, kARKObjectTrackerConfigurationKey_User, userRef);
ARKObjectTrackerConfigurationSetString(configRef, kARKObjectTrackerConfigurationKey_SiteID, "Building 1");
ARKObjectTrackerConfigurationSetString(configRef, kARKObjectTrackerConfigurationKey_SourceID, "Camera 1");
 
 
// Create the object tracker
ARKObjectTrackerCallbacks callbacks1;
callbacks1.context = NULL;
callbacks1.willBeginTracking = object_tracker_will_begin_tracking_callback;
callbacks1.didEndTracking = object_tracker_did_end_tracking_callback;
callbacks1.didCreateTrackingResult = object_tracker_did_create_tracking_result_callback;
 
gTrackerRef = ARKObjectTrackerCreate(configRef, false, &callbacks1);
ARKObjectRelease(configRef);
configRef = NULL;

This example code selects the desired cloud environment and creates a new user object with the required user identifier and password. It also sets the cloud directory where the cloud-based face recognizer should save recognition-related information.

It then creates an object tracker configuration object and sets the cloud environment, cloud user, and some additional information to help identify the camera. After that, it sets up the necessary callbacks the object tracker should invoke as it processes the video stream. Finally, the example code creates the actual object tracker object.

Note: The example code assumes that the input video stream originated from a camera. This is why we pass true to the real-time parameter of the ARKObjectTrackerCreate() function.

Start a Tracking Session

All tracking-related activities are done in the context of a tracking session. The object tracker uses a tracking session to maintain the necessary state. The following code snippet shows you how you can start a new tracking session:

ARKObjectTrackerBeginTracking(trackerRef);

Note: Always end the current tracking session and start a new session if the video source or the resolution or frame rate of the video stream has changed. For example, end the current tracking session and start a new one if the user switched cameras or selected a different capture profile.

The object tracker invokes the begin-tracking callback at the start of a new tracking session. Your application can use this callback as a signal that a new tracking session has started. The following code snippet shows an example of such a callback:

static void object_tracker_will_begin_tracking_callback(ARKObjectTrackerRef _Nonnull trackerRef, void* _Nullable context)
{
    printf("willBeginTracking\n");
}

Run a Tracking Session

A new video frame object should be created for every decoded video frame, and this video frame object should then be passed to the object tracker. The object tracker in turn runs a face detector on the video frame and it triggers face recognitions as needed. The object tracker then updates its internal list of tracked object with the result of detectors and recognizers.

The object tracker invokes the application-provided callback with the current state of the tracked object list. The application can inspect this list and trigger actions based on it.

Note: The application should create a copy of the tracked object list if it wants to hold on to the data. (e.g. if the application wants to process the tracked objects on a different thread)

The following code snippet shows how to create a video frame object and how to pass it to the object tracker:

static void video_player_frame_callback(ARKVideoPlayerRef _Nonnull playerRef, ARKVideoFrameRef _Nonnull frameRef, void* _Nullable context)
{
    ARKObjectTrackerTrackObjects(gTrackerRef, frameRef);
    ARKObjectRelease(frameRef);
}

You must provide a timestamp when you create a video frame object. This is typically the presentation timestamp of the video frame. The isSceneChange parameter of the ARKVideoFrameCreateWithPixelBuffer() function should be set to true if the video frame is the first video frame after a scene change. Scene changes in movies are often indicated by a cut transition from one scene to another. The object tracker uses this information to enhance its ability to disambiguate between persons in different scenes.

Note: Although a video stream from a camera includes key frames, those key frames do not indicate a scene change for tracking purposes. For that reason, those key frames should not be treated as a scene change.

The following code snippet shows an example implementation of the did-track that simply prints a description of the new tracking results callback:

static void object_tracker_did_create_tracking_result_callback(ARKObjectTrackerRef _Nonnull trackerRef, ARKTrackingResultRef _Nonnull resultRef, void* _Nullable context)
{
    ARKObjectPrintDebugDescription(resultRef)
}

The object tracker invokes the end-tracking callback at the end of the tracking session. Your application code can use this callback as a signal that a tracking session has ended. The following code snippet shows an example implementation of such a callback:

static void object_tracker_did_end_tracking_callback(ARKObjectTrackerRef _Nonnull trackerRef, void* _Nullable context)
{
    printf("didEndTracking\n");
}

End a Tracking Session

You inform the object tracker about the end of a tracking session by invoking the ARKObjectTrackerEndTracking() function. This allows the object tracker to clean up its internal state and lets it know it should execute pending callbacks as soon as possible.

The following code snippet shows how to end a tracking session:

ARKObjectTrackerEndTracking(trackerRef);

Detecting Persons

Once you setup and run a basic object tracker configuration object, enable the detection of persons by setting the kARKObjectTrackerConfigurationKey_ShapeDetector_Enable property in the object tracker's detector configuration object to true. This causes the object tracker to return tracked objects with a detected object type equal to kARKObjectType_Shape.

See Also