FaceCompare Android Java Sample App

The sample app demonstrates how to use the Android eSDK to create a Java Android app.

The app is organised into:

  1. A Java layer which includes the UI & presentation logic.
  2. The native layer which directly invokes the embedded SDK API and implements some of the app's business logic.

The layers communicate via the Java Native Interface (JNI). All JNI calls are done synchronously.

App's Features

The Use Case

  1. User picks an image which is used as a source.
  2. The source image is processed to detect and learn the largest face.
  3. User picks another image which is used as a target.
  4. The target image is processed to detect all faces which are compared against the face from the first image.

The results, in terms of match percentage, are displayed for every detected face.

IDE System Requirements

For Windows:

For macOS:

For Linux:

Supported Device Architecture:

Prerequisites

Download the SAFR eSDK package for Android from the SAFR Download Portal. We recommended you download both armeabi-v7a and arm64-v8a packages because the sample project is set up to build both architectures and the PlayStore actually expects both of them.

Set Up

  1. Download and unpack SAFREMBSDK_android-lite_xxx.tgz and SAFREMBSDK_android64-lite_xxx.tgz from the SAFR Download Portal.

  2. Copy SAFREMBSDK_android-lite_xxx/lib/arm-linux-androideabi into SAFREMBSDK_android64-lite_xxx/lib/arm-linux-androideabi.

  3. SAFREMBSDK_android64-lite_xxx/ should look like this:

The project's location is SAFREMBSDK_android64-lite_xxx/samples_java_android/facecompare/

Before importing into Android Studio, the project's dependency files must be set. This can be done automatically by executing the samples_java_android/facecompare/source/setup_dependency_tree.sh script from within its directory, or by manually copying the files.

Once completed, the final directory structure should look as follows:

eArgusSDK\samples_java_android\facecompare\
    |--- doc - docs directory
    |--- source - app directory
    |--- esdk_distribution 
        |--- include - header files
        |--- model - model files
        |--- lib- libraries - directory names must match to Android ABIs
            |--- arm64-v8a - content from aarch64-linux-android package if available
            |--- armeabi-v7a - content from arm-linux-androideabi package if available

Android Studio Setup

To import the project into Android Studio, go to File -> New -> Import Project…. Navigate to "package/samples_java_android/facecompare/source" - import.

Once the indexing is done, open MainActivity.java and replace OBTAIN_FROM_RN with the license key you've obtained from RealNetworks.

private static final String LICENSE = "OBTAIN_FROM_RN";

If needed, update LIB_VERSION inside the app/src/main/cpp/CMakeLists.txt file:

set(LIB_VERSION 3) # It must match to libArgusKit's version number i.e. libArgusKit-3.so version is 3

The app is now ready to build and run.

Source Files

app/src/main/cpp/native-lib.cpp

Includes native code which does eSDK invocation.

Detector, Recogniser, and Store initialization:

   const char *_modelsPath = env->GetStringUTFChars(modelsPath, JNI_FALSE);
    const char *_storePath = env->GetStringUTFChars(storePath, JNI_FALSE);
    const char *_licence = env->GetStringUTFChars(licence, JNI_FALSE);

    EARFaceRecognitionModelType faceRecognitionModelType = EARFaceRecognitionModelType::kEARFaceRecognitionModelType_MaskOptimized;
    EARRecognitionModelPerformance recognitionModelPerformance = EARRecognitionModelPerformance::kEARFaceRecognitionModelPerformance_AccuracyOptimized;

    if (configOptions != nullptr) {

        jclass optionsClass = env->FindClass("com/real/facecompare/ConfigOptions");

        faceRecognitionModelType = getFaceRecognitionModelType(env->GetIntField(configOptions, env->GetFieldID(optionsClass, "faceRecognitionModelType", "I")));
        recognitionModelPerformance = getRecognitionModelPerformance(env->GetIntField(configOptions, env->GetFieldID(optionsClass, "recognitionModelPerformance", "I")));
    }

    EARSetModelFilesSearchPath(_modelsPath);

    const size_t _length = env->GetStringUTFLength(licence);
    char *_licenceCopy = (char *) malloc(_length);

    strcpy(_licenceCopy, _licence);

    EARLicense pLicense = EARLicense();

    pLicense.key = _licenceCopy;
    pLicense.length = _length;

    // TODO handle errors
    g_pStore = EARPersonStoreCreate(_storePath, nullptr);// Don't use password for encryption
    g_lastError = EARGetLastError();

    EARFaceDetectionInitializationConfiguration detectionInitializationConfiguration;

    g_pDetector = EARFaceDetectorCreate(&pLicense, EARFaceDetectionType::kEARFaceDetectionType_Retina, &detectionInitializationConfiguration, 0);

    EARFaceDetectionConfigurationInitWithPreset(&g_faceDetectionConfiguration, kEARFaceDetectionConfigurationPreset_Generic);

    g_lastError = EARGetLastError();

    g_pRecognizer = EARFaceRecognizerCreate(&pLicense, g_pStore, faceRecognitionModelType, recognitionModelPerformance, 0);

    g_lastError = EARGetLastError();

    free(_licenceCopy);

    env->ReleaseStringUTFChars(licence, _licence);
    env->ReleaseStringUTFChars(modelsPath, _modelsPath);
    env->ReleaseStringUTFChars(storePath, _storePath);

Calculate match using confidence attribute and return results to the Java layer:

jobject confidenceScore(JNIEnv *env, jclass type, jobject sourceByteBuffer, jint sourceWidth, jint sourceHeight, jint sourceStride, jobject imageBuffer, jint imageWidth, jint imageHeight, jint imageStride) {

    jobject result;
    jobjectArray facesResult = nullptr;

    internalClearPersonStore();

    auto *sourceBytes = static_cast<jbyte *>(env->GetDirectBufferAddress(sourceByteBuffer));
    jbyte *_sourceBytes = argbToRgba(sourceBytes, sourceWidth, sourceHeight, sourceStride);

    auto *const pImageToLearn = new EARBitmap();
    pImageToLearn->pixels = _sourceBytes;
    pImageToLearn->bytesPerRow = sourceStride;
    pImageToLearn->width = sourceWidth;
    pImageToLearn->height = sourceHeight;
    pImageToLearn->pixelFormat = EARPixelFormat::kEAR_RGBA32;

    auto *imageBytes = static_cast<jbyte *>(env->GetDirectBufferAddress(imageBuffer));
    jbyte *_imageBytes = argbToRgba(imageBytes, imageWidth, imageHeight, imageStride);

    auto *const pImageToCompareAgainst = new EARBitmap();
    pImageToCompareAgainst->pixels = _imageBytes;
    pImageToCompareAgainst->bytesPerRow = imageStride;
    pImageToCompareAgainst->width = imageWidth;
    pImageToCompareAgainst->height = imageHeight;
    pImageToCompareAgainst->pixelFormat = EARPixelFormat::kEAR_RGBA32;


    EARFaceRecognitionConfiguration learnConfig;
    EARFaceRecognitionConfigurationInitWithPreset(&learnConfig,  kEARFaceRecognitionConfigurationPreset_Learn);
    learnConfig.minConfidence = 1;
    learnConfig.minCenterPoseQuality = 0.0;
    learnConfig.minContrastQuality = 0.0;
    learnConfig.minSharpnessQuality = 0.0;
    learnConfig.minFaceSearchSize = 40;
    learnConfig.minFaceSize = 40;


    // First need to add faces from the target image to make sure we get unique pids
    EARDetectedFaceArrayRef pDetectedFacesToLearnLargest = EARFaceDetectorDetect(g_pDetector, pImageToLearn, &g_faceDetectionConfiguration);


    EARRecognizedFaceArrayRef pRecognizedFacesToLearn = EARFaceRecognizerRecognizeDetectedFaces(g_pRecognizer, pDetectedFacesToLearnLargest, pImageToLearn, &learnConfig, kEARAllowLearning);
    // Get a list of stored unique faces
    size_t learntTargetFacesCount = EARRecognizedFaceArrayGetCount(pRecognizedFacesToLearn);

    if (learntTargetFacesCount != 0) {

        EARFaceRecognitionConfiguration recogniseConfig;
        EARFaceRecognitionConfigurationInitWithPreset(&recogniseConfig, kEARFaceRecognitionConfigurationPreset_Recognize);
        recogniseConfig.minConfidence = 0;
        recogniseConfig.minCenterPoseQuality = 0.0;
        recogniseConfig.minContrastQuality = 0.0;
        recogniseConfig.minSharpnessQuality = 0.0;
        recogniseConfig.minFaceSearchSize = 40;
        recogniseConfig.minFaceSize = 40;

        // Store the source face
        EARDetectedFaceArrayRef pDetectedFacesToCompare = EARFaceDetectorDetect(g_pDetector, pImageToCompareAgainst, &g_faceDetectionConfiguration);

        const size_t found = EARDetectedFaceArrayGetCount(pDetectedFacesToCompare);

        EARRecognizedFaceArrayRef pRecognisedFacesToCompare = EARFaceRecognizerRecognizeDetectedFaces(g_pRecognizer, pDetectedFacesToCompare, pImageToCompareAgainst, &recogniseConfig, kEARAllowRecognition);

        size_t facesToCompareCount = EARRecognizedFaceArrayGetCount(pRecognisedFacesToCompare);

        if (facesToCompareCount != 0 || found != 0)
            facesResult = env->NewObjectArray(facesToCompareCount, g_pJNIClassInfo->detectedFaceClass, nullptr);

        for (size_t j = 0; j < facesToCompareCount; j++) {
            const EARRecognizedFace *face = EARRecognizedFaceArrayGetAtIndex(pRecognisedFacesToCompare, j);

            // Using confidence here as a measure of match
            const double match = std::min(1.0, face->confidence) * 100;


            jobject leftEyeCenter = env->NewObject(g_pJNIClassInfo->pointfClass, g_pJNIClassInfo->pointCtr, face->leftEyeCenter.x, face->leftEyeCenter.y);
            jobject rightEyeCenter = env->NewObject(g_pJNIClassInfo->pointfClass, g_pJNIClassInfo->pointCtr, face->rightEyeCenter.x, face->rightEyeCenter.y);
            jobject noseTip = env->NewObject(g_pJNIClassInfo->pointfClass, g_pJNIClassInfo->pointCtr, face->noseTip.x, face->noseTip.y);
            jobject leftMouthCorner = env->NewObject(g_pJNIClassInfo->pointfClass, g_pJNIClassInfo->pointCtr, face->leftMouthCorner.x, face->leftMouthCorner.y);
            jobject rightMouthCorner = env->NewObject(g_pJNIClassInfo->pointfClass, g_pJNIClassInfo->pointCtr, face->rightMouthCorner.x, face->rightMouthCorner.y);


            jobject faceObject = env->NewObject(g_pJNIClassInfo->detectedFaceClass, g_pJNIClassInfo->detectedFaceMatchCtr, face->localId, face->confidence, match, face->centerPoseQuality, face->contrastQuality, face->sharpnessQuality, face->boundsX, face->boundsY, face->boundsWidth, face->boundsHeight, leftEyeCenter, rightEyeCenter, noseTip, leftMouthCorner, rightMouthCorner);


            env->SetObjectArrayElement(facesResult, j, faceObject);

            env->DeleteLocalRef(leftEyeCenter);
            env->DeleteLocalRef(rightEyeCenter);
            env->DeleteLocalRef(noseTip);
            env->DeleteLocalRef(leftMouthCorner);
            env->DeleteLocalRef(rightMouthCorner);
            env->DeleteLocalRef(faceObject);
        }


        EARRecognizedFaceArrayDestroy(pRecognisedFacesToCompare);
        EARDetectedFaceArrayDestroy(pDetectedFacesToCompare);
    }

    EARRecognizedFaceArrayDestroy(pRecognizedFacesToLearn);
    EARDetectedFaceArrayDestroy(pDetectedFacesToLearnLargest);

    if (!facesResult) {
        facesResult = env->NewObjectArray(0, g_pJNIClassInfo->detectedFaceClass, nullptr);
    }

    result = env->NewObject(g_pJNIClassInfo->detectorResultClass, g_pJNIClassInfo->detectorResultCtr, facesResult, g_lastError);

    delete _sourceBytes;
    delete pImageToLearn;
    delete _imageBytes;
    delete pImageToCompareAgainst;


    return result;
}

app/src/main/java/com/real/facecompare/ESDKWrapper.java

Includes JNI API definitions and does JNI invocations:

public ArrayList<FaceInfo> confidenceScore(Bitmap imageToLearn, Bitmap imageToCompare) {

        final int imageToLearnWidth = imageToLearn.getWidth();
        final int imageToLearnHeight = imageToLearn.getHeight();
        final int imageToLearnByteSize = imageToLearn.getAllocationByteCount();
        final int imageToLearnBytesPerPixes = (int) (1f * imageToLearnByteSize / (imageToLearnWidth * imageToLearnHeight));
        final int imageToLearnStride = imageToLearnBytesPerPixes * imageToLearnWidth;

        final ByteBuffer imageToLearnBuffer = ByteBuffer.allocateDirect(imageToLearnByteSize);
        imageToLearn.copyPixelsToBuffer(imageToLearnBuffer);

        final int imageToCompareWidth = imageToCompare.getWidth();
        final int imageToCompareHeight = imageToCompare.getHeight();
        final int imageToCompareByteSize = imageToCompare.getAllocationByteCount();
        final int imageToCompareBytesPerPixes = (int) (1f * imageToCompareByteSize / (imageToCompareWidth * imageToCompareHeight));
        final int imageToCompareStride = imageToCompareBytesPerPixes * imageToCompareWidth;

        final ByteBuffer imageBuffer = ByteBuffer.allocateDirect(imageToCompareByteSize);
        imageToCompare.copyPixelsToBuffer(imageBuffer);

        final EARResult result = confidenceScore(imageToLearnBuffer, imageToLearnWidth, imageToLearnHeight, imageToLearnStride, imageBuffer, imageToCompareWidth, imageToCompareHeight, imageToCompareStride);

        final ArrayList<FaceInfo> faces = new ArrayList<>();

        if (result.result == 0) {
            Arrays.sort(result.faces, matchComparator);

            for (EARDetectedFace detectedFace : result.faces) {
                faces.add(new FaceInfo(detectedFace, ImageUtils.extractFaceBitmap(detectedFace, imageToCompare, true)));
            }
        }

        return faces;
    }

app/src/main/cpp/CMakeLists.txt

Native build configuration:

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.

# ---|
#    |--- doc - docs directory
#    |--- source - app directory
#    |--- esdk_distribution - binaries
#          |--- include - header files
#          |--- model - model files
#          |--- lib- libraries - directory names must match to Android ABIs
#              |--- arm64-v8a - content from aarch64-linux-android package
#              |--- armeabi-v7a - content from arm-linux-androideabi package
#

cmake_minimum_required(VERSION 3.10.2)

# Declares and names the project.

project("facecompare")

# esdk directories
set(ESDK_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../../../../esdk_distribution)
set(LIBS_DIR ${ESDK_ROOT_DIR}/lib/)

# JNI wrapper
set(SOURCE ${CMAKE_CURRENT_SOURCE_DIR}/native-lib.cpp)

# headers location
set(HEADERS ${ESDK_ROOT_DIR}/include)

# dependency libs

add_library(ArgusKit-2 SHARED IMPORTED)
set_target_properties(ArgusKit-2
        PROPERTIES IMPORTED_LOCATION ${LIBS_DIR}/${ANDROID_ABI}/libArgusKit-2.so)

add_library(ArgusKit SHARED IMPORTED)
set_target_properties(ArgusKit
        PROPERTIES IMPORTED_LOCATION ${LIBS_DIR}/${ANDROID_ABI}/libArgusKit.so)

add_library(facedetector SHARED IMPORTED)
set_target_properties(facedetector
        PROPERTIES IMPORTED_LOCATION ${LIBS_DIR}/${ANDROID_ABI}/libfacedetector.so)

add_library(facerecognizer SHARED IMPORTED)
set_target_properties(facerecognizer
        PROPERTIES IMPORTED_LOCATION ${LIBS_DIR}/${ANDROID_ABI}/libfacerecognizer.so)

add_library(jpeg SHARED IMPORTED)
set_target_properties(jpeg
        PROPERTIES IMPORTED_LOCATION ${LIBS_DIR}/${ANDROID_ABI}/libjpeg.so)

add_library(c++_shared SHARED IMPORTED)
set_target_properties(c++_shared
        PROPERTIES IMPORTED_LOCATION ${LIBS_DIR}/${ANDROID_ABI}/libc++_shared.so)

add_library(libtensorflowlite_gpu_delegate SHARED IMPORTED)
set_target_properties(libtensorflowlite_gpu_delegate
        PROPERTIES IMPORTED_LOCATION ${LIBS_DIR}/${ANDROID_ABI}/libtensorflowlite_gpu_delegate.so)

add_library(libtensorflowlite SHARED IMPORTED)
set_target_properties(libtensorflowlite
        PROPERTIES IMPORTED_LOCATION ${LIBS_DIR}/${ANDROID_ABI}/libtensorflowlite.so)

# The android stuff
find_library(
        log-lib
        log)

# include headers
include_directories(${HEADERS})

# wrapper
add_library( # Sets the name of the library.
        native-lib
        SHARED
        ${SOURCE})


# Link everything.
target_link_libraries(
        native-lib
        ArgusKit-2
        ArgusKit
        facedetector
        facerecognizer
        jpeg
        c++_shared
        libtensorflowlite_gpu_delegate
        libtensorflowlite
        ${log-lib})

The rest of the code does Android-specific such as handling image picker, RecyclerView, navigation stuff, etc.

Application Screens

  1. Start

    Shows when the app starts. Clicking the plus button will open an image picker to select a source image.

  2. The largest found face is displayed in the image view.

    The Floating Action Button (FAB) is displayed when the source face is ready. Clicking the FAB will show an image picker to select a target image. After the target image is selected, the face from the source image is learned and compared with faces being found on the target image. The resultant faces with their matching scores are displayed in the horizontal list.

  3. Display results

    A 100% match.

    Another face.

  4. Configure settings

    There is a settings page to configure the face recogniser model's type and performance. By default it's set to MaskOptimised/Speed settings. When returned to the main screen, it takes some time for the eSDK to re-initialise, so please wait some time before processing new images.

Notes

See Also