Face Landmarks for ARCore Augmented Faces

In this tutorial you will learn how to get location of any face landmark for your filter using ARCore Augmented Faces API.

According to the ARCore documentation, there are 3 face region poses available at the moment:

• Left forehead (LEFT_FOREHEAD)

• Right forehead (RIGHT_FOREHEAD)

• Tip of the nose (NOSE_TIP)

But how can we get the position of other face landmarks, like eyes, mouth and so on?

Face Mesh Vertices

AugmentedFace describes a face detected by ARCore and has access to face mesh related data. One of those methods is getMeshVertices that returns a FloatBuffer.

ARCore github repository has canonical_face_mesh.fbx face model as a referance to help creators make custom textures and 3D models. According to this guide, canonical_face_mesh.fbx model got 468 point face texture mesh. Going further in the quest for face landmarks, I found this github repository with multiple images of mesh index mapping for our reference face model. One of such models is illustrated below:

Image Source: https://github.com/ManuelTS/augmentedFaceMeshIndices/blob/master/Front.jpg

Face Landmark Position

Let’s find an approximate position for LEFT_EYE landmark:

• getMeshVertices method will return 468 * 3 values (x,y,z- coordinates for 468 points)

• following the mesh mapping let’s take mesh point number 374

• The approximate position of left eye will be calculated as following:

val buffer = augmentedFace?.meshVertices
val pos = Vector3(buffer.get(374 * 3),
                  buffer.get(374 * 3 + 1),
                  buffer.get(374 * 3 + 2))

Implementation Example

For this tutorial, I will define 3 face landmarks and use it for a face filter. You can follow the next steps:

1. Extend AugmentedFaceNode

2. Define face landmarks nodes

private var eyeNodeLeft: Node? = null
private var eyeNodeRight: Node? = null
private var mustacheNode: Node? = null

3. Create Renderable for each node

4. Define face landmark positions

5. Update the nodes onUpdate

Check out the final CustomFaceNode.kt and sample source code linked below for a full demo implementation. And here is the final result: