Face Landmarks for ARCore Augmented Faces
In this tutorial you will learn how to get location of any face landmark for your filter using ARCore Augmented Faces API.
According to the ARCore documentation, there are 3 face region poses available at the moment:
Left forehead (
LEFT_FOREHEAD
)Right forehead (
RIGHT_FOREHEAD
)Tip of the nose (
NOSE_TIP
)
But how can we get the position of other face landmarks, like eyes, mouth and so on?
Face Mesh Vertices
AugmentedFace describes a face detected by ARCore and has access to face mesh related data. One of those methods is getMeshVertices that returns a FloatBuffer.
ARCore github repository has canonical_face_mesh.fbx face model as a referance to help creators make custom textures and 3D models. According to this guide, canonical_face_mesh.fbx model got 468 point face texture mesh. Going further in the quest for face landmarks, I found this github repository with multiple images of mesh index mapping for our reference face model. One of such models is illustrated below:
Face Landmark Position
Let’s find an approximate position for LEFT_EYE landmark:
getMeshVertices method will return 468 * 3 values (x,y,z- coordinates for 468 points)
following the mesh mapping let’s take mesh point number 374
The approximate position of left eye will be calculated as following:
val buffer = augmentedFace?.meshVertices val pos = Vector3(buffer.get(374 * 3), buffer.get(374 * 3 + 1), buffer.get(374 * 3 + 2))
Implementation Example
For this tutorial, I will define 3 face landmarks and use it for a face filter. You can follow the next steps:
Extend AugmentedFaceNode
Define face landmarks nodes
private var eyeNodeLeft: Node? = null private var eyeNodeRight: Node? = null private var mustacheNode: Node? = null
3. Create Renderable for each node
4. Define face landmark positions
5. Update the nodes onUpdate
Check out the final CustomFaceNode.kt and sample source code linked below for a full demo implementation. And here is the final result: