Playing with Environmental HDR mode in ARCore

Light estimation is one of the key elements for creating more realistic and immersive AR experiences. Shadows, intensity of the light, shading and reflections are just some examples of what makes virtual objects look more real. By using machine learning, environmental HDR mode in ARCore takes us one step further towards the future of mobile AR. In this post, you will learn about environmental HDR mode and how to enable it in your Android project using Sceneform.

Check out the image below where the model to the left is virtual and one the right is real. It is hard to guess which one is virtual, isn’t it? That is the power of environmental HDR mode in ARCore. Let’s dive deeper into the details.

Facts

Light Estimation with Machine Learning

Environmental HDR mode uses ML to estimate the lighting in the scene. The light estimation is also dynamic and it will change accordingly with the environment (illustrated in mannequin image above). But how does it really work?

ARCore uses TensorFlow-lite neural net to calculate the lighting environment. The model is trained on a set of images that are taken with a special rig equipped with 3 spheres: mirror ball, matte silver and diffuse sphere aligned to the bottom of the camera’s field of view. The spheres are made with different reflective properties that give variety of input data to the algorithm. The neural net was trained on approximately 50 million samples from indoor and outdoor scenes [1].

Implementation details with Sceneform

To change light estimation mode in your project, set lightEstimationMode for AR session configuration to ENVIRONMENTAL_HDR. The example below demonstrates a use case to toggle environmental HDR at runtime.

AMBIENT_INTENSITY vs ENVIRONMENTAL HDR

If you want to learn more about Environmental HDR mode in ARCore check out:

  1. Talk “Increase AR Realism with Lighting“ from Google I/O’19 [1]

  2. CVPR paper: DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality [2]

Previous
Previous

Onboarding UX for Mobile AR

Next
Next

Skeletal animation with Mixamo in ARCore