Playing with Environmental HDR mode in ARCore
Light estimation is one of the key elements for creating more realistic and immersive AR experiences. Shadows, intensity of the light, shading and reflections are just some examples of what makes virtual objects look more real. By using machine learning, environmental HDR mode in ARCore takes us one step further towards the future of mobile AR. In this post, you will learn about environmental HDR mode and how to enable it in your Android project using Sceneform.
Check out the image below where the model to the left is virtual and one the right is real. It is hard to guess which one is virtual, isn’t it? That is the power of environmental HDR mode in ARCore. Let’s dive deeper into the details.
Facts
Environmental HDR mode was introduced at Google I/O’19 during “New in ARCore“ talk.
Environmental HDR mode is available in ARCore since v1.10.0
Sceneform has recently included the support in version v1.13.0
Environmental HDR With Reflections is enabled by default in Unity SDK 🤩
For Android project, session config changes are required
Light Estimation with Machine Learning
Environmental HDR mode uses ML to estimate the lighting in the scene. The light estimation is also dynamic and it will change accordingly with the environment (illustrated in mannequin image above). But how does it really work?
ARCore uses TensorFlow-lite neural net to calculate the lighting environment. The model is trained on a set of images that are taken with a special rig equipped with 3 spheres: mirror ball, matte silver and diffuse sphere aligned to the bottom of the camera’s field of view. The spheres are made with different reflective properties that give variety of input data to the algorithm. The neural net was trained on approximately 50 million samples from indoor and outdoor scenes [1].
Implementation details with Sceneform
To change light estimation mode in your project, set lightEstimationMode for AR session configuration to ENVIRONMENTAL_HDR. The example below demonstrates a use case to toggle environmental HDR at runtime.
AMBIENT_INTENSITY vs ENVIRONMENTAL HDR
If you want to learn more about Environmental HDR mode in ARCore check out:
Talk “Increase AR Realism with Lighting“ from Google I/O’19 [1]
CVPR paper: DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality [2]
