Photorealistic Avatars: Meta’s research makes progress

Meeting people virtually as if they were in the same room: that’s Meta’s big goal. The Codec Avatars are intended to fulfill this promise one day.

In spring 2019, Meta presented research on sophisticated avatars for the first time, which are to replace video conferences one day.
Users wear a prototype VR headset that records eye and mouth movement using five sensors. With the help of an AI model, an impressively realistic avatar is generated in real time from this data, including eye movements and the finest facial expressions.
Beforehand, the users have to be scanned in a sophisticated 3D studio. The research team hopes that in the future it will be possible to create a high-quality 3D scan of the face oneself, for example with a smartphone.
Currently, Meta relies on comic-style avatars, a concession to the severely limited processing power of the Meta Quest 2.

Special chip speeds up codec avatars
RoadtoVR also reported on codec avatars, summarizing a recent paper from April 2022.
According to the paper, a group of researchers has developed a special AI chip that speeds up the calculation of codec avatars on standalone VR headsets. The chip, which is only 1.6 square millimeters in size, was optimized for the avatar system and specializes in processing sensor data. Conversely, the AI model was also revised for the chip’s architecture.
The result of these optimizations: The encoding of the codec avatars runs faster and the energy consumption as well as the heat development are lower. The XR2 chip is responsible for the decoding and the actual rendering of the avatars.
Zuckerberg previously said in a podcast with Lex Fridman that Meta will prioritize realistic social interaction in VR, even if the technology required to do so leads to more expensive, clunkier headsets.

Full story here