Real-time Facial Reenactment Technology Implementation Without Post-processing
Overcoming Limitations of Limited Driver Adoption and
Additional Deep Learning Model Training

Video technology company HyperConnect has developed a technology that creates moving facial video from just a single photograph using AI.
HyperConnect AI Lab recently announced 'MarioNETte,' a facial reenactment technology that creates moving facial (driver) video by adding motion from just a single facial photograph (target).

Existing facial reenactment technology has the limitation of causing distortion when the face to be reenacted is affected by the face providing motion, creating difficulties in commercializing the technology. To create high-quality reenactment video, only faces with similar contours to the target face could be used as drivers, or the deep learning model had to be additionally trained for each target face.

To solve this problem, HyperConnect introduced a structure (image attention block) that allows the deep learning model to focus only on necessary information from the driver's face.

Additionally, to separate and reenact information representing identity from the face and information representing expressions and movements, the company devised a technique (landmark transformer) that applies only the driver's expression and movement information to the face.

As a result, even when the form of the face to be reenacted and the face providing motion are completely different, it is now possible to create realistic facial reenactment video in real-time without post-processing if there is just one target photograph.

Ha Sung-joo, Director overseeing HyperConnect AI Lab, stated "Existing facial reenactment technology takes too long to run and requires a lot of data, so to increase video quality or model speed, we had to accept video quality loss. This research from the AI Lab presents an approach that simultaneously improves video quality and implementation speed, demonstrating the commercialization potential of facial reenactment technology," and added "We have successfully demonstrated this technology at the International Conference on Computer Vision (ICCV 2019) held last October and have expressed our commitment to advancing technology development to a level applicable to mobile services. Recognizing that facial reenactment technology has significant potential value as it can be utilized in various fields such as video calls and game AR, HyperConnect plans to create synergy effects by simultaneously improving existing services like Azar and Hakuna Live while focusing on developing new services."