Audio-to-Audio Method Applied to Reproduce Original Emotion and Voice

ElevenLabs, an AI audio specialist company, has unveiled a next-generation AI dubbing model that preserves the emotion and expression of the original voice. The technology aims to improve multilingual dubbing quality amid growing demand for content localization.

ElevenLabs announced that it is providing the AI dubbing model 'Dubbing v2' through its API service ElevenAPI.

Following its release as a user interface (UI) version last May, the model is now being expanded to API form, enabling developers and enterprises to integrate it directly into their own services and workflows.

Unlike the conventional method of converting speech to text and then synthesizing it back to speech, Dubbing v2 applies an 'Audio-to-Audio' structure that directly converts speech to speech.

ElevenLabs explained that through this approach, the emotional expression, voice characteristics, and performance of the original speaker can be preserved more naturally.

Key features include △enhanced emotion and expressiveness reproduction △multilingual translation based on speaker voice △improved lip-sync timing with video △support for 92 languages.

According to the company, supported languages include Korean and Japanese.

ElevenLabs identified content production and the global localization market as primary use cases for Dubbing v2.

Video creators and game developers can utilize it for character dialogue and cutscene localization, while enterprises can apply it to producing marketing content and promotional videos in multiple languages. Additionally, the company explained there is potential for application in overseas distribution of feature-length content such as films and dramas.

Particularly, with support for Korean and Japanese, bidirectional applications are expected, including the overseas expansion of K-content and Korean dubbing of foreign content.

However, real-time multilingual dubbing features such as live streaming are not included in this release scope.

ElevenLabs explained that through Dubbing v2, it aims to improve accessibility to high-quality dubbing work in the content production process.