LipDub Offers AI-Powered Lip Sync and Video Translation, With the Thrill of ...

Jonathan Bronfman delivered a keynote speech at the T-EDGE Conference.

AsianFin – When it comes to video production, “expensive” or “time-consuming” are some of the words that pop up first. No wonder Hollywood leads the world in filmmaking, since they have massive capital to squander. A Canadian company has come up with a solution that slashes costs and still maintains the quality.

At the recently-concluded T-EDGE Conference in Hilton Hotel in Beijing, LipDub.AI founder and CEO Jonathan Bronfman demonstrated the company’s proprietary lip-syncing technology in a keynote speech, which captivated millions of online and offline audiences in China and beyond.

The quality of lip synching is so real that an audience member would think Craig Smith, a former New York Times correspondent from Washington state, is Chinese if you just look at his lips when he inquired about a fee rate in mandarin. More awe-inspiring video clips were played during Bronfman’s speech, showcasing how to translate video content from Flemish to English and generate new videos from old ones by simply inputting new scripts. A science fiction language in Avatar, Star Trek or Game of Thrones can be quickly aced by the technology. A cinemagoer who is exhausted by gazing at subtitles and listening to a foreign language would be relieved with the new technology.

Bronfman, also the CEO of Monsters, Aliens, Robots, Zombies, known as MARZ, oversaw his company producing visual effects for Disney, Netflix and Amazon. That venture also exposed him to production of audios and videos. He realized that the industry was homogenous: everyone was using the same tech stack, and everyone was using the same pool of artists.

In 2019, he identified artificial intelligence (AI) as a tool to gain competitive edges over rivals on cost, time and capacity. Thus a world-class research team was formed, led by Daniel Cohen-Or, a professor of visual computing at Tel Aviv University and Ali Mahdavi-Amiri, an assistant professor of computing science at Simon Fraser University. Their first product was Vanity, which offers 2D aging, de-aging, beautification, wig, and prosthetic fixes up to 300 times faster than traditional visual effects pipelines. The product was well received in Hollywood due to its cost-effectiveness and uncompromising quality.

In 2021, Bronfman came across Squid Game, a Korean show on Netflix. One thing bothered him: the show was popular in English markets, but lips did not match. He called Cohen-Or and Mahdavi-Amiri and they said they would try to fix the problem. And next they created LipDub, the lip sync technology, for Hollywood!

“It’s not TikTok filter. This is real 4K high resolution. You need pose, change camera, change lighting, change occlusion,” Bronfman noted, adding that Hollywood continues to the North Star of the industry or the ultimate market for producers.

LipDub can be used in three ways: video content translation, dialogue changing in custom avatars and personalization. For example, brands have different names for their products in different markets. If they want to shoot a commercial and they need to change the name of the specific product, then LipDub can be applicable.

The lip sync technology developed by LipDub.AI also works well on CG characters. “It’s just not lip syncing. We don't want a cardboard cutout. It needs to be dynamic. So expression -- Is the character happy? Is the character sad? We need to make sure we maintain the integrity of the original performance,” Bronfman explained.

The app, which is in Beta now and will be available for public consumption on January 20, 2025, can be used in many other industries, including advertising and e-learning. However, Bronfman tends to classify user cases not by industry, but by quality.

“Let’s cut it horizontally. Let’s cut it by people who care about quality and people who don’t care about quality. We care about quality. That continues to be our North Star,” said Bronfman. “Those are people that we are going to serve.”

Bronfman noted that AI is at an inflection point. “People are pouring tons of money into the space. People want it faster, more affordably … They want to see what is the most effective, active way to promote or engage an audience in a given market or in a given sector,” he noted.

He believes that AI is the way to solve the common and persistent problem of hefty costs in the video and filmmaking industry. While acknowledging that right now it is slow and expensive to produce videos, Bronfman offered a solution. “With LipDub, videos can be reused. You can globalize your content in minutes,” he concluded his keynote speech on a high note.