Tavus taps generative AI to power personalized videos with voice and face cloning

Generative AI is already looking like the major tech trend of 2023. The ability to generate fresh content via algorithms has been thrust into the public consciousness by the likes of ChatGPT, a chatbot-style technology trained on large language models (LLMs) capable of producing essays, poems, lyrics, news articles, and even computer programs. And then there’s DALL-E, from the same Microsoft-backed OpenAI that spawned ChatGPT, which serves a similar purpose but for visual creations instead.

While some have argued that ChatGPT signals AI’s arrival into the mainstream, the truth of the matter is we’re just at the start of a new era of AI-powered applications that will transform just about every facet of industry — from consumer search and stock photography, to real estate and content marketing.

And it’s against that backdrop that a fledgling startup called Tavus is looking to make its mark by enabling companies to create “unique” videos tailored to a specific individual, but based entirely on a single initial recording.

The idea is that a sales and marketing team, for example, can issue an endless stream of video pitches to prospective customers, maybe based on textual data the prospect submitted through an online form. Or perhaps a headhunter will use the platform to send multiple personalized videos to potential candidates using data gleaned from their LinkedIn profiles.

Founded out of San Francisco in 2020 by CEO Hassaan Raza and Quinn Favret, Y Combinator (YC) alum Tavus today announced that it has raised $6.1 million in a seed round of funding led by Silicon Valley investor Sequoia, with participation from a slew of high-profile backers including Accel Partners, Index Ventures, Lightspeed Ventures, and YC Continuity.

How it works

Any company looking to create multiple personalized videos will know that it’s an incredibly time-consuming, repetitive process: recording the same message, with substantively the same content, but tweaked for different clients or candidates. And that is what Tavus is looking to address, allowing users to create their own AI video templates in minutes, and then generate an unlimited number of versions of a video from that original source.

The initial onboarding process requires the user — for example, a recruiter or sales executive — to record a 15-minute video based on a script provided by Tavus, which is used to train the AI. Then, the user records a template for each campaign they want to create.

Using a web-based editor, users can then select what elements of the video they want to personalize, specifying each variable (e.g. company, executive name or location), adding in calls to action, and so on.

Tavus also supports longer-form variables via ChatGPT-powered snippets for more personalized introductions, something that Favret says has been highly requested by its users. For example, the base script on which a video is created can be configured to include a one-sentence introduction generated from a specific individual’s LinkedIn profile.

In essence, Tavus is striving to replicate what mass-marketing software has been doing in the email realm for donkey years, bringing it to the more visually-engaging world of videos. In truth, this could raise some red flags for some — will people be all that receptive to a personalized sales pitch when they discover that the sender hasn’t really gone to the effort of making a video just for them? But more than that, there is perhaps something a little creepy about an AI-generated video that uses personal information gleaned from a database, meaning that something designed to be personal could ultimately come across as incredibly impersonal when the user learns about how it was made.

Questions raised by such scenarios will continue as AI becomes more ingrained in our everyday lives. And Favret is quick to stress that while sales and marketing are obvious use-cases for its technology, it isn’t purely about those verticals — it’s seeing uptake from “an eclectic group of users,” including recruiters, university deans, and C-level executives.

“There’s a common misconception that Tavus only works with sales and marketing teams,” Favret said. “While this is a focus of ours, we have users applying Tavus in innovative and powerful ways across the full customer journey. Many of our power users apply Tavus broadly across their organizations, including for customer success, product, recruiting, and other go-to-market related functions.”

And who exactly, is putting themselves forward for cloning?

“Typically the user clones themselves, however it’s also common for businesses to have a central figure — such as an executive or spokesperson — record the videos to have a consistent face of the company,” Favret added. “Tavus is designed for all types of users to easily be able to clone themselves in minutes.”

Under the hood, Tavus says that it uses machine learning to train a model on facial gestures and lip movements, creating a system that realistically mimics these movements in sync with synthesized audio.

As for deployment, companies can access Tavus in two main ways. Half its users do so through its web dashboard, while the others integrate Tavus into their own systems via API or integrate it natively.

“We frequently see sales teams use and deploy Tavus directly through the platform as they’re able to efficiently generate large batches of videos for campaigns they have planned,” Favret said. “Other teams will use Tavus in a more programmatic manner, integrating it directly within their systems. This allows for users to create ‘event-driven’ workflows where a Tavus video can be generated and sent after a trigger.”

By way of an ‘event-driven’ example, if a sales prospect were to submit a form on a company’s website, the company could automatically generate and send a custom Tavus video to that lead using data that the prospect themselves input.

“This empowers companies to capitalize on opportune timing for maximum conversions, without having to wait for a team member to record the video,” Favret continued.

Mis-use

At the top end of the generative AI spectrum, we’re seeing the likes of Microsoft and Google slug it out to see who can get their respective smarts into the hands of businesses and consumers the quickest, a battle that Microsoft seems to be winning at present. At the same time, we’re seeing a whole host of generative AI startups come to the fore such as GlossAi, which is using AI to help businesses easily create shareable marketing skits, while Typeface is doing something similar for marketing copy and image-generation in the enterprise.

Specific to Tavus, there have been comparable companies out there for a few years already such as Windsor, which does something similar albeit with a heavy focus on ecommerce. And then there is London-based Synthesia, backed by a swathe of high-profile investors, which is more about creating digital avatars from text for use in training and how-to videos.

So it’s clear that even before all the generative AI hype of the past few months, there was a growing movement in that direction from the startup world, something that Tavus is now looking to capitalize on. Indeed, in its short life so far, Tavus has already garnered some fairly large customers such as real estate data company CoStar and French tech scaleup AB Tasty.

However, it’s worth considering potential mis-use of this kind of technology. For instance, is there anything stopping anyone from uploading a video of someone else talking, and then creating new videos from that template? Certainly, there’s no shortage of deepfake chicanery examples from across the video and voice spectrum. And some companies — such as South Park creators’ startup Deep Voodoo — are already raising VC cash for their deepfake endeavors. As this type of technology becomes more ingrained and normalized within society, the more questions there will be over the ethical implications of all this — even if the underlying intentions are well-meaning.

According to Favret, Tavus has a built-in feature that makes it more difficult to fool the system, requiring users to perform a voice-verification and record live on the Tavus platform.

“This means that users cannot upload videos of others talking,” Favret said. “Furthermore, users have all rights to their data and likeness, allowing them to delete or remove their videos at any time. We are also very intentional with how Tavus is used — we screen each use-case before a user starts with Tavus, and ensure the use case meets our community guidelines and is ethical. Security and ethics are incredibly important to us, especially given the youth of this technology in society.”

Show me the money

In terms of business model, Tavus offers a basic intro plan aimed at smaller businesses, costing $275 per month with restrictions such as the number of videos they can create. The custom “business” plan removes these restrictions, though there is no advertised set price in place — it basically tailors the price according to how a company intends to use it.

“Tavus plans are customized to a company’s specific use-case and needs, but on a high level we operate on a usage-based model where users are charged based on a combination of the number of seats they have, as well as the number of videos they generate,” Favret said.

Prior to this latest seed round, Tavus had raised a small amount of funding as part of its participation in the YC program back in 2021. Its full roster of seed-round investors include: Sequoia, Accel Partners, Index Ventures, Lightspeed Ventures, YC Continuity, SV Angel, Hack VC, Remus Capital, Mantis Capital, Liquid2 Ventures, Zillionize, Soma Capital, GTMfund, Terra Nova, and several undisclosed angel investors.

Tavus taps generative AI to power personalized videos with voice and face cloning by Paul Sawers originally published on TechCrunch