Dialogue and Lip-Sync (Act II): Native-Audio Video Models vs. the Decoupled Voiceover-and-Lip-Sync Pipeline, ElevenLabs Voice Cloning, Sync.so, Hedra, Runway Act-Two, and Fixing the Sync That Won't Lock

Let's run the landscape first. Two big models moved this window, and the leaderboard shifted under them.

First, Google. The model Google teased back at its developer conference in May reached developers this week. Gemini Omni Flash entered public preview on June thirtieth. What makes it interesting isn't single-shot generation, it's conversational, instruction-based video. You can generate a clip, then tell it what to fix across turns, and it edits. It takes text, image, and video inputs, and it'll make or edit up to ten seconds of video with scene consistency, plus text and graphics that stay synced to what's happening on screen.

There are catches. There's no audio input at launch, and Google itself admits some scene-consistency challenges remain. Price is flat: ten cents per second of output, with a batch discount cutting that to five cents a second at scale. So a ten-second clip runs about a dollar. It carries invisible watermarking, and you'll find it in Google's AI Studio, the Gemini app, and Google's Flow filmmaking tool. The producer angle: this is Google's first surface built around iterative, tell-it-what-to-fix video editing at a flat per-second price. Go try a ten-second image-plus-text edit in Flow and watch whether a character's identity holds across refinement turns.

Shipping alongside it, Nano Banana 2 Lite, a text-to-image model that makes an image in about four seconds for roughly three and a half cents per thousand images at one-K resolution. For us, it's a cheap, high-throughput way to feed frames into an image-to-video pipeline.

Second, ByteDance. Seedance 2.5 hit its public-launch window. The keynote claims landed a couple weeks back, but as of July third the model is described as launching this week while still sitting in closed enterprise beta, with public access expected in the coming days. The reported rollout goes to ByteDance's own consumer apps first, Dreamina internationally and Jimeng in China, then CapCut in mid-July, then a third-party interface late in the month. A July ninth date is circulating, but that's rumor until ByteDance posts an actual notice.

The stated specs are big: native thirty-second single-pass generation with no stitching, up to fifty reference images, native audio, and a beta long-video mode stepping thirty-second scenes up to ninety-second drafts and then three-minute outputs. Treat the three-minute figure and that July date as reportedly. Pricing isn't final, but expect roughly what version two ran, around thirty yuan for a thirty-second clip. And there's a copyright cloud: the recognizable-scene capability reopens the likeness fight that drew cease-and-desists from Disney, Warner Brothers, Paramount, Sony, and Netflix earlier this year. ByteDance responded with content watermarking and filters blocking real faces and copyrighted characters, so longer clips are the headline, but that filtering may constrain what you can actually render.

On the leaderboard, the Artificial Analysis Video Arena still shows Seedance version two owning the audio-inclusive and image-to-video charts, while Alibaba's HappyHorse leads audio-off text-to-video. Runway's Gen four-point-five, which led at launch late last year, has fallen out of the top ten. A public Seedance 2.5 and Gemini Omni Flash could reshuffle all of it within weeks. And a footnote: Microsoft made a leading image model the built-in generator inside PowerPoint this week, a nice distribution win, small for a video crowd.

Alright. This is Act two of our studio ladder, and today we're making a character talk. We've built up a lot already. We've locked a character's look with a character sheet so the same face shows up shot to shot. We've talked about cost per finished clip and take ratios, so you know what a revision really costs. We've chained clips by feeding the last frame of one shot into the start of the next. We've locked a style with a grade. And last time we storyboarded a multi-shot scene with a shot list. Now we put words in a character's mouth, and we make the mouth match.

Here's the single most important idea for the whole episode. There are two fundamentally different roads to a talking character, and you choose between them per shot, not once forever. Let me lay them both out, because everything else today hangs off this fork.

Road A is native-audio video models, and it's one pass. You prompt a generator that produces the picture and a synchronized dialogue track in a single generation. The mouth is born matching the audio, because one model made both the sound and the image at the same time. Think Google Veo, ByteDance Seedance, Kling in its newer versions, and Grok Imagine Video. You type a description, and out comes a clip that already talks.

Road B is the classic pipeline, and it's decoupled, meaning the picture and the voice are made separately and joined later. You generate or choose a silent clip. You make a voice track on its own, either recording a human or using text-to-speech, which is software that reads text aloud in a synthetic voice. Then you run a dedicated lip-sync pass, and lip-sync just means making the mouth movement match an audio track. That pass reshapes only the mouth region of the footage you already have so it matches the voice. Tools here include Sync dot so, Hedra, Runway's Act-Two, HeyGen, and D-ID, plus a whole family of free open-weight tools like Wav2Lip, LatentSync, LivePortrait, and others you run inside a node-based workflow tool called ComfyUI.

So when does each road win? Road A wins for speed, for ambient realism, and for the simple job of just make me a talking clip. Because one model made everything, it generates matching room tone, sound effects, and micro-expressions across the whole face, not just the lips. It feels alive. But it loses the moment you need control. The model may reword or mis-say your lines, so you can't guarantee an exact script. You can't feed it a specific licensed or branded voice, because there's no slot to hand it a cloned spokesperson. You don't get clean isolated dialogue, you get a baked-together mix. And if you want to change just one word of audio, you have to re-roll the whole expensive, non-deterministic video, which means you might lose the shot you loved.

Road B wins for client work, full stop. You own the exact voiceover, whether it's a real human read or a consented clone. You can revise the script and re-sync without regenerating the picture. You keep a clean dialogue stem for your mix. And you can apply it to footage you already shot, or to a hero clip you've locked for consistency, which is exactly where your character sheet pays off. Road B's weakness is that it can look slightly less real across the rest of the face, since it's mostly moving the mouth, and it adds a second render step, so turnaround is slower.

The through-line, and I want you to remember this one sentence: native audio shrinks Road B down to nothing when you're just free-styling, but the moment a client or a real script is involved, you fall back to Road B for control. Most professional workflows are hybrid. You'll use both roads in the same project and pick per shot.

Picture a thirty-second product ad to make that concrete. The hero line, the one the client signed off on word for word, goes down Road B, with a cloned brand voice and a dedicated lip-sync pass so every syllable is exactly right. The background chatter in the cafe behind your character, the stuff nobody scripts and nobody will scrub frame by frame, you let a native-audio model improvise that on Road A. Same ad, both roads, chosen shot by shot. That's the mindset.

Now let me get under the hood, because understanding how lip-sync actually works will make every tool choice make sense. Let me define a couple of terms first. A phoneme is the smallest unit of spoken sound, like the p in pat. A viseme is the visual mouth shape for a phoneme, the way your lips actually look when you make that sound.

Here's the key insight. There are many phonemes, but far fewer visemes, because several different sounds look identical on the lips. The sounds b, p, and m all share the exact same closed-lip shape. Your eyes can't tell them apart from the mouth alone. That's why lip-sync is even possible to fake convincingly. The classic pipeline extracts phonemes plus their timestamps, a step called forced alignment, then maps each phoneme to its viseme, then drives the mouth shapes with smooth transitions between them.

Modern neural lip-sync works differently. Tools in the Wav2Lip family skip explicit visemes entirely. Instead they learn the mapping from audio straight to mouth pixels, trained against what's called a lip-sync expert, basically a second model whose only job is to judge whether the mouth matches the sound. If it's not convinced, the main model gets corrected. That's the trick that made neural lip-sync leap ahead.

There are two flavors of driving a lip-sync, and knowing which you need shapes your whole shot. Audio-driven means the thing reshaping the mouth is a sound file. You give the tool a face plus a voice file, and it moves the lips to the audio. That's Sync dot so, Wav2Lip, Hedra, Kling's lip-sync, HeyGen. Audio-driven is best when all you have is a script and a voice file. Video-driven, or performance-driven, means the input is a video of a human actually performing, usually your own webcam. The tool transfers your mouth, your expressions, your head turns, even your hand gestures onto the target character. That's Runway Act-Two and LivePortrait. Performance-driven is best when you need real emotional, gestural acting, when the timing and feeling have to be exactly right and you're willing to perform the line yourself.

A quick bit of lineage, because it explains a flaw you'll see everywhere. Wav2Lip shipped in 2020 under the title A Lip Sync Expert Is All You Need, and it's still the most influential open-source lip-sync project ever made. Sync dot so is its commercial descendant. But classic Wav2Lip only repaints a small, low-resolution box around the mouth, so that region often looks softer or blurrier than the rest of the frame. That's the signature flaw, and once you've seen it you can't unsee it. Keep it in mind, because half the tool market exists to fix it.

Let's talk voice, because in Road B the voice comes first and drives everything. The default platform here is ElevenLabs, which does voice, sound effects, and music. Let me walk the plans, and I'll spell the prices for you. The free plan is zero dollars a month with ten thousand credits, but no commercial rights and you must attribute, so it's practice only. Starter is about five or six dollars a month for thirty thousand credits, and this is the important one, it unlocks commercial rights, which is the minimum for any client or monetized work, and it includes Instant Voice Cloning. Creator is twenty-two dollars a month for about one hundred twenty-one thousand credits, and it unlocks Professional Voice Cloning, the high-fidelity clone made from longer samples. Pro is ninety-nine dollars a month for about six hundred thousand credits, and above that sit Scale at two hundred ninety-nine dollars, and Business at nine hundred ninety dollars a month.

Let me give you the credit math so those numbers mean something. On the main multilingual models, one character of text equals one credit, and the faster variants run about half a credit to one credit per character. A one-hundred-fifty-word voiceover is roughly nine hundred characters, so roughly nine hundred credits. That means a twenty-two-dollar Creator month, with about one hundred twenty-one thousand credits, gives you on the order of one hundred thirty of those reads before you pay overage. That's a lot of narration for the price.

Two clone types, and pick deliberately. Instant Voice Cloning needs about one minute of audio, it's good-enough quality, and you get it starting on the Starter plan. Professional Voice Cloning needs tens of minutes of clean studio audio, it's hyper-realistic, and it's Creator plan and up. The rule of thumb: use Professional cloning for a recurring brand voice you'll live with for a long time, and Instant cloning for a quick throwaway.

Some craft tips that'll save you real time. ElevenLabs released version three in February of this year, its flagship, covering seventy-plus languages, and it introduced Audio Tags, which are bracketed cues the model actually performs. You can write a cue for excited, for whispers, for sighs, for sarcastic, even for clapping, right into the text, and it delivers on them. Version three also handles multi-speaker dialogue with interruptions. Now, pacing. On every model except version three, you can insert pauses with break tags up to three seconds each. Version three does not support those break tags. Instead it uses its own pause tags, a short pause and a long pause written as cues. And go easy, because too many break tags in one generation can garble the audio.

Two more voice tips. If the model mispronounces a name or a brand, you can use phoneme tags to spell out the pronunciation using a phonetic alphabet, and that works on the English version-one model and the English fast variants. And timing: generate the voiceover first, read its exact duration, and cut the shot to the voiceover. Or trim and re-time the voiceover with pauses until it fits your shot. The voice drives the picture by default, and that's a habit worth burning in.

Now the legal and ethical part, and please do not skip this, because it's where people get sued. On any paid ElevenLabs plan you keep commercial rights to your outputs perpetually, even after you cancel. But note two things. ElevenLabs keeps a perpetual license to use your content to train their models, though they commit not to commercialize your actual voice on its own without permission. And music outputs are explicitly not guaranteed to be unique. Two different users can get near-identical tracks, which is a real risk for a client jingle. Sound effects, at least, can be opted out of that sublicensing.

The bigger issue is cloned voices. Tennessee passed a law called the ELVIS Act, which stands for Ensuring Likeness Voice and Image Security Act, effective July first of 2024. It's the first US law to name a person's voice as a protected property right, covering both a real voice and a simulation of it, and violating it is a criminal misdemeanor plus a civil lawsuit. At the federal level, the NO FAKES Act was reintroduced in April of 2025 but hasn't passed as of the middle of this year. And more than twelve states now have their own voice-cloning statutes. So here's the practical rule: clone only your own voice, or a voice you have explicit written consent for. A verbal sure doesn't meet the bar. Get a signed release that names the specific use case. ElevenLabs even builds a consent confirmation step into the cloning flow, and that's there to protect you as much as anyone. Think of it this way. The cheap, fast part of this job is generating the voice. The expensive part, the part that ends careers and lands lawsuits, is generating a voice you had no right to use. Get the paperwork first, every time.

Beyond ElevenLabs, know your alternatives. There's MiniMax, also called Hailuo, with a prompt-to-performance speech model. There's PlayHT, Cartesia, Fish Audio, OpenAI's text-to-speech, and Microsoft's Azure Neural text-to-speech, which is worth a special note because Azure ships viseme output right alongside the audio, which is handy if you're driving lip-sync yourself.

Okay, let's go tool by tool through the dedicated lip-sync options, Road B, with prices for the ear. Sync dot so, formerly Sync Labs, is the category reference. It's audio-driven, and it works on live-action, on three-D, and on AI footage. Its cheaper model, lipsync-2, costs four cents per second at twenty-five frames per second, and it's the budget workhorse. Its quality pick, lipsync-2-pro, costs about one point seven times as much, so roughly seven cents a second, and here's why it's worth it. It uses diffusion-based super-resolution instead of the old mouth-box approach, so it goes up to four-K, it preserves teeth and beards and freckles, it handles a larger face region up to three hundred fifty by three hundred fifty pixels, it does obstruction detection so a hand passing over the mouth doesn't break it, and it does active-speaker detection, meaning in a shot with several people it automatically applies sync only to the person actually talking. No per-speaker training needed. There are older beta models up to about thirteen cents a second. Subscriptions run five dollars for Hobbyist, nineteen for Creator, forty-nine for Growth, and two hundred forty-nine for Scale, and you can reach it through several hosting platforms.

Hedra, and specifically its Character-three model, takes a photo or image and turns it into a talking avatar. It's audio-driven at the phoneme level, and it's what people call omnimodal, meaning it drives the mouth plus micro-expressions, eye contact, and head movement, so the whole face speaks, not just the lips. Reviewers regularly call its lip-sync best-in-class for a single portrait. Pricing: a free tier with one hundred credits and a watermark, Basic at fifteen dollars a month for fifteen hundred credits, Creator at thirty dollars for fifty-four hundred credits, and Professional at seventy-five dollars for fourteen thousand four hundred credits. At seven-twenty resolution, Character-three costs about six credits per second, so a thirty-second talking head is about one hundred eighty credits, which means the thirty-dollar Creator plan buys you roughly eleven minutes of seven-twenty video a month.

Runway Act-Two is the performance-driven flagship. You capture a full-body performance, your facial expressions, and your hand gestures from a driving video, usually your webcam, and it transfers all of that onto a character image or video. The lip-sync comes automatically from the driving video's speech. There's a gesture-control setting that adds hand and body movement even when you're driving a still image. Runway also added Voices to Act-Two, so you can feed it generated speech, not only your own recording. Its predecessor, Act-One, did face and expression transfer, and Act-Two adds body, gesture, and better sync on top. It's priced through Runway credits. Reach for it when the emotion and timing have to be exactly right and you want to act the line yourself.

Kling actually straddles both roads. It has a dedicated lip-sync feature where you upload audio and it syncs a character's lips, running about eight cents a clip through aggregators. And it has native simultaneous audio-visual generation, added in version two-point-six and improved in version three this year. Its consumer plans are free, then about seven dollars a month for Standard, twenty-six for Pro, and sixty-five for Premier. A five-second Standard generation is twenty credits, so four credits a second, and Pro mode is thirty-five credits per five seconds, so seven credits a second.

A few more. Hailuo, from MiniMax, in its oh-two version does not generate audio or lip-sync on its own, so you pair its silent ten-eighty-p video with MiniMax's separate speech models and a lip-sync pass, which is textbook Road B. A ByteDance-adjacent tool called OmniHuman does audio-driven digital humans with body language. HeyGen is an avatar platform, strong for photoreal recorded avatars and for large-scale video translation and dubbing. Its pay-as-you-go pricing starts at five dollars, where one dollar buys about a minute of standard avatar video, its higher-tier avatar runs four dollars a minute at ten-eighty-p, and translation runs about two dollars per source minute. D-ID focuses on real-time conversational avatar agents with the cheapest entry point, under five dollars a month billed annually. So think HeyGen for expressive recorded avatars and dubbing at scale, and D-ID for live interactive agents at the lowest entry price.

And if you want free and do-it-yourself, the open-weights world inside ComfyUI is rich. Wav2Lip is fast with accurate sync but low-res mouth, so you pair it with a face-restoration tool like CodeFormer or GFPGAN. LatentSync uses audio-conditioned latent diffusion for higher fidelity. LivePortrait is high-fidelity and emotion-aware, and it's video-driven. SadTalker makes a fast talking head from a single image. MuseTalk runs in real time, and InfiniteTalk handles long-form. The trade-off in one breath: SadTalker is fast, LivePortrait is premium, and Wav2Lip plus a restoration pass gives you accurate sync but needs that extra cleanup step.

Let me cross over to Road A, the native-audio frontier, so you know the standings. Google Veo three-point-one launched last October with a four-K upgrade in January. It generates video and audio in a single pass, synchronized dialogue, sound effects, ambient audio, and music at forty-eight-kilohertz stereo, with lip-sync accuracy reportedly under one hundred twenty milliseconds, and it does native vertical framing. It's arguably best-in-class for a talking head straight to camera. On pricing, the Fast tier is about fifteen cents a second, Standard and Quality about forty cents a second both with audio, and a Lite tier around three to five cents a second at seven-twenty with no audio, and there's no free tier. Note that the original Veo three is deprecated and shuts down June thirtieth, so migrate to three-point-one. The weak point: you have to explicitly prompt for speech, and Veo can be finicky about exactly which words it says, so exact-script control is where it struggles.

ByteDance Seedance two-point-oh launched in February with a unified multimodal architecture that takes text, image, audio, and video inputs at once, with native audio and lip-sync, two-K output, multi-shot storytelling, and director-level camera control, and it currently leads text-to-video-with-audio on the arena. Kling in versions two-point-six and three does native simultaneous audio-visual generation, and version three adds up to six connected shots at four-K sixty frames per second. Grok Imagine Video one-point-five from xAI shipped a preview at the end of May. It does image-to-video, or text, with native synchronized audio, so music, sound effects, and lip-synced dialogue in one generation, at four-eighty-p or seven-twenty-p, up to fifteen seconds, and it recently added multilingual dialogue in Japanese, Spanish, French, and Chinese with lip-sync matched to the language.

But remember where Road A still fails for client work, because it's the same list every time. You can't pin the exact script, the models paraphrase or mumble. You can't inject a specific licensed or branded voice, there's no clone input. Revisions mean re-rolling the whole non-deterministic clip, which is costly at forty cents a second and risks losing the shot you liked. You get a baked mix, not clean dialogue stems. And voice identity isn't consistent across separate generations. So use native audio for spec work, social, and B-roll, and use dedicated lip-sync for locked client lines. And go bench these yourself on the Artificial Analysis Video Arena, where rankings come from blind human A-B votes and churn monthly, because you want to test on your specific face, lighting, and language.

Now let me hand you the copyable end-to-end workflow, the one you'll actually run. Six steps: script, voiceover, visual, sync, check, fix.

Step one, script and board the line. Keep lines short, because sync drift grows with line length. Write phonetically clean lines, and avoid tongue-twisters and dense clusters of hard consonants.

Step two, voiceover first. Record a human, or generate or clone in ElevenLabs. Lock the delivery using version-three Audio Tags for emotion, break tags on the non-version-three models for pacing, and phoneme tags for tricky names. Export a clean voice file. The voiceover drives the edit, its duration sets your shot length.

Step three, get the visual. Either, on Road A, generate a native-audio clip and you're basically done. Or, on Road B, generate or select a silent hero clip that's front-facing, well-lit, with the mouth clearly visible and minimal head motion, ideally reusing your locked character frame from the character sheet.

Step four, the lip-sync pass, for Road B. Feed the footage plus the voice file to Sync dot so's pro model for best detail retention, or Hedra Character-three for the best whole-face result on a single portrait, or Runway Act-Two if you want to act it yourself via webcam. For multi-person shots, turn on active-speaker detection.

Step five, check on a big screen. Sync errors that are invisible on a phone are glaring on a television. Watch at one hundred percent and frame-step through the mouth.

Step six, fix in the edit, in the NLE. NLE stands for non-linear editor, the timeline software you assemble and finish in. Your options are DaVinci Resolve, which is free and pro-grade for editing, color, and effects, Adobe Premiere Pro, the industry standard, or CapCut, which is free and fast for vertical social with AI automation baked in. We're going to devote a whole upcoming episode to that assembly and finishing craft, so I'm only glossing it here.

A few real-world complications. For multiple lines, sync each line as its own short pass rather than one long take, because it's tighter and you can redo one line without touching the others. For multiple characters, generate and sync each separately, and in a two-person shot use active-speaker detection or mask each face. Version-three multi-speaker can voice a back-and-forth in one generation, but you still sync the faces individually.

And here's the editor's oldest trick, and I want you to treat it as a first-class technique, not a last resort: the cutaway. When a single syllable just won't lock, cut away on that word to a listener's reaction, or a hand, or a product insert, or a wide shot, then cut back. The ear fills the gap, the audience never notices. Cutaways also let you shorten any clip whose sync drifts near the end. Learn to reach for a cutaway on purpose. Here's a concrete case. Say your character has a six-word line and the fifth word, some hard plosive, pops the mouth no matter what you try. Don't burn ten more renders fighting it. Cut to their hand picking up the product on that exact word, then cut back for the last word. Total fix time, about thirty seconds in the edit, and it reads as intentional coverage rather than a patch.

One more mindset, and it saves real money. Rough everything in cheap modes first, Sync's four-cent-a-second model, Kling Standard, Veo Fast or Lite. Get your timing and blocking right. Then re-render only the keeper lines in the expensive high-detail model, the pro lip-sync or Veo Quality, exactly once. Here's the math: at forty cents a second, a ten-second Veo Quality clip is four dollars, and re-rolling it ten times for revisions is forty dollars. That's your cost-per-finished-clip lesson coming back around. Rough cheap, finalize expensive.

Finally, the pitfall, the sync that won't lock, and how to recognize and fix each version, because you will hit these. If the mouth reads as synced but the words are mushy and soft, your source audio has weak consonant attack, so re-record or re-generate the voiceover with clearer diction and a touch of compression that boosts consonants, and slow the read a little. If you get the uncanny flappy mouth, where the jaw over-opens out of proportion to the speech, that's a classic older-model flaw, so switch to a diffusion model like the pro lip-sync or LatentSync, or to Hedra, reduce audio-gain spikes, and pick a source frame where the mouth starts closed and neutral.

More of them. Garbled teeth or a warping jawline mean the mouth region can't render fine dental detail, so use the pro model's diffusion super-resolution, which was literally built for teeth and beard fidelity, or restore the face region afterward. Drift over a long line, where it starts synced and ends off, is accumulated timing error, so shorten the line, split one long sentence into two synced clips, or add micro-pauses to the voiceover so the model re-anchors. A visible pop or lip distortion on p and b sounds is a plosive overloading the model, so de-plosive the voiceover with a pop filter first. A mouth box that's visibly softer than the rest of the face is that signature Wav2Lip flaw, so use a model that preserves the full-resolution region, or upscale just the face, or feed a higher-res source. And a bad source shot, a profile, backlit, or moving face, causes perpetual mis-sync, because audio-driven models need to see the mouth. The real cure there is prevention: front-facing, evenly lit, low head motion.

Your recognition heuristic, and I'll leave you on this. Watch the corners of the mouth and the closure on b, m, and p sounds, because that's where fakes break first. And always do your quality-control pass on a large display, because phones hide everything. Get the voice right, choose your road per shot, rough it cheap, finalize it once, and cut away when a word won't cooperate. That's dialogue and lip-sync. Next time, we take everything we've generated and assemble it into a finished piece in the edit.