OCDevel
Walk
EpisodesResources

MLA 027 The AI Media Pipeline: Voice, Music, ComfyUI, APIs, and Finishing

Jul 14, 2025 (updated Sep 13, 2026)

Click to Play Episode

How to automate AI media end to end: clone your own voice on open TTS, pick music that's actually licensed, run ComfyUI graphs headless, design around fal, Replicate and provider queues, finish with ffmpeg, and stay inside licensing at every layer.

Multimedia Generative AI Mini Series

Take your next AI deep dive with youTake your next AI deep dive with you
Agents, transformers, or the topic you keep putting off. Gnothi turns what you want to learn into a series for your podcast app.Agents, transformers, or the topic you keep putting off. Gnothi turns what you want to learn into a series for your podcast app.Create an AI series →Create an AI series →

Resources

Resources best viewed here
Loading...

Show Notes

Companion show. This episode is the overview of the media pipeline. For weekly, hands-on coverage of the video half, from a first usable clip to scenes that cut together, listen to AI Video Generation on Gnothi.

Pipeline, not prompt

Once you need thirty clips with the same character, a narrator who sounds identical every episode, a matched music bed, word-accurate captions and a platform-safe export, the prompt is one node in a graph and the graph is the product. The engineering lives in the edges: how one model's output becomes the next model's input, how failures retry, what each job cost, and whether the run is reproducible next week. Model choice at the nodes is covered in the two sibling episodes; this one covers everything else: voice with ElevenLabs and Qwen3-TTS, licensed music, ComfyUI on your own card, the fal and Replicate APIs, ffmpeg assembly, and licensing.

Voice: cloning, open TTS, and consent

ElevenLabs remains the reference point (current flagship Eleven v3, character-based pricing). Professional voice cloning is locked to the requester's own voice behind a live voice check, and the terms require consent attestation for any uploaded voice. On the open side, Breeze TTS 2 topped the open-weights column of the Artificial Analysis speech arena in August 2026, ahead of Fish Audio's S2 Pro; the code is Apache 2.0 but the weights are research/non-commercial, so it is not a commercial self-hosting option. The working set for programmers: Qwen3-TTS (Apache 2.0, 0.6B/1.7B, cloning from seconds of reference audio; the preset-speaker variant does not clone, see my Qwen3-TTS voice cloning guide), Chatterbox (MIT, emotion control, watermarked output), and Kokoro (82M parameters, Apache 2.0, faster than real time on CPU). Fish Audio plays both sides with open weights and a cheap hosted API. Quantized Qwen3-TTS runs podcast-length synthesis on CPU-only instances; see Quantized Qwen3-TTS on CPU and the broader open-source TTS roundup. Hosted alternatives for prototyping: OpenAI text-to-speech and Gemini speech generation.

Consent is the legal boundary. Tennessee's ELVIS Act added voice to right of publicity; the federal NO FAKES Act cleared Senate Judiciary in June 2026; EU AI Act Article 50 transparency duties apply from 2 August 2026; Denmark is amending copyright law to cover a person's face and voice.

Music and sound effects

Warner settled with both Suno and Udio; Universal settled with Udio, which became a no-download walled garden; UMG and Sony are still litigating against Suno, whose terms now grant commercial rights rather than ownership to paid subscribers. Eleven Music is trained on licensed data via Merlin and Kobalt deals and cleared for commercial use on self-serve plans, excluding film, TV and larger games; ElevenLabs sound effects are cleared on any paid plan and support loops. Google exposes Lyria and Lyria RealTime through the Gemini API. Open models: ACE-Step 1.5, YuE, Stable Audio Open (community license, best open option for short effects), HeartMuLa, and Meta's MusicGen, which is non-commercial.

ComfyUI and local generation

ComfyUI is a workflow runtime with a GUI for designing graphs. Comfy raised $30M at a $500M valuation in April 2026 and ships a desktop app, Comfy Cloud, and API nodes that call paid providers from inside a local graph. Programmatic use is the same /prompt endpoint and websocket the front end uses: export the workflow in API format, patch fields, post, poll. Wrappers like comfyui-api and comfy-pack turn a graph into a scalable service. Alternatives: SwarmUI, InvokeAI, the Krita AI plugin.

Hardware: full-precision Flux.2 and Qwen-Image do not fit consumer cards; fp8 and GGUF quantization bring them to 16 to 24 GB. For video, the open Wan releases lag the API versions; the 5B variant does 720p on 24 GB (about 8 GB with Comfy offloading), the 14B variant officially wants 80 GB at full precision and needs GGUF to be consumer-viable, and Wan2GP targets low-VRAM cards. Rule: local for iteration, cloud for volume.

APIs and aggregators

Start with an aggregator, move to a provider API only for a feature or price it lacks. fal is queue-first: submit, get a request ID, poll or webhook, per-output pricing on popular models, per GPU-second for custom deployments. Replicate has the broader catalog beyond image and video, bills per second of compute for open models, packages custom models with Cog, and joined Cloudflare with the same API. RunPod serverless is the raw GPU option for a custom ComfyUI graph. Provider APIs have converged on the same shape: Veo via the Gemini API (billed per output second, audio included), Kling API (post, store task_id, poll /v1/tasks), Runway API, ElevenLabs API.

Design rules: every generation is a job in a durable queue keyed on a hash of inputs, model and seed; store the provider's request ID next to your job ID; honor 429 retry-after with a token bucket per provider; persist prompt, seed, inputs and outputs in object storage; route to a second provider on 5xx. Cost per usable second is list price times your rejection rate.

Assembly and finishing

ffmpeg is the programmer's editor: concat, overlay, sidechain ducking, caption burn-in, crop to 9:16, loudnorm and export. Human-in-the-loop editors: DaVinci Resolve (free version is a real editor; Studio unlocks most Neural Engine features), Descript with its transcript-as-timeline and Underlord assistant, and CapCut for short-form auto-captions, with a caution about its June 2025 terms change. Upscaling: Topaz retired Video AI for the subscription Topaz Video with the Astra model; open-side, SeedVR2 is single-step, runs on 8 GB and plugs into ComfyUI; Real-ESRGAN for clean stills; RIFE for frame interpolation. Captions: WhisperX gives word timestamps within about 50 ms via forced alignment plus diarization, emitting SRT/VTT; generate styled word-pop overlays from its JSON. Delivery: 1080x1920 9:16, H.264/AAC, roughly 10 to 12 Mbps, 30 fps; YouTube recommended upload settings; target -14 LUFS integrated with a -1 dBTP ceiling as the last pipeline step.

Licensing across the stack

Five layers, and the output is only as clean as the dirtiest node. Weights: Apache/MIT models (Qwen3-TTS, Kokoro, Chatterbox, open Wan) are clean; FLUX.2 dev is non-commercial without a separate license; Stability's community license allows commercial use under a revenue threshold; MusicGen is non-commercial. Output: the Copyright Office holds that prompts alone are not authorship, and the Supreme Court denied cert in Thaler v. Perlmutter in March 2026, so keep evidence of the human selection and editing. Training data: licensed models are the safe path while label suits continue. People: right of publicity, get written consent. Disclosure: YouTube auto-labels via SynthID and C2PA content credentials since May 2026, and labels on Veo and C2PA-stamped content are permanent. Attach credentials and disclose.

Two pipelines

Social clip (30 s, 9:16, run 200 times): character sheet from an image-editing model stored with prompt and seed -> templated script -> Qwen3-TTS narration with word timings -> per-shot image-to-video jobs via fal keyed on input hash, webhook completion -> cached licensed music bed -> ffmpeg concat, duck, styled captions from timing JSON, crop, loudnorm, 1080x1920 export -> content credentials and disclosure -> human review queue.

Narrated explainer (8 min, 16:9, weekly): human-written script (where copyright rests) -> chunked TTS stitched with short silences -> LLM shot list with timestamps tagged diagram/image/video -> deterministic diagrams, styled images, a few video clips upscaled with SeedVR2 -> licensed music and SFX generated once -> Resolve or ffmpeg assembly, captions from narration timings, loudnorm, 1080p/4K export -> title, chapters from the shot list, disclosure, credentials. Both are the same graph with different shot counts and aspect ratios.

Related episodes

The companion show for the video half of this pipeline is AI Video Generation on Gnothi.

Transcript

A note on how this one was made. I generated this episode with Gnothi, my tool that takes a topic, researches it, writes the chapters, and reads them aloud. This episode is the overview. The dedicated Gnothi show on AI video generation goes further, from a first usable clip to scenes that cut together. O C devel dot com slash video. That's O C D E V E L dot com, slash video.

This is the third and last episode in the media generation sequence. The first was AI Image Generation and Editing in 2026, the second was AI Video Generation in 2026, and this one is the connective tissue: the AI media pipeline. Voice, music, sound effects, running open models on your own hardware in ComfyUI, driving all of it from an API, assembling and delivering the result, and staying inside licensing while you do it. It's written for a programmer who wants to automate, not for someone who wants to click around in a web app.

Let me say what I mean by pipeline, because it changes how you think about every tool in this episode. When you use an image or video model through its website, the unit of work is a prompt. You type, you wait, you look, you type again. That's fine for one picture. It falls apart the moment you need thirty clips with the same character, a narrator who sounds the same in every episode, a music bed that matches the cut, captions that land on the right frame, and an export that YouTube won't re-encode into mush. At that point the prompt is one step in a graph, and the graph is the product. The interesting engineering is in the edges: how the output of one model becomes the input of the next, how you retry when a generation fails, how you keep track of what cost what, and how you make the whole thing reproducible so you can run it again next week.

The previous two episodes were about picking the models at the nodes of that graph. This one is about everything else. I'll go through the pieces in roughly the order they show up in a real project: voice first, because narration is usually the spine of anything longer than a clip; then music and effects; then local generation and the hardware it takes; then the hosted APIs and how to design around them; then assembly and finishing; then licensing across the whole stack; and finally two example pipelines described out loud, one for a short social clip and one for a narrated explainer. I'll close by pointing you at the show that goes deeper on video.

So that's the frame. Voice first.

Text to speech has had the most dramatic open-weights story of anything in this sequence, and I have a personal stake in it because this episode is being read by a synthetic voice. Two years ago the answer was that ElevenLabs was the only thing that sounded like a person, and everything open sounded like a very good robot. As of this recording that is no longer true, and on some leaderboards it has flipped.

Let me start with ElevenLabs anyway, because it's still the reference point everyone measures against. Their current flagship speech model is Eleven v3, and the product has grown into a whole audio suite: speech, dubbing, sound effects, music, and a conversational agents platform. Two things about it matter for a pipeline. The first is pricing shape. You pay per character of text, with plan tiers that bundle a monthly character allowance and an API that draws from the same pool. So the cost of a narrated episode scales with script length, and a forty minute script is a meaningful number of characters. The second is voice cloning, which comes in two flavors. Instant voice cloning takes a minute or so of audio and gives you something usable in seconds. Professional voice cloning wants at least thirty minutes and ideally hours of consistent studio audio, and it's locked to your own voice. ElevenLabs makes you pass a live voice check, a sort of voice captcha, where you read prompts on the spot so they can confirm the person providing the samples is the person being cloned. You cannot professionally clone someone else even with their consent, and the terms require you to attest consent for any voice you upload. I'll come back to why that matters legally in the licensing section, but as a product decision it's a tell: the consent problem is the thing the whole category is nervous about.

Now the open side. The way I track this space is the public speech arenas, where humans listen to two anonymous clips and pick the better one, and models get an Elo rating like chess players. The reason to trust that over vendor demos is that vendor demos are the best five seconds they ever produced. On the Artificial Analysis speech arena, as of this recording, the leading open weights model is something called Breeze TTS 2, released in August of this year, which by their numbers scored above ElevenLabs' own flagship. It's a three billion parameter model that does cloning, voice design from a text description, and voice direction, meaning you can tell it how to deliver a line, and the reports say it runs on a twelve gigabyte card. I'll flag one thing: the code is Apache licensed but the weights are released for research and non-commercial use, so it's a model to try rather than one to self-host in a business. Behind it on that same board is Fish Audio's S2 Pro, and Fish deserves a mention on its own because they play both sides: their earlier S1 model was released under an open license and they also run a cheap hosted API, which for a lot of people is the actual sweet spot. You get open-weights economics without owning a GPU.

Below the leaders there are three models I'd call the working set, and I'd pick among them by what you're optimizing for. Qwen3-TTS from Alibaba, released in January, is the one I use. It's Apache licensed, it comes in small sizes, six hundred million and one point seven billion parameters, it clones from a few seconds of reference audio, and it does voice design from a description. One trap that's easy to hit: the model variant that ships with preset speakers does not do cloning. Cloning needs the base variant, and voice design is a third variant. The naming makes it easy to grab the wrong one. Chatterbox from Resemble AI is the other model I'd point a programmer at, because it's MIT licensed with no strings, it's about half a billion parameters, it clones from a few seconds, and it has an emotion exaggeration knob that helps with narration where you want a bit more life. There's a multilingual version and a faster turbo version. And then Kokoro, which is tiny, eighty-two million parameters, Apache licensed, runs faster than real time on a laptop CPU with no GPU at all, and sounds far better than its size suggests. It doesn't clone; it ships with a set of voices. If you need to synthesize a lot of text cheaply and you're not attached to a specific voice, Kokoro is the answer.

That CPU point generalizes. I run quantized Qwen3-TTS on cheap CPU-only cloud instances for my own audio, and it produces podcast-length output at a quality that's hard to tell from a GPU run, for very little compute. Three years ago that sentence would have been a joke. The reason it's possible now is that these models are small. A one or two billion parameter speech model is a rounding error next to the language models people are used to, and once you quantize it, the CPU can keep up with real time. So the hardware story for voice is: you probably don't need a GPU, and if you have one, almost any recent card with eight or more gigabytes handles the whole open lineup.

Two more hosted names worth knowing even if you go open. OpenAI has a TTS endpoint that's cheap and steerable with plain language instructions, which is handy when you want to say "sound bored" without training anything. And Google exposes TTS through the Gemini API with multi-speaker support, which is the fastest way to prototype a two-host dialogue. Neither clones your voice. The rule I'd give you: hosted for prototyping and for anything where latency matters, open weights for anything you'll run in volume or where the voice is yours.

Which brings me to consent. Cloning your own voice is fine. Cloning a voice you have written permission to clone is fine. Cloning anyone else is where the law has moved fastest. Tennessee's ELVIS Act added voice to the state's right of publicity back in 2024. The federal NO FAKES Act, which creates a right over digital replicas of a person's voice and likeness, cleared the Senate Judiciary Committee this June and is waiting on a floor vote as of this recording. In Europe, the AI Act's transparency article, article fifty, became enforceable on the second of August, and it requires machine-readable marking of AI output and visible labels on deepfakes. Denmark went further and is amending its copyright act to give every person rights over their own face and voice. If you're building a product that lets other people upload reference audio, you need a consent step, and you probably want to watermark your output. Chatterbox, for what it's worth, embeds an inaudible watermark in everything it generates, which is exactly the right default.

So that's voice. Now music and sound effects, which is a shorter story because it's mostly a legal story.

The two consumer names are still Suno and Udio, and what happened to them over the past year is the clearest example in this whole sequence of licensing eating a product category. Both were sued by the three major labels in 2024 for training on copyrighted recordings. Warner settled with both of them in late 2025. Universal settled with Udio, and the terms turned Udio into what people are calling a walled garden: you can generate and remix inside the app, but the settlement restricts downloading what you make, which for a pipeline is a dealbreaker. Suno retired its unlicensed models under the Warner deal and, as of this recording, is still in active litigation with Universal and Sony, plus a loss in a German court this summer over specific compositions. Suno's terms also shifted in a way that matters: paid subscribers used to be told they "own" their output, and the language now is that you're granted commercial rights, with monthly download caps on the lower tier. Practically, Suno still makes the most listenable full songs with vocals of the current crop, and if you're on a paid plan you can use them commercially. But you're building on a foundation that is being renegotiated in court, and you should keep the receipts for which plan you were on when you generated what.

The alternative that was built to avoid all of that is ElevenLabs' music model, launched last August with licensing deals with Merlin, which represents tens of thousands of independent labels, and Kobalt, the independent publisher. The pitch is that it was trained only on licensed data and every track is cleared for commercial use, with a carve-out: on self-serve plans the license excludes film, television, and larger game productions, which need an enterprise deal. For a podcast intro, a YouTube bed, or a marketing clip, that's the safest button to press. Their sound effects product is the same story: any paid plan, cleared for commercial use, and it does loops and controllable durations, which is what you actually need for ambience under a scene. Google has music in the Gemini API too, under the Lyria name, including a real time streaming version you steer with prompts while it plays, which is a fascinating thing to hook up to a controller even if it's not a production tool yet.

On the open side, music is a generation behind speech, but it's moving. The models to know are ACE-Step, which got a one point five release this year and is the fastest of the group; YuE, which generates full songs with vocals from a lyric sheet; Stable Audio Open, which is trained on Creative Commons audio and released under Stability's community license, meaning commercial use is allowed under a revenue threshold; and Meta's MusicGen, which is old now and, importantly, non-commercial. There's also a newer family called HeartMuLa that claims the top of the open heap; I haven't run it, so I'll leave it at a name. For sound effects, Stable Audio Open is the best open option, because effects are short and don't need the long-range structure that trips up open music models. If you want a foley hit or a room tone, it's fine. If you want a three-minute song with a chorus that comes back, the hosted tools still win.

So that's audio. Now the part that makes this a programmer's episode: running open models yourself, and the hardware that takes.

ComfyUI is still the standard for local generation, and the reason is that it stopped being a hobbyist tool and became infrastructure. It's a node graph: every model load, every prompt, every sampler, every upscale is a box, and you wire them together. In 2023 that looked like an obstacle. In 2026 it's the point, because a graph is a program. The company behind it raised thirty million dollars this April at a half billion valuation, ships a desktop app so you're not managing a Python environment by hand, runs a hosted cloud so you can execute a graph on their GPUs, and has what they call API nodes, which are nodes inside your local graph that call out to paid providers. So one workflow can generate a base image locally with an open model, send it to a hosted video model for animation, and bring the result back for an open upscaler, all in one canvas. The alternatives are worth one line each. SwarmUI puts a friendlier front end over a Comfy backend. InvokeAI is the polished commercial-feeling option with a proper canvas. The Krita AI plugin gives you generation inside a real painting program, which illustrators like. But if you're automating, you use Comfy, because the graph is the thing you can save as JSON and run headless.

That JSON is the whole API story. When you press queue in the ComfyUI interface, the front end serializes your graph and posts it to an endpoint called prompt on the local server, and there's a websocket that streams progress. That's the entire API. You export your workflow in API format, which is the same graph with the display metadata stripped, you patch the text and seed fields in code, you post it, you poll or listen on the socket, and you fetch the output image from the history endpoint. You can build a pipeline on nothing more than that plus a queue in front of it. When you need to scale, there are wrappers that turn a workflow into a horizontally scalable HTTP service, and several hosts will run your graph as a serverless endpoint so you pay per generation. The mental model is: ComfyUI is a workflow runtime that happens to have a GUI, and the GUI is for designing the graph, not for running it.

Now hardware, and I'll be concrete. For images, the open models most people run are the Flux family and Qwen-Image, and the full-precision versions of the big ones are thirty-billion-parameter-scale models that do not fit on a consumer card. What makes them usable is quantization: eight-bit floating point roughly cuts memory by a quarter to a third at a small quality cost, and the GGUF format lets you go to four or five bits and fit a model that was twenty-eight gigabytes into about eight. So a sixteen gigabyte card runs the good image models comfortably at reduced precision, and a twenty-four or thirty-two gigabyte card, which is what the top consumer cards ship, runs them well. For video, the reference open model is Alibaba's Wan line, and the nuance that matters: the versions they release as open weights, under Apache two, lag the versions they sell through their API. The current open Wan's small five billion variant does seven twenty p on a twenty-four gigabyte card with offloading, and Comfy's own offloading can squeeze it into eight, while the fourteen billion variant officially wants eighty gigabytes at full precision and only becomes consumer-viable through the same GGUF trick. And even then, a five second clip is minutes of generation, not seconds. That's the shape of local video: you can do it, it's slow, and you need patience or a rented GPU. Apple Silicon works for images and for all the speech models, and is painful for video.

The rule I'd give you is: local for iteration, cloud for volume. Your own card is where you learn the model, tune the prompt, find the seed, build the LoRA. When you need two hundred variations by tomorrow, you push the same graph to a hosted GPU. The graph is portable; that's the whole reason to have built it as a graph.

So that's local. Now the hosted APIs, which is how most automation actually ships.

There are two ways to call a generation model from code. The first is the provider's own API: Google's Veo through the Gemini API and Vertex, OpenAI's image and video endpoints, Kling's developer platform, Runway's developer API, ElevenLabs' API, and so on. The second is an aggregator that fronts hundreds of models behind one account and one billing relationship, and the two that matter are fal and Replicate. My default recommendation for a programmer is: start with an aggregator, move to a provider's own API only when you need a feature or a price the aggregator doesn't expose.

Fal is the one I'd start with for image and video, because it's built around a queue, and a queue is what you need. You submit a request, you get back a request ID immediately, and you either poll a status endpoint or give it a webhook URL that it hits when the job finishes. Queue time and cold starts aren't billed on their shared endpoints. Pricing is per output for the popular models, so a certain amount per image or per second of video, and per GPU-second if you deploy your own model on their hardware. The catalog is the point: the same three lines of code call Flux, the current Wan, Kling, Veo, and Seedream, and swapping the model is a string change. Replicate is the other one, historically the broader catalog, especially for things that aren't image and video, like speech, transcription, and upscaling, and it was acquired by Cloudflare late last year and now sits inside their AI infrastructure with the same API. It has the same shape: create a prediction, get an ID, poll or webhook, and it bills per second of compute on named hardware for open models and per output for the closed ones. If you want to package your own model, their Cog tool makes a container out of a Python file. RunPod is the third name, more of a raw serverless GPU with fast cold starts than an aggregator, which is where I'd put a custom ComfyUI graph.

The provider APIs all look roughly alike now, which tells you the pattern has converged: post a job, get a task ID, poll a tasks endpoint, download the result promptly because the signed URL expires. Kling's documentation says that almost verbatim. Veo through Gemini bills per second of output video, with a fast tier and a standard tier a few multiples apart, and audio included. Runway's API exposes their current generation models for text-to-video and image-to-video without going through the consumer app. The prices move every quarter, so I won't say numbers, but I will say the shape: a usable second of frontier video costs meaningfully more than a usable image, and a usable image costs meaningfully more than a thousand characters of speech. Budget accordingly, and remember that a usable second isn't a generated second. If one in three clips is a keeper, your real cost per second is three times the list price.

That leads to the design advice, which is the part I actually care about. First, treat every generation as a job in a durable queue, not a synchronous call. Your worker submits, records the provider's request ID next to your own job ID, and exits. A webhook or a poller marks it done. If your process dies mid-generation, you lose nothing, because the provider is still running it and you still have the ID. Second, make the job idempotent by keying it on the hash of the inputs, model, and seed, so a retry doesn't double-bill. Third, budget for rate limits by putting a token bucket in front of each provider, because they will throttle you, and the throttle usually comes as a four twenty-nine with a retry-after header you should honor rather than hammer. Fourth, store every input and output, including the prompt and the seed, in object storage with the job ID, because a month from now you'll want to regenerate a clip with one word changed, and that only works if you can reproduce the original. And fifth, fall back across providers. The same model is often available on two aggregators and the provider itself, and a small routing table that tries the next one on a five hundred error saves you from a Sunday outage.

So that's calling models. Now the unglamorous half of every project: assembly and finishing.

The generative part of a media pipeline is maybe a third of the work. The rest is editing, upscaling, captions, loudness, and export, and the tools for that are older and more boring and matter more. Start with the editor. If you're a programmer, ffmpeg is your editor, and I mean that literally. Concatenating clips, overlaying a music bed under narration, ducking the music when the voice comes in, burning captions, cropping sixteen by nine to nine by sixteen, normalizing loudness, and exporting to a platform's spec are all one-line filter graphs. If you can only learn one tool from this episode, learn ffmpeg's filter syntax. Everything else in this section is a nicer front end on the same operations.

For a human in the loop, three editors. DaVinci Resolve is the one I'd pick for anything that has to look good, because the free version is a real professional editor with color grading and audio mixing, and the paid Studio version is a one-time purchase that unlocks most of the AI features: voice isolation, magic masks, the transcription-based editing, hardware-accelerated encoding. Descript is the editor that treats the transcript as the timeline: you delete a sentence in the text and the video cut happens. Its AI assistant, which they call Underlord, will remove filler words, pick clips, and draft show notes from instructions, and for talking-head and podcast content it's the fastest editor in this group. CapCut is the default short-form editor with the best auto-captions, and I'll add one caution: they revised their terms in June of last year to claim a broad perpetual license over uploaded content, including biometric data, then said in a follow-up that user rights hadn't changed. Read the terms before you put a client's face through it.

Upscaling is where I'd spend real attention, because most generation happens at seven twenty p or lower and everything ships at ten eighty or four K. The commercial standard is Topaz. They retired the old Topaz Video AI product last September in favor of a subscription product called Topaz Video, and the model to know in it is Astra, which does creative upscaling, meaning it invents plausible texture that wasn't in the source. On the open side, SeedVR2 is the one to know this year: it's a single-step diffusion upscaler, it runs on an eight gigabyte card, and it plugs straight into ComfyUI. Real-ESRGAN is the old reliable for images and still fine for clean sources. And for frame rate, RIFE interpolation turns a sixteen frame per second generation into a smooth twenty-four or thirty, which is often the difference between "AI-looking" and not.

Captions are a solved problem, and the solution is Whisper plus alignment. Plain Whisper gives you timestamps that are off by up to half a second, which is fine for a transcript and terrible for word-by-word captions. WhisperX runs a faster Whisper backend, then forces alignment with a phoneme model, and gets you word-level timestamps within about fifty milliseconds, plus speaker labels if you want them. It emits SRT and VTT directly. The trick for social captions, the ones that pop up a word at a time, is that you already have the words and times, so you generate the styled overlay from that JSON rather than paying an app to do it. And if the narration came from your own TTS, some of the speech models will give you word timings for free, which skips transcription entirely.

Last, delivery. The short-form platforms all want ten eighty by nineteen twenty, nine by sixteen, H two sixty four with AAC audio, at roughly ten to twelve megabits per second, thirty frames per second unless it's fast motion. YouTube long-form wants the same codecs at higher bitrates, and if you upload four K you get the better encoder even when people watch at ten eighty. On audio, the number to remember is minus fourteen LUFS integrated, the loudness measure the platforms normalize to; YouTube and Spotify both land there, TikTok is around the same and Instagram slightly quieter. If you deliver louder, the platform turns you down and any clipping you baked in stays. ffmpeg's loudnorm filter does this in two passes, measure then apply, and I'd make it the last step of every pipeline with a true peak ceiling of minus one dB.

So that's finishing. Now the section I'd put on a sticky note above your monitor: licensing across the whole stack.

Every node in your graph has its own license, and the output is only as clean as the dirtiest node. Let me go layer by layer. Model weights first. Apache two and MIT models, which as of this recording include Qwen3-TTS, Kokoro, Chatterbox, and the open Wan releases, are clean: you can run them commercially, modify them, and ship the output. Then there's the family of "community" or "non-commercial dev" licenses. The Flux dev models are the important case: they're released under a non-commercial license, and if you want to use them in a business you request a commercial license from Black Forest Labs or use their paid pro endpoints. Stability's community license allows commercial use under an annual revenue threshold. Meta's MusicGen is non-commercial. The common failure mode is someone building a product on a dev model because the weights were on Hugging Face, which is not the same as being allowed to sell what they produce.

Second layer: the output itself. In the United States, as of this recording, purely AI-generated output has no copyright. The Copyright Office's position is that a prompt is an instruction conveying an idea, not authorship, and the Supreme Court declined this March to hear the Thaler case, leaving the human-authorship rule in place. What that means in practice is that you can use your generated clip, but you probably can't stop someone else from using it too, unless you've added enough human creative work, selection, arrangement, editing, to make the whole thing yours. For a pipeline, the practical implication is: keep the evidence of the human decisions. Your edit decision list, your rejected takes, your script. That's what a registration would rest on.

Third layer: the training data behind the model. That's what the label lawsuits are about, and the safe path is a licensed model like ElevenLabs music or Adobe's Firefly line, and the risky path is anything currently in litigation. That's not a moral judgment, it's a risk judgment about whether the product you're depending on will still exist in its current form next year.

Fourth layer: the people. If any human's voice or face is in the output, that's a right of publicity question independent of copyright, and it's where the law is tightening fastest, per the NO FAKES and ELVIS discussion earlier. Get consent in writing, keep it with the project, and don't clone anyone who hasn't signed.

And fifth, disclosure and provenance. YouTube has required disclosure of realistic synthetic content since last year, and as of this May it doesn't wait for you: it reads Google's SynthID watermark and C2PA content credentials in the file and applies a permanent label, and it can display a "captured with a camera" credential the other way. The C2PA standard is the machinery behind that: cryptographically signed metadata that records what tool made a file and what was done to it. Adobe attaches it in Firefly and Photoshop, Google's models watermark with SynthID, and the EU now requires machine-readable marking of AI output. My advice is to stop fighting it. Attach credentials, disclose, and let the label be there. The audiences who care will trust you more, and the ones who don't won't notice.

So that's licensing. Now let me put it all together by describing two pipelines out loud, start to finish.

The first is a social clip: a thirty-second nine by sixteen video with a consistent character, a voiceover, music, and captions, and the goal is that I can run it two hundred times with different scripts. Step one is the character sheet. I generate a hero image of the character with an image editing model, and then, using the same image as a reference, I generate the poses and expressions I'll need, and save them all with the prompt and seed. That's covered in the image episode; the point here is that they're inputs to the graph, stored in object storage, not one-offs. Step two is the script and the voice. A language model writes thirty seconds of narration from a template, and my TTS worker, running Qwen3-TTS with my cloned voice, turns it into a WAV and a word-timing file. Step three is video. For each shot in the script, an image-to-video call goes through fal to the current best model for that kind of motion, with the character reference image as the start frame; each is a job in my queue, keyed on the hash of its inputs, with a webhook that marks it done. Step four is the music bed: a single call to a licensed music API for a thirty-second instrumental with a mood tag from the template, which I cache and reuse across runs because it doesn't need to change every time. Step five is assembly in ffmpeg: concatenate the shots, overlay the narration, duck the music under it, generate word-by-word captions from the timing file as a styled overlay, crop to nine by sixteen if the model gave me sixteen by nine, run loudnorm to minus fourteen, and export at ten eighty by nineteen twenty, H two sixty four, ten megabits. Step six is provenance: attach content credentials and write the disclosure text into the upload metadata. The whole thing runs unattended, and the human step is a review queue where I thumb through the finished clips and reject the ones with a sixth finger.

The second pipeline is a narrated explainer: an eight-minute sixteen by nine video, one voice, diagrams and illustrative footage, the kind of thing you'd put on a YouTube channel every week. This is closer to what makes this show. Step one is the script, which is the most human step and the one I'd never automate fully; it's where the copyright rests. Step two is narration, the same TTS worker as before but with a longer script, chunked by paragraph so a failure re-runs one paragraph, not eight minutes, and stitched with a small silence between chunks. Step three is the visual plan: a language model reads the script and emits a shot list with timestamps, each shot tagged as either a diagram, an image, or a video clip. Step four fans out: diagrams render deterministically from code, images go to an image model with a house style reference, and the few real video clips go through the queue to a video model, with SeedVR2 upscaling anything that came back at seven twenty. Step five is a music bed and maybe a couple of sound effects from a licensed source, generated once and reused. Step six is assembly, which for this one I'd do in Resolve if a human is going to touch it, or ffmpeg if not: lay the narration on the timeline, drop each visual at its timestamp, cross-dissolve, duck the music, generate captions from the narration timings, loudnorm, and export at ten eighty p or four K. Step seven is metadata: title and description drafted from the script, chapters from the shot list timestamps, the disclosure box ticked, credentials attached. The thing to notice is that both pipelines are the same graph with different shot counts and aspect ratios, and that's the payoff of thinking in pipelines rather than prompts: you build it once.

Let me recap. Voice is the most mature open piece: cloning your own voice on an Apache or MIT model on modest hardware is normal now, hosted tools are for prototyping and latency, and consent is the boundary the law is drawing. Music is a licensing story: the labels have turned the consumer tools into licensed or walled products, the licensed-data generators are the safe button, and open music is a generation behind. ComfyUI is a workflow runtime whose GUI is for designing graphs you run headless, quantization is what makes the big open models fit on consumer cards, and local video is possible but slow. Hosted APIs have converged on submit, get an ID, poll or webhook, and your job is a durable queue with idempotent jobs, rate limiting, stored inputs, and provider fallback. Finishing is ffmpeg plus a few good models for upscaling and captions, with loudness and platform specs as the last step. And licensing is five layers, weights, output, training data, people, and disclosure, each of which can be the dirtiest node in the graph.

That closes the media generation sequence. If you want to go deeper on the video part, from getting a first clip you'd actually use to cutting shots together into a scene, the dedicated Gnothi show on AI video generation picks up exactly there; it's at O C devel dot com slash video.