Good evening, everyone, and welcome to Week 8. For seven weeks, almost everything we generated was silent. Even the beautiful clips in the last lecture, the minute-long Helios videos and the Wan-Dancer dances, needed someone to add the sound afterwards. Tonight is about the moment that changed. Generated video now speaks, sings, breathes and makes its own footsteps, in one pass. Characters can now hold a conversation with you in real time. And that raises an old question in a new form: who is the performer? Tonight has more videos than any lecture so far, so please make sure you can hear the speakers.
Let's start with a game. I'm going to play a five-second clip with the sound off. [Play muted.] Two characters, a dark room, a robot. Turn to your neighbour for twenty seconds: what is being said? What do the voices sound like? What else would you hear in that room? Keep your guesses. Notice that you are already doing what film audiences always do: your brain is filling in sound from the image.
Now with the sound on. [Play with sound.] How close were you? Here is the point for tonight: nobody recorded these voices, and nobody synced them. The picture, the two voices, the reverb of the room and the timing of every word came out of one model, in one generation. This clip is from Ovi, an open model released by Character.AI in 2025. A year and a half ago, making this would have taken four or five separate tools and an editor.
Here is the map for tonight. Story one: sound is no longer post-production. Since Google's Veo 3 in May 2025, the leading video models generate picture and sound together, and this month the open models caught up. Story two: lip-sync grew into acting. The question is no longer 'do the lips match', but 'is this a good performance, and how do I direct it'. Story three: characters that listen. Real-time avatars now react while you are still speaking. Story four: who is the performer? An AI 'actress' has been cast in a feature film, and the actors' union has written new rules. We'll have presentations and a break after story two, and we'll finish with a debate.
A quick recap, because tonight builds on almost everything. Week 3 taught us that models see tokens, not pixels; tonight sound becomes tokens too. Week 6, Yaofu's quantization talk, and Week 7's Self Forcing and KV cache are exactly the tricks that make real-time characters possible, so story three is where those lectures pay off. And Week 5's question, who owns a model you adapt, becomes much sharper when the thing being copied is a person's voice and face.
Before the technology, one idea from film theory, especially for the art students. The French composer and theorist Michel Chion, in his book Audio-Vision from 1990, coined the word 'synchresis', from synchronism and synthesis. When a sound and an image happen at the same instant, your brain welds them into one event, even if they have nothing to do with each other. That is why Foley works: a sound designer snaps celery for a breaking bone, and you believe it. It is also why bad AI lip-sync is so painful: the welding is automatic, so when the timing is off, everyone notices. Broadcasters measured this: in the ITU's tests, viewers start to notice when sound arrives about 45 milliseconds early or 125 milliseconds late. At 24 frames per second, that is one to three frames. Synchronization is not a detail. It is the whole illusion.
Story one. For a hundred years of cinema, sound was something you added after the picture: dialogue recorded or re-recorded, Foley performed, music composed and mixed. AI video copied that workflow at first. This story is about how that changed in about eighteen months, and what is happening inside the model when it does.
Here is how you would have made a talking scene a year ago, and honestly how many of you made Assignment 1. Generate a silent clip. Generate or record the voice. Run a lip-sync model like Wav2Lip, from 2020, which repaints the mouth to match the audio. Run a video-to-audio model like MMAudio to add footsteps and room tone. Then mix it all in an editor. Five tools, and every hand-off is a chance for something to drift: the voice is calm but the face is angry, the footsteps land a frame late, the mouth moves but the cheeks don't.
The timeline. Lip-sync as a repair step goes back to Wav2Lip in 2020. At the end of 2024, MMAudio made video-to-audio Foley fast, about a second for an eight-second clip. Then in May 2025 Google released Veo 3, the first major model to generate picture and sound jointly, and its demo clips went viral within days. Within a few months the open-source world answered with Ovi from Character.AI, then LTX-2 from Lightricks with open weights in January this year, and ByteDance's Seedance 2.0 in February. In the last three months: MiniMax H3, Kandinsky 6.0, Kling 4.0 and Vidu Q4. Native sound is now the default, not the feature.
This is one of the clips Google showed when it launched Veo 3 in May 2025. [Play.] An old sailor, a line of dialogue, wind, the sea. None of it was recorded, and none of it was synced by hand. Google's model card describes Veo 3 as a latent diffusion model that denoises audio latents and video latents together. That one sentence is the whole of story one.
Three short Veo 3 clips, three kinds of sound. [Play each.] Effects: every toss of the wok has its sizzle and clatter. Music: the bow and the note move together. Dialogue: a detective interrogating a rubber duck, which only works if the comic timing works. Vote with a show of hands: which sound welds to the picture best? Usually it's the effects, because they are tied to one visible event; dialogue is the hardest.
Here is who ships native audio now. On the closed side: Google's Veo 3.1, ByteDance's Seedance, Kuaishou's Kling 4.0 and ShengShu's Vidu Q4, which came out two weeks ago with voice references. On the open side, and this is the big change this year, you can download and run LTX-2 from Lightricks, Ovi from Character.AI, MiniMax H3, and Kandinsky 6.0, released this month under the MIT license. For your assignments, this means you can now get sound and picture together without paying for an API, if you have a good GPU.
Three open models from the last six months. [Play each.] Kandinsky 6.0, released this month under MIT, here a dog in a living room with room sound. MiniMax H3, a 33-billion-parameter single-stream model, with stereo sound. And Baidu's NAVA, only 6.3 billion parameters, with two chess players talking; it currently has the best lip-sync score among open models. You can download all three. Do check each license before using them commercially; they differ.
And from the closed side, two clips from ByteDance's official Seedance 2.0 page. [Play both.] On the left, a Chinese ink painting comes alive, and the hooves and wind are generated to match the style. On the right, a concert pianist: hands, music, and the reverb of the hall together. Seedance accepts up to nine images, three videos and three audio clips as references, which is why it's popular with commercial studios.
Now under the hood, and I'll keep it visual. This sequence is adapted from Carnegie Mellon's Generative AI course, lecture 24 on audio. Raw sound is a pressure wave, recorded as tens of thousands of numbers per second: forty-eight thousand for studio audio. Nobody feeds that straight into a transformer. First, we run many short Fourier transforms over overlapping windows, which turns sound into an image: time on one axis, pitch on the other. That's a spectrogram, and the 'mel' version warps the pitch axis to match human hearing. Then, just as a video VAE compresses frames, an audio VAE compresses the spectrogram. In LTX-2, one second of sound becomes about twenty-five latent vectors of 128 numbers. Now sound is a short sequence of tokens, the same kind of object the video model already understands.
Here is that pipeline run on a real clip: the Veo 3 sailor we just heard. On top, the raw waveform: sixteen thousand numbers per second, so eight seconds is a hundred and twenty-eight thousand numbers. Below, the log-mel spectrogram: a hundred frames per second, eighty bands from low to high pitch. You can read it. The horizontal stripes are the harmonics of his voice on vowels. The vertical bursts are consonants and splashes. The noisy texture between words is the wind and the sea. For art students: this is why audio models borrow image-model ideas. A spectrogram is an image of sound.
A worked example with real numbers, in the same style as last week's pixels-to-tokens slide. Ten seconds of 720p video at twenty-four frames per second is two hundred and forty frames. The video VAE compresses four times in time, so about sixty-one latent frames. Each latent frame becomes three thousand six hundred tokens, which we computed last week, so about two hundred and twenty thousand video tokens. The same ten seconds of sound, after LTX-2's audio VAE, is two hundred and fifty tokens. Roughly eight hundred and eighty times fewer. That is why LTX-2 gives picture a fourteen-billion-parameter stream and sound a five-billion one. Picture is dense in space; sound is dense in time.
There are four ways to wire sound and picture together, and every model you'll meet is one of them. One: picture first, then a second model listens to the video and adds sound. That's MMAudio, or Tencent's HunyuanVideo-Foley. Good for Foley, but the picture can never react to the sound. Two: sound first, then animate a face or body to it. That's the talking-head family, story two. Three, the most popular open design this year: two towers, one for video and one for audio, denoising at the same time and passing messages to each other in every block. Ovi, LTX-2 and Kandinsky 6 work this way. Four: throw all the tokens into one sequence and let a single transformer attend to everything. MiniMax H3 and daVinci-MagiHuman do this. It's simpler, but you can't give the two modalities different sizes.
Let's open up LTX-2, because its weights and paper are public. Two streams. The video stream is big, about fourteen billion parameters, and every video token knows where it is in space and time: x, y and t. The audio stream is smaller, five billion, and its tokens only know time. In every block, the two streams look at each other through cross-attention in both directions: video asks audio 'what is happening now?', audio asks video 'what should I sound like?'. And here's the clever bit: when they look at each other, only the time positions are used. A video token at two seconds attends most naturally to the audio at two seconds. That shared clock is how a door slam lands on the right frame. The paper reports up to twenty seconds with stereo sound, and about one point two seconds per diffusion step on an H100, which is eighteen times faster per step than Wan 2.2.
This is the real overview figure from the LTX-2 report, so you can see that my simplified diagram was honest. Read it left to right. Audio is turned into a mel spectrogram and compressed by an audio VAE; video is compressed by a video VAE. Text goes through a large language model, Gemma 3, and then through two small 'connectors', one for each stream. In the middle, the audio stream and video stream are stacked N times with the cross-attention arrows between them. At the end there are two losses, one for audio and one for video, trained together.
One more detail for the technical students, because it's the key to sync. Video tokens and audio tokens arrive at different rates. If you number them naively, token 30 of the video and token 30 of the audio are at different moments, so attention looks at the wrong sound. These two figures from MMAudio, at CVPR 2025, show the attention affinity between visual and audio tokens. On the left, default positions: the bright band drifts. On the right, positions scaled to real time: a clean diagonal, each frame paired with its own moment. LTX-2's 'temporal RoPE only' and Ovi's 'scaled RoPE' are the same idea.
Here's LTX-2 run in the other direction: the input is an audio file, a scream, and the model generates a face and body performing it. [Play.] Because the model has both streams, the same weights can go text to audio and video, image to audio and video, or audio to video, depending on which stream you hold fixed. That flexibility is a big reason open two-stream models are popular with artists: you can bring your own voice track.
Ovi, from Character.AI with Yale. [Play.] Two people, two voices, turn-taking, a car interior. The dialogue was written in the prompt; the model decided the timing, the voices and the faces. Ovi's design is the symmetric version of two towers: both towers have the same shape, the video one starts from Wan 2.2, and the audio one was trained from scratch on hundreds of thousands of hours of audio. It's open under Apache 2.0, so it's a good one for your assignments.
Here's Ovi's own figure. Two towers of exactly the same shape: video on the left, initialised from Wan 2.2, audio on the right, trained from scratch on a huge amount of audio. Because the towers match block for block, each fusion block can do self-attention, then cross-attention to the text, then cross-attention to the other tower. Notice the snowflake and flame icons: frozen parts versus trained parts. A practical lesson for anyone building models: reuse what already works and train only what's new.
How do you actually build one? The Kandinsky 6.0 report, published this month, spells out a recipe that most teams now follow. Step one: don't start from scratch; reuse a strong video model. Step two: train the new audio tower alone first, on text-to-audio, so it learns what the world sounds like. Kandinsky used about forty million audio tracks. Step three: join the towers and train them together on video clips that have meaningful sound, about seven million segments after filtering. Step four: polish, with fine-tuning, reinforcement learning, and distillation down to ten steps so it's fast. Baidu's NAVA follows a similar path from Wan 2.2.
The unglamorous truth is that data is the hardest part. Most online video is bad training data for sound: music pasted over vlogs, dubbed films, voice-overs where the speaker isn't on screen. Teams use a sync score, a model that checks whether lips match speech, to throw those clips away. Kandinsky keeps a clip with a person on screen only if its sync score is high. Second, captions now describe the sound, not just the image, with spoken words in quotation marks, which is why your prompts should do the same. Third, multi-speaker scenes need to know who is talking: NAVA attaches a voice fingerprint to each line of speech so two characters keep two different voices. For scale, NAVA trained on about fifteen million clips using a hundred and sixty H100 GPUs for about four weeks.
Practically, how do you prompt for sound? Think in five layers, like a sound mixer does. Picture, as before. Dialogue: who speaks, the exact words in quotation marks, and how they say it. Effects: sounds tied to visible actions, the wipers, the turn signal. Ambience: the always-on bed of the place, rain on the roof. And music: decide whether it's inside the scene, a radio in the car, or a score over the scene, and keep it under the voice. Two practical tips from people who use Seedance a lot: keep each spoken line short, and don't ask for big head turns while someone is talking, because motion and lip-sync compete.
Here are three prompts you can copy tonight. A dialogue scene: notice the exact words in quotation marks, how they're said, one effect tied to an action, the ambience, and 'No music', because otherwise many models add a score. A sound-led shot with no speech, where every sound is tied to a visible action. And music inside the scene, where we say the music is played on screen, and where the camera should be on the last note. Same five layers as the table.
Second game. You are the Foley artist now. [Play muted.] Write down every sound you would put in this shot, in order. Wheels on asphalt, leaves, the board hitting a crack, wind? Thirty seconds. [Play with sound.] Now compare. What did the model include that you forgot? What did you include that it didn't? In my experience, models are good at the continuous bed, like wheels and ambience, and less reliable on the exact, single events. That's where a human sound designer still adds the most.
Third game, a blind A/B test. Same burning grassland, same picture, sound added afterwards by two different video-to-audio models. [Play A, then B.] Hands up for A. Hands up for B. Why? The reveal: A is Tencent's HunyuanVideo-Foley from 2025, B is MMAudio from the year before. Tencent's paper reports better synchronization and semantic match for A on their benchmark, but notice the question you just answered is not 'which is more accurate' but 'which do I believe'. That's synchresis again, and it's what next week's evaluation lecture is about.
Can we measure sync without a neural network? Yes, roughly, and I did it on the wok clip for tonight. The green line is how much each video frame changes: motion. The orange line is how much new energy appears in the spectrogram: sound onsets. Many peaks coincide: the toss and the clatter happen together. This is the simplest possible version of what models like Synchformer learn to do.
Now slide the sound against the picture in ten-millisecond steps and score the match at each shift. The best match is about forty milliseconds from zero, one frame at twenty-four frames per second, well inside the range the ITU found viewers can't detect. That is exactly what SyncNet does, except SyncNet learns the features instead of using my hand-made ones. Keep this picture in mind for story two.
A term from film studies that's very useful when prompting: diegetic sound comes from inside the story world; the characters could hear it. Non-diegetic sound, like a film score, is only for the audience. [Play both.] On the left, Ovi's street saxophonist: diegetic. On the right, UniVerse-1, a 2025 model that stitched together a pretrained video model and a pretrained music model, plays a guitarist. Joint models are very good at diegetic music because they can see the instrument. They are much less controllable for score, so many filmmakers still add the score themselves.
Where does it still break? Six things to watch for in your own work. Two people talking: the wrong mouth moves, or both do. ByteDance itself lists multi-person lip-sync as an open problem. Motion versus mouth: a head turn or a laugh in the middle of a sentence and the lips lose track. Audio artefacts: metallic voices, clipped words, music swelling over dialogue. Language: quality varies a lot across languages and dialects. Exact events: the crash lands a few frames early, and thanks to synchresis, you notice. And voice continuity: the same character can sound different in the next shot, unless you give the model a voice reference, which is where story two begins.
Can you run it yourself? The numbers are moving fast. LTX-2 is about eighteen times faster per step than Wan 2.2 at similar size, because the audio stream is small and the video compression is aggressive. And in March, a team from JD, Fudan, Peking University and HKU published OmniForcing, which distills LTX-2 into a streaming model: a five-second clip with sound in under six seconds instead of three minutes, about twenty-five frames per second on one GPU. The tricks are exactly what Yaofu and I showed you: fewer steps by distillation, chunks with a KV cache, and quantization. Keep that in mind for story three.
For the sound designers and composers in the room: what changes? The model now gives you a first-draft mix that's already in sync, with room tone and footsteps for free, diegetic music that matches the instrument on screen, and temporary dialogue for pitching. What stays human? Sound design was never about adding every sound; it's about choosing what not to hear, where the silence goes, whose point of view we're in. The exact event, the hit on the cut. The score, the emotion the characters can't hear. And the decision about whose voice is used, which is story four.
For the room, two minutes with your neighbour. If the model writes the sound together with the picture, who is the sound designer? The person who wrote 'rain on the roof' in the prompt? The model? Or the thousands of Foley artists and recordists whose work trained it? There's no right answer yet, but you'll need one for your assignment credits.
Story two. Once a model can make a character speak, the question changes. It's no longer 'do the lips match the audio?' It becomes the question every director asks: is this a good performance, and how do I get a better one? This story walks through how talking-head models grew into acting models, how we measure sync, and how you direct them.
What is a performance made of? Let's break it into six parts. Voice: pitch, pace, breath, accent. Timing: pauses, interruptions, the beat before the punchline. Mouth: lips, jaw and teeth matching each sound. Face and eyes: brows, blinks, gaze. Body: head motion, shoulders, hands. And listening, what you do while the other person is talking, which actors will tell you is half of acting. The first models only did the mouth. Today's models attempt all six, and listening is the newest, which is story three.
The lineage, quickly. In 2020, Wav2Lip repaints only the mouth of an existing video. In April 2024, Microsoft's VASA-1 animates a whole face and head from one photo at forty frames per second. In February 2025, ByteDance's OmniHuman-1 does the full body, gestures and singing, trained on over eighteen thousand hours of human video. In mid-2025, MultiTalk and InfiniteTalk from the MeiGen-AI team handle two people and unlimited length. Then in August 2025, OmniHuman-1.5 does something that sounds like a director: a language model first reads the audio and writes a plan for the acting, and the video model follows the plan. And in 2026, with native audio-video, the voice itself is generated together with the face.
Before the machine learning, some phonetics, because it helps you judge results with your own eyes. Sounds are phonemes; the mouth shapes you can see are called visemes, and many sounds share one viseme. The ones that matter most are the bilabials, p, b and m, where the lips must fully close. If a character says 'Bob' and the lips never meet, everyone notices. F and v, where the teeth touch the lower lip, are hard for models. Open vowels are easy. And tongue sounds like t, d, k are mostly hidden, so models get them for free. When you review a generated performance, watch the p, b and m sounds. That's the fastest quality check there is.
How does a voice actually drive a face? The audio goes into a pretrained speech model, usually wav2vec 2.0 from Meta or OpenAI's Whisper. These models were trained to understand speech, so their features already encode which phoneme is being spoken, and a bit of emotion. We resample those features so there is one vector per video frame. Then inside the video transformer, every frame's tokens use cross-attention to ask 'what sound is happening right now?' A reference photo keeps the identity. VASA-1 and OmniHuman use wav2vec; LatentSync and MuseTalk use Whisper. The nice idea here is reuse: the video model doesn't have to learn phonetics from scratch, it borrows it from a speech model.
How do papers measure lip-sync? Almost all of them use the same tool: SyncNet, from Oxford in 2016. Take five frames of the mouth, a fifth of a second, and the matching fifth of a second of audio. SyncNet turns both into vectors, trained so that matching pairs are close and mismatched pairs are far apart. Now slide the audio backwards and forwards, up to fifteen frames each way, and find the shift where the distance is smallest. That smallest distance is Sync-D, lower is better. Sync-C is the confidence: how much that best shift stands out from all the wrong ones, higher is better. If the best shift isn't zero, the clip is out of sync by that many frames.
Here's how to read the scoreboard. On the lip-reading dataset LRS2, real video scores about 7.8 Sync-C and 6.7 Sync-D. Wav2Lip, from 2020, scores 7.8 and 6.4, a better Sync-D than reality. Did it beat reality? No. Wav2Lip was trained against a SyncNet-style judge, so it learned to please that judge. That's a general lesson for next week's lecture on evaluation: when a metric becomes a training target, it stops measuring what you care about. Below, on the newer Verse-Bench, the open joint models: NAVA from Baidu leads, then Ovi, LTX 2.3 and daVinci-MagiHuman, all close together. Only compare numbers within the same benchmark.
A game to feel the numbers. Three versions of the same sailor clip. I re-timed two of them. [Play A, B, C.] Vote: which one is in sync? The answer: B is the original. In A, the sound is 250 milliseconds late, six frames, and most of you probably saw it. In C, the sound is 120 milliseconds early, about three frames. Early sound is much easier to notice than late sound, which matches the ITU numbers: about 45 milliseconds early versus 125 late. Why the asymmetry? In real life, light arrives before sound, so our brains are used to late sound and suspicious of early sound.
Microsoft's VASA-1 from 2024. [Play.] The Mona Lisa, rapping. The trick is a learned face space that separates who you are from how you move: identity and appearance on one side, head pose and facial dynamics on the other. A surprisingly small diffusion transformer, about twenty-nine million parameters, generates only the motion, which is why it can run at forty frames per second on one RTX 4090 with about 170 milliseconds of delay. Microsoft never released the model or an API, explicitly because of the risk of impersonation. Remember that when we get to story four.
VASA-1's own figure, and the idea is worth stealing. On the left, training: the model learns to generate a motion latent, a compact code for head pose and face dynamics, from audio features. On the right, generation: your photo is encoded once into appearance and identity, then only the motion is generated, and a decoder turns appearance plus motion into frames. Because the motion code is tiny compared with pixels, it can run at forty frames per second. Generate the small thing, render the big thing.
Here is why VASA-1 matters for directing. [Play.] Same photo, same audio, four different emotional settings. Because motion is separated from identity, emotion, gaze direction and head distance become separate knobs. That's the beginning of directing a performance rather than generating one. Ask yourself as you watch: which one would you cast?
ByteDance's OmniHuman. [Play.] Now the whole body performs: hands on the keys, breathing, posture. The training trick is to mix conditions: some clips come with only text, some with audio, some with pose, so the model can use eighteen thousand hours of messy human video instead of a small clean set. Version 1.5, last August, adds something that looks like a director's note: a multimodal language model first listens to the audio and writes a structured plan, what emotion, what gesture, when, and the video model follows it.
OmniHuman-1.5 in a dramatic scene. [Play.] Don't watch the lips this time; they're fine. Watch the eyes, the pauses, where he looks away. That's the part the language-model plan is supposed to add: intention. Is it convincing? Where does it feel generic?
And here's how OmniHuman-1.5 adds intention. The authors borrow the psychology idea of two systems. System two, slow thinking: multimodal language-model agents listen to the audio, look at the image, read the text, and write a plan: emotion, action, shot, for each moment. System one, fast acting: a diffusion transformer with text, audio and video branches renders the plan. It's literally a director and an actor.
Singing is harder than speech: long vowels, breath, the body moving with the rhythm, and often an instrument. [Play both.] Left, Alibaba's Wan2.2-S2V, built for film: a singer on a ship in a storm, with the camera and body moving with the song. Right, Tencent's HunyuanVideo-Avatar: strumming hands and breathing from one photo and a song track. Watch the hands: are they playing the right rhythm?
Two people is much harder than one, because the model must decide which mouth each voice belongs to. [Play both.] MultiTalk, on the left, takes one audio track per person and learns to bind each track to the right face; Tencent's HunyuanVideo-Avatar, on the right, does duets and adds an emotion module. Listen to the turn-taking. Remember that ByteDance still calls multi-person lip-sync an open problem.
One of the most immediate uses is dubbing. [Play.] InfiniteTalk takes an existing video and a new voice track and regenerates not just the lips but the head, body and expression, for any length. Hollywood got here first: the 2022 thriller Fall used a company called Flawless to change more than thirty swear words so the film could get a PG-13 rating, without reshooting. They call it 'vubbing'. Ask yourself: when an actor's face now speaks Japanese, whose performance is it?
Here is a clip from Vidu's homepage. [Play.] A fortune teller, candlelight, a voice that suits her. Vidu doesn't say which model version made this one, but its new Q4 model, released two weeks ago, lets you pin the character, the wardrobe, the set and the voice, each with its own reference. That's the direction for all of these tools: from typing a wish to assembling a cast.
So how do you direct? Mostly with references. Identity: Vidu Q4, released two weeks ago, takes up to fifteen reference images to lock a character, costume and set. Voice: Vidu Q4 and Seedance accept reference audio, so a character sounds the same in shot two as in shot one. The most powerful option for artists: bring your own track. Record the line yourself, with exactly the timing you want, and use an audio-to-video model to act it. And for scenes with several people, tell the model which line belongs to whom. A rule of thumb: the more you specify by reference, the less you leave to chance, and the more consent you need from whoever's face or voice that is.
Here's a practical table for directing delivery. An emotion label alone, like 'angry', gives you the average angry face from the training data. Directors and actors talk instead about playable actions: what the character does. 'Angry but held in' means clipped speech, even pitch, a tight jaw, a fixed stare; you can write 'through clenched teeth, without raising his voice'. 'Nervous' means rising sentence endings, filler words, gaze darting away. Put the behaviour in the prompt, not just the label. This works for actors and, it turns out, for models.
Exercise, five minutes, in pairs. Here's one line: 'I never said you could leave.' Write three prompts that make it a threat, a joke, and a plea. For each, describe the voice, the face, and one gesture, using playable actions, not labels. If you have Seedance, Kling, Vidu or LTX-2 access on your laptop, try one. I'll ask two pairs to read their prompts aloud.
None of this is entirely new. In 2002, Andy Serkis played Gollum through motion capture, and people argued whether that counted as acting. In 2021, the documentary Roadrunner synthesised three lines in Anthony Bourdain's voice after his death, and the director's casual reply, that they could have an ethics panel about it later, caused an uproar. In 2022, James Earl Jones licensed his archived voice so Darth Vader could keep speaking. Also in 2022, ABBA Voyage opened in London: digital ABBA built by Industrial Light and Magic from about a hundred and sixty cameras of motion capture, more than four million visitors by this year. And last year, The Brutalist used AI to refine the actors' Hungarian vowels, and it became an Oscar controversy. The difference now is scale and cost: what took ILM five weeks takes one photo.
Let's pause here for this week's student presentations, and then a ten-minute break. A small challenge for the break: pick any demo from tonight, play it on your phone with the sound off, then on, and notice what changed in how you read the face.
Welcome back. Story three. Everything so far was offline: you write a prompt, you wait, you get a file. Story three is about characters you can talk to, live. That needs two things the offline models don't have: they must be fast enough, and they must listen, not just speak. This is also where Yaofu's quantization lecture and last week's Self Forcing come back.
Real time means two different numbers, and people mix them up. Throughput: can you make video at least as fast as it plays? At twenty-five frames per second, that's forty milliseconds per frame. Latency: after you stop talking, how long until the character responds? In human conversation, across ten languages studied, the typical gap between turns is around two hundred milliseconds. Here's the remarkable part: planning a spoken reply takes a human over six hundred milliseconds. So we don't wait for the other person to finish; we predict the end of their sentence and start preparing early. A real-time avatar must do the same. A model can have high throughput and still feel slow, if the first frame takes two seconds.
Here's where the time goes in a typical talking avatar today, as a pipeline. First the system must decide you've stopped talking, which takes a few hundred milliseconds. Speech recognition, about a hundred. The language model's first words, about four hundred. Text-to-speech, first audio, about a hundred and fifty. Then the video model's first frame, anywhere from under two hundred milliseconds to over a second depending on the model. Plus the network. Run in sequence, that's about one and a half seconds, seven times the human gap. The fixes: overlap everything by streaming, so each stage starts before the previous one finishes; predict the end of the turn; and make the video model work in small chunks. Or, the newest idea, merge the whole pipeline into one model.
Why can't a normal video diffusion model just talk back? Three multipliers. A typical sampler denoises thirty to fifty times. Classifier-free guidance, from Week 3, runs each step twice. And worst for conversation: in a bidirectional model every frame attends to every other frame, including the future, so the first frame isn't finished until the last frame is. That's seconds to minutes. Great for a film, useless for a conversation.
The good news is that you already know the three tricks. One, from last week: generate in small chunks, looking only backwards, and keep a KV cache of the past so you don't recompute it; Self Forcing trains on the model's own output so errors don't snowball. Two: distillation. Train a fast student to jump to the answer in one to four steps. Character.AI's TalkingMachines uses a method called DMD to go from twenty-four model evaluations to two. Three, from Yaofu's lecture: quantization, eight or four bits, so the same GPU does more work per millisecond. And a systems trick on top: run the transformer and the decoder on different GPUs so they work in parallel.
Here's what streaming looked like already in 2024, with VASA-1. [Play a part.] The interface takes a photo and live audio and the face responds as you speak. Up to forty frames per second at 512 by 512, about 170 milliseconds of delay, on a single gaming GPU. It works because VASA only generates a compact motion code, not pixels from scratch.
TalkingMachines from Character.AI, 2025. [Play.] A FaceTime-style call with a cartoon fox. Under the hood: they took Wan 2.1, the open image-to-video model, and turned it into a real-time avatar model. Distillation cuts each chunk from twenty-four model evaluations to two. Video is made in chunks of three latent frames, and the KV cache keeps two things: the very first chunk, which anchors who the character is, and the previous chunk, for continuity. That anchoring trick, keep the first frame forever, appears in almost every streaming avatar paper.
How do you turn a slow film model into a live one? TalkingMachines' figure shows it. Training clips are cut into chunks of three latent frames, and the reference image is always chunk zero. Each chunk is only allowed to look at itself, the first chunk and the previous chunk, a sparse causal mask. Then a slow bidirectional teacher, twenty-four steps, supervises a fast causal student that uses two. And as in last week's Self Forcing, the training data can include the student's own generated frames, so it learns to recover from its own mistakes.
And the systems trick from the earlier slide, in their own figure. Case one: a single GPU denoises a chunk, decodes it with the VAE, then sits idle. Case three: one GPU decodes while another already denoises the next chunk, a pipeline, like an assembly line. Case four adds more GPUs for the diffusion steps themselves. The red line, time to first frame, moves earlier each time. For engineers: real time is as much scheduling as modelling.
StreamAvatar, from Tencent Hunyuan and university partners, goes beyond the head: gestures and the upper body, live. [Play both.] They distilled the open Wan 2.2 model into a three-step streaming student: twenty-five frames per second with about 1.2 seconds of delay, on two H800 GPUs. And like TalkingMachines, it keeps the first frames permanently in memory, which they call a reference sink, so the person doesn't slowly turn into someone else.
The newest piece is listening. Avatar Forcing, from KAIST and NTU Singapore, at CVPR this year. It takes your audio and your head motion as inputs, so while you're talking, the avatar nods, smiles, or laughs at your joke. Technically it's diffusion forcing: it generates ten-frame blocks with a two-frame look-ahead and a rolling KV cache. The clever training idea: how do you teach 'good listening' without labels? They make a bad example by running the same model with the user's input removed, so it ignores you, and train the model to prefer the version that pays attention. About half a second of delay on an H100, nearly seven times faster than the previous system, and people preferred it over eighty percent of the time. [Play.] The left column is the real person, the middle is the previous method, the right is Avatar Forcing. Watch the right one while the user is talking.
Inside Avatar Forcing. The key is the input on the left: not just the avatar's own audio, but the user's audio and the user's motion, through a dual motion encoder. Then a causal generator writes the next block of motion, conditioned on the previous clean blocks and a reference frame, and a decoder renders it. The listening behaviour comes from that extra input, the other person.
Here's a scoreboard of real-time systems. Faces only, like VASA-1 and Hedra's Livatar, are very fast: Livatar reports a hundred and forty-one frames per second and 0.17 seconds on a mid-range A10 GPU. Systems that generate sound too, like OmniForcing and Hallo-Live, are around three-quarters of a second to one second. Full-body StreamAvatar is about 1.2 seconds on two H800s. And Wan-Streamer from Alibaba, one model for everything, about 200 milliseconds model-side, but only at 192p for now. Careful: these aren't directly comparable. Resolution, face versus body, and what counts as 'delay' all differ between papers. That's another evaluation lesson for next week.
Two philosophies. The cascade, which most products use: separate models for turn detection, recognition, language, speech and avatar. Easy to upgrade one part, but each hand-off adds delay, and the tone of your voice is lost the moment it becomes text. The end-to-end approach: Alibaba's Wan-Streamer, from June, is one transformer that takes in text, audio and video and puts out audio and video, in 160-millisecond units, with one GPU 'thinking' and passing its memory to another GPU 'performing'. It can respond to how you said something, not just what. It's a proof of concept at low resolution, but I think this is where things are going.
Here is Wan-Streamer in a live call. [Play part.] The person in the small window talks; the character answers with voice and video, from a single model. Notice the interruptions and the timing. The resolution is low; this is version 0.2 of a research system. But compare the rhythm of this conversation with the cascade we costed out a few slides ago.
Let me show you some work from our own lab, because story three runs on data. TalkVid is a talking-head dataset we built with collaborators at CUHK-Shenzhen and Sun Yat-sen University; it's in CVPR 2026 Findings. About twelve hundred and forty hours of talking video, filtered down from over six thousand hours, with seven thousand seven hundred speakers, from children to over sixty, in fifteen languages, mostly at 1080p or higher, and including the upper body, not just the face. And a benchmark, TalkVid-Bench: five hundred clips balanced by age, gender, ethnicity and language, so you can see who a model works for and who it fails.
How do you turn web video into training data? A filtering pipeline. Collect over thirty thousand videos at 1080p or higher. Cut them into shots and drop anything shorter than five seconds or without speech. Score them for visual quality with an aesthetic model and keep only the sharp, well-lit ones. Track motion and drop clips that are jittery, or frozen like a still image. Finally check the head: pose, rotation, is the face complete in frame. We checked the filters against human judges, and they agree about ninety-five percent of the time. The payoff: a model trained on our balanced 160-hour subset gets better video quality scores than the same model trained on older datasets.
Here's the actual funnel from the paper, with real hours at each stage: from the raw pool on the left, through segmentation, the speech filter, the quality filter, the motion filter and the head filter, down to 1,244 hours. Most data is thrown away, and that's normal. Good datasets are mostly decisions about what not to keep.
And what the quality filter actually does, with example frames from the paper. Left: dropped, dim and soft footage. Right: kept, sharp, well lit, stable. This matters for artists too: a model learns the look of its data, so a filter like this quietly decides the look of every avatar trained on it.
Practically, for your projects: what runs where? On a laptop or a mid-range GPU, MuseTalk does real-time lip-sync by repainting the mouth in a single step, and LatentSync works offline with about eight gigabytes. On one RTX 4090 or 5090 you can run light real-time talking heads, and the big fourteen-billion-parameter avatar models offline at 480p, but not live. Live full-body avatars with sound still need several data-centre GPUs. Getting those to run live on a single consumer card is an open research problem; it's one our lab is working on, and it needs all three tricks at once.
Where are live characters already being used? In Japan, the Korean company ESTsoft signed an agreement with NTT and the taxi company Nihon Kotsu in February to put its real-time avatar in taxis, interpreting for tourists. ElevenLabs, the voice company, added avatars in June. Character.AI, whose research we saw tonight, is building video calls with fictional characters. And in performance, ABBA Voyage showed digital performers can fill an arena; a responsive version is the obvious next step.
For the artists: two works by Holly Herndon and Mat Dryhurst that turn this technology inward. At the 2024 Whitney Biennial, xhairymutantx: they trained a model on costumed images of Herndon, so whatever you prompt, you get a distorted version of her, a likeness she controls. And The Call at the Serpentine in London: they recorded fifteen UK choirs, built singing models with Ircam, and created a data trust so the singers share control over how their voices are used. For your own projects, this is a good pattern: use your own voice and face, or people who said yes, and make the consent visible in the work.
For the room. Would you talk to a live avatar of your teacher, say, for office hours at 3 a.m.? And harder: would you talk to a live avatar of someone who has died, built from their videos? What would make it feel respectful, and what would make it feel wrong? Two minutes with your neighbour.
Story four. We've seen that a model can now give a performance, in real time, with a voice. So who is the performer? The person whose face it is? The actors whose work trained it? The person who prompted it? Or the character itself? This year, that stopped being a seminar question. It became a contract, several laws, and a feature film.
Meet Tilly Norwood. She was created by the London AI studio Particle6, founded by the actor and comedian Eline Van der Velden. In July 2025 she starred in a comedy sketch called 'AI Commissioner'. In September, at the Zurich Film Festival, her creator said talent agencies were interested in representing her, and the reaction was furious: actors including Emily Blunt and Melissa Barrera objected, and big agencies said they would not sign her. In July this year Particle6 announced a feature film, Misaligned, with Norwood in the lead. And last month she did a press tour of seventy-five interviews, including one on Piers Morgan's show where she switched into Cantonese for fifteen seconds. That clip has been viewed over twelve million times.
Let's watch a little. [Play about thirty seconds of the AI Commissioner sketch.] That's how Tilly Norwood was introduced: in a comedy sketch, as a joke about the industry. In March she released a music video answering the backlash, with eighteen human crew credited and the vocals generated by Suno. Compare that with ABBA Voyage, where the performers themselves agreed and a live band plays, and with Holly Herndon's Holly+, her own voice model that she lets other people use, on her terms. Same technology, four different answers to 'who agreed?'
Here are the two positions, as fairly as I can put them. The actors' union, SAG-AFTRA: Norwood is not an actor but a character generated by a program, trained on the work of countless performers without permission or compensation, with no life experience to draw on; and producers who want to use a synthetic performer must notify and bargain with the union. Her creator's position: she is a creative work, a piece of art, like an animated character, made with a new tool alongside animation, puppetry and CGI, not a replacement for anyone. Keep both in mind; you'll argue one of them in fifteen minutes.
In June this year, SAG-AFTRA members ratified a new four-year contract with the studios, with over ninety-one percent voting yes. On synthetic performers: producers say they won't use one in a human role unless it adds 'significant additional value', and they have to go to the union first. The chief negotiator was honest that this is a strong disincentive, not a ban. On digital replicas, copies of real actors: informed consent and fair pay, including for actors who have died; a real business reason before anyone is scanned; no replica can replace a performer during a strike; and extra protection for minors. Video-game voice and motion actors won similar consent rules after an eleven-month strike that ended in July 2025.
The law is catching up, unevenly. California, since January 2025: a contract clause that lets a studio make your digital replica is void if it's vague and you had no lawyer or union, and replicas of dead performers need the estate's consent. In the US federally, the NO FAKES Act would create a right to your own voice and likeness; it passed committee in June but was blocked in the Senate three weeks ago, so it is not law. In mainland China, since September 2025, AI-generated content must carry both a visible label and a hidden metadata label, which applies to anything you publish there. In the EU, since August this year, deepfakes must be disclosed. And Denmark has a bill in parliament this month that would protect your face and voice almost like copyright. I'm not a lawyer, and this isn't legal advice, but the practical rule for your work is simple: label it, and get consent.
Three cases about voice, each with a different lesson. Lehrman and Sage versus Lovo: two voice actors were paid a few hundred dollars on Fiverr for what they were told were research recordings, which became clone voices sold to customers. Last year a New York court let their claims under publicity rights and contract go forward. Lesson: what the recording is for matters. 'Sky': Scarlett Johansson declined to voice ChatGPT, then a voice that many thought sounded like her appeared; OpenAI paused it and said it was another actress's natural voice. Lesson: resemblance alone can be a problem. And Darth Vader in Fortnite: James Earl Jones's estate consented, but the union still complained that an AI voice took paid work away from actors. Lesson: consent of one person doesn't settle the effect on everyone else.
What do audiences think? A survey by the research firm NRG with TheWrap in March: fifty-six percent say AI actors will never be as good as human actors; only seven percent say they already are; and about three in five say releasing a film with an AI-written script is unacceptable. But here's the twist: the largest group said AI use wouldn't change their interest in seeing a film. Audiences judge the work, and also the story of how it was made. That's useful to know when you write your artist statements.
Two flashpoints from this year with the same kind of tools and opposite receptions. In February, a viral clip made with Seedance 2.0 showed two famous Hollywood actors fighting. Neither had agreed. The Motion Picture Association and SAG-AFTRA condemned it, and ByteDance pledged fixes. In June, at Tribeca, the artist Ash Koosha premiered Dreams of Violets, made with Kling, Claude and Gemini, about this year's protests in Tehran, billed as the first fully AI-generated feature at a major festival. The difference wasn't the model. It was whose likeness, whose story, and who agreed.
今年两个工具相似但反响相反的事件。2月，一段用Seedance 2.0制作的爆红片段展示两位好莱坞明星打斗，两人都未同意；美国电影协会和SAG-AFTRA予以谴责，字节承诺整改。6月翠贝卡电影节上，艺术家阿什·库沙首映了用可灵、Claude和Gemini制作、关于今年德黑兰抗议的《Dreams of Violets》，被称为首部在主要电影节展映的全AI长片。区别不在模型，而在于用了谁的形象、讲谁的故事、谁同意了。
So here's a checklist for Assignments 2 and 3, both due November 30. Consent: only clone or animate a real person's face or voice if they agreed, in writing; using yourself is the easiest. Label: say clearly which faces, voices and sounds are generated; in mainland China and the EU that's now required. Credit: name the models, the voice references, and every human who performed, recorded or wrote. And keep your sources, last week's rule: prompts, seeds, reference audio, model versions and raw takes.
Debate time. The motion: 'An AI performer that never existed should be allowed to win an acting award.' Left side of the room argues for, right side against. Two minutes to prepare in groups, then two speakers per side, one minute each, and the whole room votes. Use tonight's material: synchresis, OmniHuman's acting plan, the union's position, the creator's position, and the audience survey.
If you're stuck, here are starting points, but try to find better ones. For: animated performances already move us, so awards could judge the result, not the method; or the award could go to the humans who made the choices; or there could be a separate category. Against: acting awards honour a person's lived skill and the risk of being seen; the synthetic performance is averaged from thousands of uncredited actors; and prizes would reward replacing the very people the model learned from.
Four stories, one line each. Sound: picture and sound now share one model and one clock, and the human job is choosing what to hear. Performance: direct with references and playable actions, and check the p, b and m sounds. Real time: chunks, fewer steps and fewer bits turn a renderer into a conversation partner that listens. And the performer: consent, credit and labels are now contracts and law, not just good manners.
How does tonight connect to your assignments? For Assignment 2, when you reverse-engineer an AI video, reverse-engineer the sound as well: which layers were generated, and by which model? Then remix it, with a native-audio model or your own voice track. For Assignment 3, your game: try giving one character a voice and a face that reacts. A talking portrait with MuseTalk or LatentSync, or a short generated cut-scene with dialogue, is enough. Both are due November 30. And use your own voice, or a friend's who agreed, and label what's generated.
Three readings for this week, one for each kind of student, and you reflect on any one for the quiz. Technique: the LTX-2 technical report. Read the abstract, look at the architecture figure we discussed, and listen to two of the demos. Art and tech: The Call, by Holly Herndon and Mat Dryhurst at the Serpentine: choirs, singing models, and a data trust. Culture: Variety's report on Tilly Norwood's first feature film, read together with the union's statement. And two optional technical papers: Avatar Forcing, and our TalkVid dataset paper.
Quiz 08 is online now: two technique questions, on joint audio-video generation and on the real-time tricks; two art questions, on synchresis and on directing a line; and one reading reflection. One to three sentences each. As always, it's due at the end of the semester, and you can edit your answers until then.
This slide collects every demo from tonight so you can find them again. They're also on the course website, on the Lecture 8 page under Demos. All the videos belong to their creators and were shown for teaching.
Next week, Week 9: critique, evaluation and responsible release. We'll generalise tonight's trap, a model 'beating' real video on a metric, and ask what our scores really measure. Please bring one generated clip with sound for a studio critique. And we'll cover model cards, watermarks and labels.
To sum up tonight in one line: generated video learned to speak, to act, and to listen. Our job is to choose what we hear, to direct how it's said, and to ask whose voice it is. Thank you, and see you next week.
The last few slides are an appendix for after class: practical tools, a workflow, prompts, and checks you can use for Assignments 2 and 3. I'll go through them quickly; they're all on the website.
Here's a toolkit table: for each job, an open model, its size or hardware as reported by the authors, and the licence. Picture and sound together: LTX-2, or the lighter Kandinsky 6 Lite and NAVA. Two-person dialogue: Ovi or MultiTalk. Adding Foley to a silent clip: MMAudio or Tencent's HunyuanVideo-Foley; note MMAudio's weights are non-commercial. Lip-syncing existing footage: LatentSync or MuseTalk, which run on modest GPUs. Re-dubbing: InfiniteTalk. Talking or singing bodies: Wan2.2-S2V or HunyuanVideo-Avatar. Always check the licence before commercial use.
A workflow you could do in one evening. One: record the line yourself on your phone in a quiet room. Your timing is the performance. Two: generate or paint one front-facing image of the character with the mouth clearly visible. Three: animate it with an audio-to-video model, or lip-sync an existing clip. Four: add ambience and Foley under the voice. Five: run the checklist on the next slide and re-generate only the shot that fails. Using your own voice avoids every consent problem from story four.
And the review checklist. Lips: full closure on p, b and m. Timing: scrub frame by frame at a hard consonant or an impact and check the sound lands on that frame. Voice: the same voice across shots, breaths in human places. Eyes and body: motivated gaze, hands in rhythm. Room: reverb that matches the space, ambience continuous across cuts. And credits and labels written down before you submit.
Finally, how to measure sync on your own work. The reference SyncNet code is on GitHub; three commands give you the offset in frames, Sync-D and Sync-C. And to train your own eyes, use ffmpeg to make a copy with the sound delayed by 125 milliseconds, the edge of what viewers tolerate, and compare. That's exactly how I made tonight's lag game.
