Full English script with concise Chinese companion explanations.
The English text is the full script. Chinese explanations are condensed, not word-for-word translations. Each entry links to the earlier slides it recaps.
Good evening, and welcome to Week 11. This is our last lecture before the final presentations, so tonight I won't add anything new. We'll look back instead, and in more detail than usual. For ten weeks we've moved fast: neurons, diffusion, editing, customization, quantization, minute-long video, sound, avatars, worlds, robots, and critique. Tonight we'll walk back through every lecture in order, with the technical content first: the architectures, the equations and the numbers. For each one I'll put the original slides back on screen, the exact slides you saw that night, and we'll ask what you should still remember and how it connects to your projects. On almost every slide you'll see the lecture and slide number, like "L03 · 54". On the website each of those is a link, so this deck works as an index to the whole semester. After the review and a break, we have the last weekly presentations.
Here are ten Tuesdays on one line, one sentence each. Week 1: a generated video is a learned prediction about change. Week 2: write the edit contract, meaning what changes, what stays and what adapts. Week 3: the model predicts and the sampler takes a step; seed, guidance and steps are different knobs. Week 4: video has two kinds of time, and the artist's intention comes first. Week 5: whose model is it? Week 6, Yaofu's talk: store less, using fewer bits per number. Week 7: the unit of AI video became the scene, and tools can vanish. Week 8: sound and picture share one clock, and we asked who the performer is. Week 9: add an action and video becomes a place. Week 10: is it AI, is it good, is it fair, is it allowed? If you can say these ten sentences and give an example for each, you're in good shape.
Lecture 1, September 1st. The title slide said: how machines learn to make moving worlds. It was the longest lecture of the term, a hundred and two slides, and it tried to give you a mental model that would survive the next product release. We went from a single neuron to CNNs, attention and transformers, latent models, diffusion and flow, video as a temporal system, the 2026 landscape with Seedance and MiniMax H3, and finally a full AI filmmaking workflow. The question that night was simple: what is a generated video, really? The answer: a learned prediction about change. Let's look at four ideas from that night that you've used every week since.
This is the map from Lecture 1, and it held all semester. Data supplies examples. Representation decides what becomes compact and comparable: latents and tokens. The model learns relationships in that representation. Sampling turns what was learned into one particular output. The useful part is the debugging question. When a face drifts, ask which stage it is: missing data, a weak identity representation, too little temporal modelling, or an inference setting? And remember Lecture 4: data is never neutral. Filtering decides what reaches training, so a model's defaults reflect choices other people made.
Here's Lecture 1's first half compressed into one line. A neuron computes a weighted sum, adds a bias and applies a nonlinearity, and training adjusts those weights to reduce error across examples. For diffusion, the target is the noise that was added. A CNN scans one learned filter across the image, because nearby pixels usually belong together. Attention lets every token choose what matters, so the red scarf in frame twenty can attend to the scarf in frame two. Attention alone doesn't know order, which is why position encodings like RoPE add time, height and width. Then a VAE compresses first and the generator learns to create in that smaller space. Diffusion and flow learn the path from noise to data. Every system you met afterwards, including Helios, LTX-2 and Genie, is a rearrangement of these five parts.
Why do all these models work in a compressed space? Count. A ten-second clip at twenty-four frames per second is two hundred and forty frames before any sound. Attention compares every token with every other token, so cost grows with the square of length. Lecture 7's rule of thumb: six times the length means roughly thirty-six times the comparisons. So a video VAE squeezes space and time first. MiniMax H3 reported sixteen times spatial and four times temporal compression, and then groups the latent into patch tokens, small blocks of space-time. Bigger patches are cheaper but lose fine motion and text. That trade-off decided which details survived in your Assignment 1 films.
Here's the technical core of Lecture 1's video section. After the VAE, patchification groups latent values into tokens, and each token is a small block of space-time that carries content plus coordinates. Larger patches cut cost but lose fine motion and text, and smaller patches keep detail but lengthen the sequence. Attention compares a set of vectors, so on its own it doesn't know left from right or before from after. Position encodings inject coordinates, and RoPE rotates query and key components so relative position changes similarity. In video that needs three axes, time, height and width, which is why H3 and HunyuanVideo describe spatiotemporal RoPE. Last, which tokens may talk to which: full spatiotemporal attention is the most flexible and grows quadratically, while sparse, sliding, factorized or block attention limits who attends to whom. HunyuanVideo 1.5 uses selective and sliding-tile attention. The question to ask of any architecture: what is a token, how does it know where it is, and who can it see?
This is the second half of Lecture 1, and it was the workflow behind Assignment 1. Current models are shot engines, reference engines and editing engines. A film still needs dramatic structure, coverage, continuity, selection, sound and rhythm. So we used five gates: the brief, the world bible, the animatic, shot generation, and post-production. Each gate produces something you can inspect before you move on. Two rules from that night are worth repeating. Define change before style: "a courier must deliver one memory before the city deletes her identity" gives you action and consequence, while "cinematic cyberpunk woman" mostly gives you decoration. And use the least generative task that solves the shot: text-to-video when you want invention, image-to-video to keep a designed frame, first-last-frame to control endpoints, and video-to-video to keep blocking and timing.
And this is the habit Lecture 1 ended on, which I've repeated almost every week since. Start from a known-good workflow and run it once before you add anything. Then hold the shot constant and vary one decision: camera strength, endpoint frame, guidance or motion reference. Generate three to six candidates and write your observations before you choose a favourite. Changing model, seed, prompt, duration and reference all at once creates variety, but almost no knowledge. Then keep the usable interval, not the prettiest thumbnail: watch at full speed, mark in and out points, and keep one representative failure. Finally, edit first and repair only what survives the cut. The last slide of that night said that technical literacy becomes creative agency. This slide is what that means in practice. The check-yourself question is the exercise from that night: design one eight-second shot before you open a generator.
Lecture 2, September 8th: generative editing. We spent the whole evening with one object, an amber ceramic pear with a blue stem, and asked it to survive three things: a change of environment, an art direction for a fictional exhibition called AFTER RAIN, and a moving camera. The question was: when you ask for a change, what exactly should stay the same? It sounds like a technical question, but it turned out to be an artistic one, because "the same" has more than one meaning. This was also the night we first met flow matching, classifier-free guidance, ControlNet and LoRA, and the first metric that rewarded the wrong thing.
Remember the pear. The gallery becomes a forest. The edit contract had three columns. Change: the environment. Keep: the pear's identity, its pedestal, the framing. Adapt: the light and the reflections, because a bright gallery window reflected in a dark forest would be faithful and wrong. Then Lecture 2's subtle point: "keep it the same" has two meanings. Semantic sameness means a visitor recognizes the same sculpture. Exact sameness means identical pixels in a region, and only copying or compositing can promise that. Write this contract before you generate, and check each column separately afterwards. It works for a still image, a video shot, and a game asset in Assignment 3.
Here's Lecture 2's rule as a table. Text says what should change. A mask says where change is allowed, though it won't fix seams or reflections on its own. A spatial condition, meaning edges, depth or pose through ControlNet, says what shape or structure to follow. ControlNet keeps the pretrained path frozen and adds a trainable copy joined by zero-initialized convolutions. A reference image, through IP-Adapter, carries appearance that words can't describe precisely, instantly but less faithfully. And when a concept has to recur across many shots, you adapt the weights with LoRA or DreamBooth. The mistake to avoid is adding adjectives when the missing information is spatial.
Lecture 2 opened a current image editor from the input side. The source image enters twice: semantic features through a vision-language component, which help with what the pear is and what "it" refers to, and visual latents through the VAE, which keep the particular shape and texture recognizable. The target stream starts as noise, and a transformer processes it with those conditions. The key point: during training we have the desired edited target, so we can measure the error. At inference we have the source and the request, but the target doesn't exist yet. Showing an editor a reference conditions one generation. It doesn't teach new weights; that's adaptation, like LoRA. Where does instruction-following come from? Training triples of source, instruction and target, starting with InstructPix2Pix's synthetic data. And if a pair says "make it ceramic" but the target also moves the camera, the model learns that unwanted association. The preservation contract starts in the training examples.
Two small calculations from Lecture 2 that catch people out. First, image latents. A 1024 by 1024 RGB image is just over three million values. With sixteen-times spatial reduction and sixty-four latent channels, it becomes 64 by 64 by 64, which is 262,144 values. Sixteen times smaller along each axis doesn't mean sixteen times fewer values, because the channel count changes too. Here it's about twelve times fewer. Second, video latents. Forty-five frames with a special first frame and four-times temporal compression: forty-five is four k plus one, so k is eleven and there are twelve latent frames, which fit a three-by-four virtual grid. Simply dividing forty-five by four misses the first-frame convention. And one more piece from Lecture 2's appendix: InstructPix2Pix has two guidance scales, one for the image and one for the text. Set both to one and the middle terms cancel, leaving the fully conditioned prediction.
The middle of Lecture 2 was pure art direction. The brief for AFTER RAIN said "fragile", and we asked how to make fragility visible. Scale, restrained light and a reflection less stable than the object are hypotheses you can test, not a formula. Then Hokusai's Great Wave. Look for Mount Fuji: it's small and distant, and the wave's rhythm creates the tension. We didn't borrow "Hokusai style". We borrowed a relationship: a small, vulnerable form facing a much larger environment. Then reference roles: one image for the subject, one for the composition, one for the material, and you check for unwanted transfers by removing one reference and seeing what disappears. And negative space has a job: it makes the object feel small and gives the title somewhere to live. If you reference an artist in your final presentation, say which relationship you borrowed.
Two more habits from Lecture 2 that I expect to see in your final presentations. First: a lucky image or a reliable direction? You compare two prompts once and the second gives a wonderful poster. Was it the wording, or a favourable random sample? Several outputs per condition tell you. And there are two different questions: would I exhibit this image, and could I build a series in this direction? Second: a critique with observable evidence. Name a visible feature, explain its effect, and propose a revision. For example: "the crisp reflection makes the collage feel photographic, so I'd simplify its shape." Compare that with "I don't like the reflection." The first gives the artist something to do. And a process record, with the intention, the invariants and one rejected result, makes that critique precise instead of a guess.
Then the edit has to move. In Lecture 2's third part, a video edit has to keep its promises in the next frame. Three lessons. Consistency isn't stillness: the stem may hide during a turn, but it shouldn't change colour without a reason. Four good frames can hide a bad video, so a contact sheet is publicity stills, not the performance, and you review full playback around turns and occlusions. And the frozen video wins the wrong test: a naive frame-difference score prefers a video that never moves. That was our first meeting with a metric that rewards the wrong thing. Keep it in mind, because Thread 4 is built on it.
Lecture 3, September 15th: inside the generator. The subtitle asked "why does it produce this?", and we answered with arithmetic you could do on paper. A traveller in a yellow coat on a long platform was our running example, because the colour, the identity and the perspective were easy to check separately. We covered noise and denoising, VAEs, CLIP and attention, transformers, flow matching, seeds, guidance and steps, LoRA and ControlNet, a look at 2026 image and video models, and the first glimpse of Astra, world models and robot policies, which came back in Lecture 9. If you remember one sentence from that night, make it this one: the model predicts, the sampler takes a step.
Lecture 3 explained why a picture can match the topic and still be wrong. CLIP learns from pictures paired with captions, so it learns associations. But captions mention the person and leave out the railing, so precise relations are weakly supervised. That's why you get a station, a traveller and a suitcase, but the suitcase is on the wrong side, or you ask for two objects and get three. So your checks should include the relations that carry the meaning, not just a list of nouns. Then continuity: a seed is not a character description. The character sheet defines what must stay: the silhouette, the surface details like buttons and pockets, and ownership, meaning which hand holds the prop. A visible side of a suitcase should change when the camera moves. Its handle shouldn't change construction for no reason.
The second engine part is attention. Each token makes a query, a key and a value. Queries ask, keys match, values contribute, and the softmax weights sum to one. In Lecture 1 the example was a red scarf: a token in frame twenty can attend to the tokens that carried the scarf earlier. Cross-attention is how words reach image regions, which is why Lecture 5's Custom Diffusion only fine-tuned the key and value projections of cross-attention and still learned a new concept. In Lecture 4 we split it into spatial attention within a frame and temporal attention across frames. One warning from that lecture still matters: attention connects features, but it doesn't guarantee understanding.
Now the core idea from Lecture 3, with the numbers we did by hand. Training adds noise to a clean image: a weighted clean value plus weighted noise. With a clean value of point eight, noise of minus point two, and weights point six and point eight, we got point three two. The network learns to predict the noise, or, in flow matching, the direction of movement from noise to data: the target minus the starting noise. At generation time there is no target. We follow the learned directions. And keep the division of labour from Lecture 3: the model predicts, the sampler takes the step. Change the sampler and the same network can give a different picture.
These are the three knobs you touched in every assignment. The seed is an address into a repeatable random process, not an instruction. Seed forty-two doesn't mean wide shot. To repeat a result, save the whole workflow, not just the seed. Guidance pushes past the prompted prediction: unconditioned plus scale times the difference. With predictions of two and three, a scale of four gives six, outside the range. That's why strong guidance burns colours and makes samples look alike. Steps set how accurately you follow the learned field, not whether the field is right. Fifty steps won't fix a wrong hand. And a distilled model trained for one to four steps is a different system, not a slow model stopped early.
The end of Lecture 3 turned the knobs into an experiment. To ask whether guidance changes subject clarity and variety, keep the model, prompt, size, sampler and steps fixed, and try different guidance values on the same set of seeds. Write the comparison as a table before you run it: rows are seeds, columns are guidance values. Decide your judging questions beforehand, because if you decide what counts as success after seeing the images, you'll favour the one you happened to like. Record failures too, because leaving them out changes what the grid represents. And label your figures: a progress picture from partway through one run answers a different question from finished runs at fifteen and fifty steps. This is exactly what the reconstruction log in Assignment 2 asks for: at least three attempts, one change each.
Pen out. Four symptoms from real student logs this term. One: same prompt, but every run gives a completely different composition. Two: the subject is unmistakable, but colours are burnt and every sample looks the same. Three: the result is fine, but it takes forty seconds and you need it live in a gallery. Four: a character's face changes after she turns away and turns back. For each one, write down which knob you'd touch, or whether it isn't a knob at all. Thirty seconds. Then compare with your neighbour.
Here are the answers. A is the seed: fix it for comparisons and record the whole workflow, and use references when you need a particular layout. B is guidance set too high: lower the scale and compare across the same list of seeds, changing one variable at a time, as Lecture 1 asked. C is cost: a distilled model, fewer steps, quantization from Yaofu's talk, or a smaller resolution, and you measure seconds, not step counts. D is the trick question. It isn't a knob. It's memory over time: the representation and the context window. We'll meet it again in Lectures 7 and 9, and after Lecture 9 you'll see it's one problem wearing three costumes.
Lecture 4, September 22nd: generative AI for art, images and video. It was built around five readings and a running example, an empty studio with a painting on the wall, and it moved through five areas: video generation models, visual art, filmmaking, visual computing, and the creative industries. The question was: what changes when an image moves? A still suggests possibilities, but a video commits to a sequence of events. If a hand rises, it has to belong to the same person as it moves. This was also the lecture where the art students' questions and the technical students' questions started to be the same questions.
Three ideas from Lecture 4's first part that you need to read any video paper. First, two kinds of time. Frame time is the time inside the scene: a person enters, crosses the studio and looks at the painting. Denoising time is the sequence of computational steps that produces the sample. A denoising step isn't an extra frame. Whenever you see a time index, ask which one it is. Second, spatial attention relates things within a frame, such as the hand and the object it holds, and temporal attention relates the hand now to the hand a moment later. Third, object motion and camera motion are different. A person can cross a still frame, or the camera can circle a still person, and both change pixel positions. If you want to control them separately, you have to specify them separately.
Then data. Lecture 4 showed a pipeline that selects clips, filters them and attaches captions, and every stage changes what reaches training. Filtering trades coverage for usable examples: removing difficult material may improve average quality while leaving gaps in certain actions or styles. If your project depends on an unusual visual language and the model keeps resisting it, rewriting the prompt may not help, because the gap may be in the data. And here's what filmmakers said they wanted in the survey from that night: character identity, body movement, camera control and local editing. All four showed a gap between how important artists rated them and how well tools performed. That chart predicted much of the semester: Lecture 5 on identity, Lecture 7 on movement and dance, Lecture 8 on performance.
The visual-art part of Lecture 4 started from the artist, not the tool. An intention becomes useful when you connect it to visible decisions. Suppose the intention is an uneasy memory: delayed movement, missing details, or a familiar room with one altered object. Each suggests different controls, while "make it more artistic" suggests nothing. A scenario gives the work a purpose: the same image could be a print, a storyboard frame, or one frame of an installation. Content and style need separate decisions: if a character must stay recognizable while the image becomes painted, decide which features stay, such as silhouette, proportions and pose, and which may vary, such as brushwork and palette. The exercise was one intention and one visible intervention, plus a way to check whether the audience reads it.
The filmmaking part of Lecture 4 looked at real art films from a 2025 survey. The headline: the films used on average 3.14 video-generation tools each, because different stages have different needs. Overthinking built a nostalgic visual language through image adaptation, asset creation, animation and editing. Clown turned inconsistency into expression: frame-by-frame style change supporting a fragmented identity. That's a deliberate choice, and an accidental change doesn't automatically mean anything. For Pixi kept a recurring character through iterative generation and region editing. Fish Tank went through 3D, with manual UV mapping and topology, ordinary production work that doesn't disappear. And volumography captured space, so the camera became a decision made after capture, which led straight to splats in Lecture 9. The common lesson: a short film still needs decisions about what changes, when, and what the audience should feel.
Lecture 4 ended with a warning that became Lecture 10. A technical score answers a particular question: prompt alignment, visual similarity or some aspect of perceptual quality. It can't establish whether an artwork succeeds. A deliberately ambiguous image may score badly on literal prompt matching and still produce exactly the intended experience. Quality has several meanings: an image can be sharp and fail to convey the action, and a sequence can match the prompt and feel emotionally wrong. So make separate judgments: technical continuity, clarity of the reveal, and the audience's interpretation. And the last slide: a process others can understand. Save references and meaningful before-and-after versions, record models and settings, and explain the decisions that changed the work, including editing and sound. That's the same advice as Lecture 7's project folder and Lecture 10's credit block, a month earlier.
Lecture 5, September 29th: making the model yours, with two tracks running together. The technical track covered Textual Inversion, DreamBooth, Custom Diffusion, concept ablation, LoRA, adapters and speed. The art track covered copyright, memorization, living artists, provenance and bias. Many of the slides were adapted from Jun-Yan Zhu's course at CMU, including the moongate and his dog Stark. The question opened and closed the evening: whose model is it? Keep your answer from that night in mind, because I'll ask you about it at the end of this section.
Why customize at all? Lecture 5 named two bottlenecks. The first is linguistic: you can't describe the exact line quality of your own drawing, or a friend's face, in words. The second is data: personal things, like your dog or your family's shop, aren't in the training set. The moongate was the example. Ask Stable Diffusion for a moongate, the circular gateway you can find in Hong Kong gardens, and you get something plausible but wrong. The model doesn't know the concept. Then Stark, Jun-Yan's dog. Ask for a dark grey weimaraner and you get a perfectly good weimaraner, but never Stark. A description isn't an identity, and that's exactly the character drift many of you fought in Assignment 1. After customization, "V-star dog wearing sunglasses" gives you Stark, in sunglasses, and a moongate in snow is something no training photo showed.
Lecture 5 in one table: the key question is what gets trained. Textual Inversion keeps the whole model frozen and learns one new word, an embedding. It's good for styles and weak on identity. DreamBooth fine-tunes all the weights and preserves identity best, but it overfits, so you regularize with images of the general class. Custom Diffusion trains only the key and value projections in cross-attention, so the update is small and concepts combine. LoRA adds a thin overlay of two skinny matrices: for a 4096 by 4096 layer at rank sixteen, that's about 131 thousand numbers instead of 16.8 million. IP-Adapter trains nothing per subject; an encoder turns your reference into a visual prompt. The rule of thumb from that night: a style can live in a word, but a character who must stay the same across shots needs a change to the weights.
The same mechanism has three faces. Overfit: after learning "moongate", the fine-tuned model puts moongates into every image of the moon. The fix is regularization images, real photos of related concepts, because what you leave out of a dataset teaches as much as what you put in. Memorize: at internet scale, Carlini and colleagues generated from the captions of heavily duplicated training images, and some outputs matched the originals almost exactly. A living artist's style can be memorized too, which is why "in the style of Greg Rutkowski" became a famous prompt. Forget: concept ablation, also from Kumari's group, makes the model map "Grumpy Cat" to an ordinary cat and remove the Rutkowski look while keeping the boat and the water. That's what an opt-out inside the model could look like. For your own small datasets: remove duplicates and include what the model should keep.
The arithmetic behind the most common customization method, which came up in Lecture 2 and was the centre of Lecture 5. A layer's weight matrix W stays frozen. LoRA learns an update that's the product of two skinny matrices, B times A, with rank r. For a 4096 by 4096 layer, a full update has 16,777,216 entries. At rank sixteen, the two factors together have two times 4096 times 16, which is 131,072, a factor of 128 fewer for that update. That's why a LoRA file is small and you can swap overlays on the same base model. Two cautions from those nights. The base weights still exist, so the whole model didn't shrink by 128. And a small update doesn't prove the concept generalizes: test it on poses and settings the training images didn't show. Custom Diffusion makes a similar bet differently, training only the key and value projections of cross-attention.
Lecture 5 ended with bias, and it belongs in a lecture about customization, because every base model's defaults come from someone else's dataset. PULSE, in 2020, took a pixelated face and searched a generative model for a sharp face that matched. Where do the details come from? Not from the input: they come from the model's prior, and for a well-known pixelated photo of Barack Obama, PULSE produced a white face. Then the truncation trick: a setting that improves quality by pulling samples toward the average also reduces diversity, and the majority group grows as truncation increases. A technical setting turns out to have a social effect. In text-to-image models, "manager" gave mostly white men in suits. Quick fixes, like rewriting prompts behind the scenes, help. For artists, the longer-term answer may partly lie in building your own data and shaping your own models.
Lecture 5 began with a show of hands, and I asked you to remember where you put your hand. A foundation model has seen billions of images by millions of people. You add twenty of yours and fine-tune it. Whose work is the output: the company's, yours, or the people in the training data? That night we saw how memorized images are found, how a living artist's style becomes a prompt word, and how concept ablation can make a model forget Greg Rutkowski while keeping the boat and the water. Hold your answer. We'll come back to it in Thread 4 with consent, and you'll need it for your credit block.
Week 6 was Yaofu's talk on quantization for faster model inference. There were no slides on our site, so here's how it kept coming back in later lectures, which is the best measure of a good lecture. The idea: keep each number in the model with fewer bits, like saving a photo as a smaller JPEG. Lecture 7 did the arithmetic for a fourteen-billion-parameter model like Helios or Wan-Dancer: at sixteen bits, two bytes per parameter, that's twenty-eight gigabytes for the weights alone, fourteen at eight bits, seven at four bits, before activations, the VAE and the text encoder. That's the difference between a datacentre GPU and one you might own. And Lecture 5 put it in a family of three: adapters change less, distillation steps less, quantization stores less. Lecture 8 added that real-time avatars use all three at once.
Lecture 7, October 13th: what's new in AI motion. From this lecture on, the format changed to four stories from this year, each with an "under the hood" section adapted from courses at Stanford and CMU, a question for the room, and a debate at the end. The four stories: the five-second clip is over; music becomes the choreographer; motion leaves the screen; and Sora is gone. The question underneath was what happens when generation gets long, fast and physical. It was also the night Assignment 2's project-folder advice was born.
Lecture 7 in four numbers. In 2024 a typical clip was about five seconds. Early this year Seedance 2.0 reached about fifteen. By summer, Seedance 2.5, Wan 3.0 and Kling 4.0 were at thirty seconds in one pass, long enough for an entrance, an exchange and an exit. And Helios, an open research model, made about a minute at about nineteen and a half frames per second on one H100, close to real time. There were two routes: denoise all frames at once, which is consistent but expensive, or write forward in time, autoregressively, like a language model writes words, which can stream but has to fight drift. The question we asked was what the director's job becomes when one generation is a whole scene.
Here's the under-the-hood part of Story 1. Why was five seconds the wall? At twenty-four frames per second, thirty seconds is six times the frames, and attention compares everything with everything, so roughly thirty-six times the comparisons. Three ideas broke the wall. Autoregression: write the video forward in chunks, the way a language model writes words, so frames can stream. The KV cache: store each layer's keys and values from earlier chunks so the past isn't recomputed. If the cache stays bounded or compressed, as in Helios, each new chunk costs about the same no matter how long the video gets. And distillation: train a student to jump to where the teacher would end up, through consistency models or DMD. Helios's distilled version uses three steps instead of about fifty. Each of these came back in Lecture 8's avatars and Lecture 9's world models.
Story 2. Wan-Dancer, released in July by Alibaba's Tongyi Lab with open weights. Give it a music track, one photo of the dancer and a style word, and it makes more than a minute of dance at 720p that hits the beat, in five styles: Chinese classical, K-pop, street, tap and Latin. The one idea worth remembering is that it works at two levels. The global level plans the whole song's shape: intro, verse, build, chorus and outro. The local level renders the steps beat by beat. That's how a human choreographer works, and it's why the dance stays coherent for a minute. To dance, a model has to hear beat, tempo, phrases and energy. The limits: one photo means the back of the costume is invented, and we can't check whose dances were in the training videos. Who's the choreographer here?
Story 3: a humanoid learned a K-pop dance challenge from phone video. There were four steps. Film a dancer. Lift the 3D joint motion from flat video. Practise in simulation with reinforcement learning, because a robot that copies a human pose exactly will fall over: it's 1.3 metres tall with 23 joints, different proportions and slower motors, so it needs retargeting and practice. Then transfer to the real robot across the sim-to-real gap, where friction, motor delays and flex never match the simulator, and domain randomization helps. Why dance challenges? They're short, repeated by thousands of people, on the beat, and filmed full body, which makes them almost perfect training data. The big idea: video is becoming training data for bodies. A dance you post now has two audiences, people and machines. We asked whether a generated video that teaches a real body to move is still "fake".
The last stop in this thread is about time in a different sense: how long your work survives. Sora became a product in December 2024. OpenAI announced its shutdown in March, the app closed in April, and the API was switched off on September 24th. It lasted about eighteen months as a product. So, from Lecture 7: keep your sources, not just your outputs. One folder per project, with a README that gives the model, version, date and settings, plus folders for prompts, references with credits, raw generations including the interesting failures, and the edit. Pin versions: the exact model ID and, for open weights, the commit hash. This isn't busywork. It's half your A2 report, and it's the evidence for your final presentation.
Lecture 8, October 20th: when AI video learned to talk. It opened with a clip played with the sound off, then on. One model had made the picture, the voices, the effects and the music. The four stories: sound is no longer post-production; directing a performance, not a prompt; characters that listen; and who is the performer? Underneath was the idea from film theory that sound is half the picture. It was also the most legal and ethical lecture until Lecture 10: likeness, voice, the new SAG-AFTRA contract, and synthetic performers.
Lecture 8 started with film theory. Michel Chion's synchresis: when a sound and an image happen at the same instant, your brain welds them into one event. That's why Foley works and why bad lip-sync hurts. Then the architecture. There are four ways to wire sound and picture: picture first, sound first, two towers, or one sequence. LTX-2 uses two towers: a fourteen-billion-parameter video stream and a five-billion audio stream that look at each other in every block using only time positions, so a token at two seconds attends to audio at two seconds. That shared clock is the whole trick. And the fastest human quality check: watch p, b and m, where the lips must close.
Lecture 8 gave you both an eye test and a number for lip-sync. The eye test is phonemes and visemes. Many sounds share one mouth shape, and the bilabials p, b and m need the lips fully closed. If a character says "Bob" and the lips never meet, everyone notices. The number is SyncNet, from Oxford in 2016. It takes a fifth of a second of mouth and audio, embeds both, slides the audio up to fifteen frames each way, and reports Sync-D, the best distance, where lower is better, and Sync-C, the confidence, where higher is better. Then the trap. On LRS2, real video scores about 6.7 Sync-D, and Wav2Lip scores 6.4, better than reality, because it was trained against a SyncNet-style judge. When a metric becomes a training target, it stops measuring what you care about. I promised we'd return to that in Lecture 10, and we did.
Two technical reasons joint audio-video models stay in sync. First, positions. Video and audio tokens arrive at different rates. If you number them naively, video token thirty and audio token thirty are at different moments, so attention looks at the wrong sound. MMAudio's figure showed it: with default positions the bright band drifts, and with positions scaled to real time you get a clean diagonal. LTX-2's temporal-only RoPE and Ovi's scaled RoPE are the same fix. Second, data, which Lecture 8 called the unglamorous truth. Most online video is bad training data for sound: music pasted over vlogs, dubbing, off-screen voice-overs. So teams filter with a sync score and caption the sound, with spoken words in quotation marks. TalkVid, from our lab, went from over 6,000 hours through speech, quality, motion and head filters to 1,244 hours, and the filters agreed with human judges about 95 percent of the time. Good datasets are mostly decisions about what not to keep.
Story 2 was about direction. A performance has six parts: voice, timing, mouth, face and eyes, body, and listening, which actors say is half the job. The technical lineage moved from repainting a mouth with Wav2Lip, to a whole head from one photo with VASA-1, to full body and singing with OmniHuman. VASA-1's lesson: generate motion, not pixels, then render. OmniHuman-1.5 added intention: a slow-thinking planner of language-model agents writes emotion, action and shot for each moment, and a fast diffusion transformer acts it out. And then the practical table: an emotion label like "angry" gives you the average angry face from the training data. Directors use playable actions instead. "Angry but held in" becomes "through clenched teeth, without raising his voice." And the strongest option for artists: record the line yourself.
Story 3 made Lectures 6 and 7 pay off. Real time means two numbers: throughput, meaning forty milliseconds per frame at twenty-five frames per second, and latency, how long after you stop talking the character answers. People leave a gap of about two hundred milliseconds between turns, even though planning a reply takes over six hundred, because we predict the end of the other person's sentence. A normal video diffusion model can't do this: thirty to fifty steps, doubled by guidance, and bidirectional attention, so the first frame isn't finished until the last. The fixes you already knew: chunks with a KV cache, distillation, as in TalkingMachines going from twenty-four evaluations to two, and quantization. Avatar Forcing at CVPR this year added listening: the avatar takes your audio and head motion as input, so it nods and laughs while you speak.
And here's the toolkit that kept coming back. Change less: adapters like LoRA. Step less: distillation, a student that jumps where the teacher would land. Helios went from about fifty steps to three, and TalkingMachines from twenty-four model evaluations to two. Store less: quantization, Yaofu's Week 6 talk, eight or four bits per number. Remember smartly: generate in chunks, look only backwards, and keep a KV cache so the past isn't recomputed. Why it matters: at twenty-five frames per second you have forty milliseconds per frame, and in conversation people expect a reply in about two hundred milliseconds. Lecture 9 added the payoff: Gaussian splats went from about 0.07 to over a hundred frames per second. Speed changes what something is.
Story 4. Tilly Norwood, an "actress" who never existed, created by Particle6. The backlash in 2025 was immediate, and this year she was announced as the lead in a feature, Misaligned. Then what the actors won: the 2026 SAG-AFTRA contract, ratified by over ninety-one percent. Producers say they won't use a synthetic performer in a human role without significant additional value, and must go to the union first. The negotiator called it a strong disincentive, not a ban. For digital replicas: informed consent and fair pay, including for actors who have died. And voice: in Lehrman and Sage versus Lovo, voice actors paid a few hundred dollars for "research" recordings found clones of their voices for sale, and a New York court let their claims go forward. Two flashpoints this year had the same tools and opposite receptions: a viral fake fight between two stars without consent, and Dreams of Violets at Tribeca, made with disclosure.
故事四。Tilly Norwood,一个从未存在的“女演员”,由Particle6创造。2025年反对声立刻出现,今年她被宣布担任长片《Misaligned》的主角。演员们争取到的:2026年SAG-AFTRA合同,超过91%投票通过。制片方承诺除非有显著额外价值,不在人类角色中使用合成表演者,并须先与工会协商,谈判代表称这是强力抑制而非禁令。数字复制品:知情同意和公平报酬,包括已故演员。声音:Lehrman和Sage诉Lovo案,配音演员拿几百美元录“研究用”录音,后来发现声音克隆在出售,纽约法院允许诉讼继续。今年两个热点用同样的工具却得到相反的反应:未经同意的两位明星打斗病毒视频,和在翠贝卡首映、做了披露的《Dreams of Violets》。
Lecture 9, October 27th. It opened with a question: I push this cup off the edge of the table, what happens next? A video model can draw the answer. A world model has to answer what happens if you push it. The four stories: from video to playable worlds; 3D is back as Gaussian splats; robots that learn by watching people; and does a world model need pixels at all? We noticed you had already seen two of the slides: Lecture 3 introduced world models, the round-trip test and VLAs, and Lecture 7 gave us the KV cache and the dancing robot.
CMU's definition: a world model takes the previous state and an action and predicts the next state. A video model answers "what happens next?" A world model answers "what happens next if I do this?" Under the hood the network can be almost the same autoregressive video model as Helios, but the action arrives every step, so it can't generate the clip in advance. It has to wait for you. Where do actions come from if the internet has no key-press labels? Genie's latent action model looks at two frames and squeezes the change into one of only eight codes, which turn out to mean left, right and jump. At play time it's a loop: tokenize the frame, turn your key press into a latent action, predict the next tokens, decode, twenty-four times a second. Genie 3 runs at 720p in real time, with promptable world events like "a storm starts".
The round-trip test started in Lecture 3, with Astra and a doorway. Lecture 9 turned it into the main test of any world model. See a red door, turn away for ten seconds, turn back. A good world model shows the same red door. A weak one shows a blue door, or no door, because it only remembers what's in its context window. Genie 3 holds visual memory for about a minute. Lecture 3 added an important caution: the test checks spatial and temporal agreement under a short sequence of actions, not all of physics. Contact, materials and control need other tests. And the same capability is now serious work: Waymo built a variant of Genie 3 that also generates lidar, so engineers can rehearse the rare, dangerous situations you can't film safely.
This is the slide I most want you to remember tonight. In Lecture 7, long video drifts: the model trains on perfect previous frames but generates from its own imperfect ones. That's the train-test gap, or exposure bias. Self Forcing trains on its own rollouts, and Helios shows the model drifted video on purpose. In Lecture 9, a world model forgets: turn away from the red door, turn back, and it's blue. Genie 3 holds about a minute of visual memory. Also in Lecture 9, a robot copying an expert makes a small error, lands somewhere the expert never went, and makes a bigger one. It's the same shape three times: you train on clean data and then live on your own mistakes. The fixes rhyme too: train on your own outputs, keep a memory, and practise.
Story 2. Pixels or geometry? Genie draws the next frame: alive, but forgetful, with nothing to export. Marble, Atlas and HY-World build a scene first: a file you can open in Blender, Unity or a headset, consistent forever, but mostly static. The geometry side comes from Stanford's two steps. NeRF in 2020 made a scene a function: ask a network what colour and density sit at a point seen from a direction, many times per ray. It's beautiful, but slow. Gaussian splatting in 2023 put millions of soft coloured blobs only where something exists, and threw them onto the screen like paint. Each blob has a position, shape, opacity and colour. Rendering went from about 0.07 frames per second, to about nine with Instant-NGP, to over a hundred. This autumn splats became a normal file format, in Blender, three.js and glTF. Speed changed what it is.
Here's a pattern that showed up in three lectures without being named. VASA-1 doesn't generate pixels for a talking face. It generates a tiny motion code for head pose and expression, then renders frames from your photo, which is why it runs at forty frames per second. Genie learned actions nobody labelled by squeezing the change between two frames into one of only eight codes, which turned out to mean left, right and jump. Gaussian splats put soft blobs only where something exists instead of asking a network about empty air. In each case the creative and technical win came from choosing what to represent. When your own project is too slow or unstable, ask what the small thing is.
Story 3 brought Lectures 3 and 7 together. A VLA starts from a vision-language model and attaches an action expert that uses diffusion or flow matching, the same generation method from Lecture 3, to output joint commands instead of pixels, about a second of actions at a time. The data problem is that robot demonstrations are scarce. λ0 from Amap solved it with HumanVerse: people wear a camera at the robot's head height, plus body and hand trackers, and just live, which is about 3.4 times faster than collecting robot data. Then three stages: learn hand-object interaction from human video, translate it into the robot's body, and finish with six hundred real demos. It reached 62.5 percent on four tasks against forty for π0.5, and only fifteen without human data. Copying alone plateaus around eighty percent, and homes need ninety-nine, so practice through reinforcement learning, often inside world models, is the next step.
Story 4 was a debate between two camps. One generates the world: predict the next frame in pixels or splats, so you can watch the prediction, train robots inside it and sell it as video. That's Genie, Gemini Omni, Atlas and NVIDIA's Cosmos 3, an open model with a reasoner and a generator that outputs video and robot actions. The other camp predicts in latent space: hide part of a video and predict an abstract summary of the missing part, never its pixels. That's Yann LeCun's JEPA. V-JEPA 2 was pretrained on over a million hours of video, then adapted with under sixty-two hours of robot video to plan moves. It imagines action sequences in embedding space, picks the one closest to the goal image, takes one step and re-plans, without drawing a single frame. The motion was: a video generator is a world model. We said robots will decide.
Lecture 10, last week, November 3rd. It began with a game: score a picture from one to ten, learn that it's Théâtre D'opéra Spatial, the AI image that won the Colorado State Fair and was refused copyright, and score it again. The picture didn't change by one pixel; only what you knew about it changed. The four stories: looking, measuring, showing and signing, about two thirds art and one third technique. It pulled together questions you'd been circling all term, and it's the lecture closest to what you'll do in your final presentation, because there you have to argue that your work is good and that it's honestly yours.
Story 1 started with history, because the question is old. In 1859 Baudelaire called photography the refuge of every failed painter, and then he sat for Nadar. In 1917 Duchamp's friends defended a signed urinal with one sentence: "He CHOSE it." That's still the best argument for AI art, with the warning that Duchamp's choice cost him something. Around 1969 Vera Molnár drew with a plotter and was mostly ignored. Then this year: on October 2nd Pope Leo told artists there's an ontological difference between art and what a machine generates, and three weeks later an artist answered him in an open letter. And Merriam-Webster's word of 2025 was "slop": low quality, in quantity. What's the opposite of slop? Not handmade. I've seen plenty of handmade rubbish. The opposite of slop is decided.
Last week's most useful slide, with the whole semester mapped onto it. Is it AI? That's fact, answered with evidence: credentials, watermarks, the maker's log. Is it good? Judgment, answered by looking and arguing. No machine settles it, and Lecture 4 already warned that a score answers a narrow question. Is it fair? Ethics: whose data, whose consent, whose likeness, whose job. That's Lectures 5, 8 and 9. Is it allowed? Law, and it differs by country and by festival: the lawsuits in Lecture 5, the SAG-AFTRA contract in Lecture 8, the EU's Article 50 last week. Most fights online happen because someone asks question two and someone else answers question three. Remember the label experiment from that night: telling people an image was made by a human barely changed how much they liked it, but changed how much they thought it was worth by more than three times as much. In your final presentation, say which question you're answering.
So if scores can't decide, how do you judge? With methods. See, Think, Wonder from Harvard's Project Zero: what do you see, only facts; what do you think about it; what does it make you wonder? Liking comes last. Feldman's four steps: describe, analyze, interpret, and only then judge. We did it on Edmond de Belamy. And the four questions from Eryk Salvaggio's reading, made for AI images: what's here too often, what's missing, where does it break, and most important for you as makers, what did the artist fight? Which part goes against the default? That's where your authorship lives, and it's what the panel will look for on the 17th and 24th.
如果分数不能决定,怎么判断?用方法。哈佛Project Zero的“看-想-疑”:看到什么,只说事实;你怎么想;它让你好奇什么?喜欢放在最后。Feldman四步:描述、分析、解释,最后才评判,我们用《Edmond de Belamy》练过。Eryk Salvaggio读物中为AI图像设计的四个问题:什么出现得太多?缺了什么?哪里崩坏?对创作者最重要的是:艺术家对抗了什么?哪部分违背了默认?那就是你的作者身份所在,也是17日和24日评审要看的。
Story 2 was the technical part, adapted from CMU and Stanford. We measured it ourselves that afternoon. PSNR compares pixel to pixel, so a picture moved by eight pixels, which you can't tell from the original, scores worse than a wrecked JPEG. FID passes thousands of images through a network, fits a blob to each cloud of points, and measures the distance. It judges a crowd, never one image, so there's no such thing as the FID of your film, and a copycat that replays its training set scores a perfect zero. VBench stops giving one number and gives sixteen, like a critic saying "strong motion, weak hands." Elo from blind votes: a hundred points means winning about 64 percent of votes, and forty-three points is about 56, nearly a coin flip. And the LAION aesthetic score, trained on ratings from a photo contest site, chose Stable Diffusion's training data. Vera Molnár scored 4.87, below the cut-off.
Here's every score we met this term on one slide, each with its trap. Raw frame difference, Lecture 2: the frozen video wins. PSNR and SSIM, Lecture 10: an image shifted by eight pixels, which you can't tell from the original, scores worse than a wrecked JPEG. FID: it judges a crowd, never one image, and a copycat that replays its training set scores a perfect zero. Sync-C and Sync-D, Lecture 8: Wav2Lip scored better than real video, because it was trained against that judge. The LAION aesthetic score: Vera Molnár's plotter drawing fell below the training cut-off. Elo: a 43-point gap between first and tenth is about 56 percent of votes, nearly a coin flip. The common law is Goodhart's: when a measure becomes a target, it stops being a good measure. So use scores to compare versions of your own pipeline, and never let one pick your work for you.
Story 3 went to museums. Dataland opened in June in Los Angeles: 25,000 square feet, Refik Anadol's Large Nature Model, wristbands, scent, even taste. The critics disagreed, and the useful exercise was asking which of our four questions each critic was answering. Museums make claims too, like permission-based data or one iPhone charge per visit, and a good critic asks how you'd check them. Then Madison: Beeple's Diffuse Control transformed the museum's own collection through a diffusion model. A letter signed by more than nine hundred people objected. We found two layers of consent: the collection, which the museum could answer for, and the model's training data, which no single museum can. And at the V&A, Vera Molnár. The same word, "generative", covers very different decisions: she wrote the rule, chose about one percent randomness and could explain every line.
Here's the fairness question, gathered from four lectures. Consent has layers, and one answer rarely covers all of them. The training data: Lecture 5's lawsuits and memorized images, and the second layer at Madison, the billions of images behind the model, which no single museum can answer for. Likeness: Lecture 8's Tilly Norwood, and the 2026 SAG-AFTRA contract requiring informed consent and fair pay for digital replicas, including for actors who have died. Places and bodies: Lecture 9 asked who owns the scan of a place, and whose homes become robot training data. And the collection: Madison could answer this layer, because it had the rights to the works on screen. When you write the consent line in your credits, say which layers you can answer.
And finally, signing. There are three layers with three different promises. A visible label promises only that someone told you. Signed metadata, C2PA, hashes the file, writes a claim, signs it and embeds it. It's tamper-evident, because our edited file read as invalid, but stripping it takes one line of code: a re-save, a screenshot, a resize. So a credential proves a lot when it's there, and its absence proves nothing. The pixel watermark is the third layer. The EU's art exception says you must still disclose, but in a way that doesn't spoil the work, so end credits can be enough. Then use the credit block. It counts your human work: forty-one attempts, six selected, edit and grade. It should read like a statement of what you did, not an apology.
The technical half of Story 4, as we tested it that afternoon. C2PA works like the padlock in your browser. Hash: boil the file's bytes down to a fingerprint, and one changed bit gives a completely different fingerprint. Claim: write the facts, such as created, digital source type "trained algorithmic media", and training and mining not allowed. Sign it with a private key, and embed it in the file. Note what it doesn't say: it doesn't say the claim is true, only that this signer said this about this exact file. Our demo also read "signing credential untrusted", because nobody had agreed to trust my certificate. Then we broke it: a JPEG re-save, a screenshot and a resize all removed it silently, while editing pixels with the manifest kept read as invalid, a hash mismatch. Last, the watermark. The open DWT-DCT method kept a hundred percent of its bits untouched, sixty-seven percent after a normal JPEG save, against fifty for a coin flip, and almost nothing after a crop, rotation or blur. SynthID is a learned watermark built to survive that, and it's what caught the Nikon video.
Now let's turn the review into your final presentation. The course promised six outcomes, and each one maps to something from this term. Explain a pipeline: draw your own four-stage diagram with real model names and versions. Direct a coherent work: show decisions, not just outputs, with your edit contract or beat sheet. Build and document an experiment: one variable at a time, attempts counted, with a side-by-side, as Lecture 1 asked. Critique with evidence: run the four steps on your own work and say what you fought against the default. Account for AI use: your credit block, with versions pinned and a consent line. Present a refined project: one clear intention, and what you'd do next. If each of these gets one slide, you have a strong presentation.
The practical part. Tonight, after the break, is the last week for weekly presentations. November 17th and 24th are the final presentations, so check your slot on the sign-up page and add your slides link. Assignments 2 and 3 are both due November 30th at 23:59 Hong Kong time. Every quiz, including tonight's Quiz 11, is due at the end of the semester, and you can retrieve and edit your answers until then. Quiz 11 is a review quiz, and each question asks you to connect two lectures, so it doubles as preparation for your final.
Let me end with two sentences, one from the first night and one from the last. From Lecture 1: when an output surprises you, don't only ask whether you like it. Ask which decision produced it, and what controlled comparison would teach you more. That's where technical literacy becomes creative agency. And from Lecture 10: generating is now cheap; looking, deciding and signing your name are not. Between those two sentences is everything we did this term. Thank you. Let's take a ten-minute break, and then the last weekly presentations.