Good evening, everyone, and welcome to Week 9. For eight weeks we have been making pictures that move: images, motion, long takes, and last week, video that talks. Tonight the picture stops being a picture. It becomes a world you can walk into, a 3D place you can export to Blender, and finally a body: a robot that learned what to do by watching people. Three words for tonight: worlds, splats and bodies.
Let's start with a tiny game. [Hold a cup near the edge of the desk.] If I push this cup off the edge, what happens next? Shout it out. Where does it land, what sound, does it break? Now, what if the floor were carpet? What if I pushed it gently? Notice what you just did. You predicted the future of a scene, and you changed your prediction when I changed the action. That is exactly what tonight's models try to learn: what happens next, given what I do.
Four stories tonight, same format as the last two weeks. One: video that you can steer, frame by frame, which is what people now call an interactive world model. Two: 3D comes back, and generated worlds become Gaussian splats you can export, with one of the biggest deals of the year behind it. Three: robots that learn by watching people, including a model released yesterday. Four: a real disagreement in the field. Does a world model need to draw pixels at all? We'll finish with a debate.
You already own most of the pieces. In Lecture 03 we asked what a world model predicts, we did the round-trip test with Astra, and we saw a vision-language-action model for the first time. In Lecture 07, Helios generated a minute of video in real time using a KV cache and distillation, and a humanoid learned a K-pop dance from phone video. Last week, real-time avatars that listen. Tonight we add one thing to each: an action in the loop, explicit geometry, and a real body.
Here are two slides from Lecture 03. On the left, the question 'what is being predicted?': a pixel, a compact state, or an action's result. On the right, the VLA: the robot sees the scene, reads the task and acts. In September these were one slide each, near the end of a hundred-slide lecture. Tonight each of them becomes a whole story. If you have your Lecture 03 notes, keep them open.
This is how Matt Gormley and Aran Nayebi at CMU organise their world-models lecture, which we also borrowed from in Lecture 03. Three paradigms. First, generate a 3D scene, then render it, like World Labs' Marble. Second, interactive video, like DeepMind's Genie: you press a key and the next frame appears. Third, a latent world representation, where you predict an abstract state instead of pixels, like Meta's V-JEPA. Our four stories follow this map, and robots borrow from all three.
Story one. A video model gives you a clip. A world model gives you a place: you press a key, or type an event, and it draws the next moment for you, live.
CMU's definition is simple: a world model takes the previous state of the world and an action, and predicts the next state, or a distribution over next states. The state might be a picture, a 3D scene or an abstract vector. The important word is action. A video model answers 'what happens next?' A world model answers 'what happens next if I do this?' That one extra input changes the medium from film to game.
A quick history. In 2018, David Ha and Jürgen Schmidhuber trained an agent to drive inside its own dream of a car-racing game, and called the paper World Models. In 2024 a Google team ran DOOM entirely inside a diffusion model, GameNGen, at about twenty frames per second. The same year, DeepMind's first Genie learned playable 2D platform games from internet video, at about one frame per second. Genie 2 went 3D. Genie 3, August 2025, reached real time: 720p at 24 frames per second, consistent for a few minutes. And in 2026, the public could try it, and Google started calling a product a world model.
Two ancestors worth seeing. On the left, Ha and Schmidhuber's car racer. One half is a real frame from the game; the other is the same frame as the agent's compressed model remembers it. The agent learned to drive partly inside this blurry dream. On the right, GameNGen. There is no game engine here at all: every frame of DOOM is generated by a diffusion model reacting to the player's keys. The left idea is 'practise inside a model'; the right idea is 'the model is the game'. Genie joins the two.
Genie 3. [Play.] You type a prompt, a world appears, and you walk around in it with the keyboard. It runs at 720p and 24 frames per second, in real time. If you look away and come back, the wall you painted is still there, for about a minute. And there is a new kind of control DeepMind calls promptable world events: while you are walking, you type 'a storm starts', or 'a deer runs across', and the world changes. For a director, that is a new verb.
In 2026 Google turned Genie 3 into a public prototype called Project Genie. It combines Genie 3 with the Nano Banana Pro image model and Gemini. Notice the limits: US Ultra subscribers only, and each world lasts sixty seconds. That cap tells you what these things cost to run: every frame is generated, live, for one person. People immediately made imitations of famous Nintendo games, and video-game company stocks dipped. Hold that thought for the debate.
Here is the same model doing serious work. Waymo built a variant of Genie 3 for self-driving. It generates not only camera views but lidar, the laser depth sensor on the car's roof. Engineers can control the car's actions, the layout of the street, what other drivers do, and the weather. Why? Because the dangerous situations you most need to train on, a child running out between parked cars, are the ones you can never film on purpose. A world model is a place to rehearse the rare.
In May, Google launched Gemini Omni and called it a world model in the keynote. In practice, most people meet it as a video generator and editor: you give it text, audio, images or video, and you change the camera, background or style by talking to it. Every output carries a SynthID watermark. I show this to make a point about words: 'world model' is now also marketing. Part of tonight is learning to ask, which kind?
Under the hood, the difference is one input. A video model predicts the next frame from the past frames and a text prompt. A world model also takes your action at every step: a key press, a joystick, or a typed event. The network can be almost the same autoregressive video model we met with Helios in Lecture 07. But because the action arrives every step, the model can't generate the whole clip in advance; it must wait for you and answer quickly.
Here's a real problem. To learn what happens if I press left, you need video labelled with key presses, and the internet doesn't have that. Genie's trick, shown in this CMU slide: a latent action model looks at two consecutive frames and asks, what small action could explain the change? It must squeeze the answer into one of only eight codes. Nobody labels anything. After training, the codes turn out to mean things like left, right, jump. At play time, your key press simply chooses a code.
At play time, Genie is a loop of three parts. A video tokenizer turns the frame into tokens, like the compressed representation in Lecture 03. Your key press becomes a latent action. A dynamics model, a transformer, predicts the next frame's tokens from the past tokens and the action. Decode, show it to you, repeat. Twenty-four times a second, in Genie 3's case.
Let's be the latent action model. I'll show two frames from a game you've never seen. Each row has only eight hand signals to explain the change between them. Agree quickly as a row. [Show frames from the Genie qualitative results slide, or any two frames from a platform game.] Now compare rows. Did you all invent the same actions? Probably left, right and jump appear in every row, but the rest differ. That's exactly what happens in the model: the codes are discovered, not given, so they are useful but not always tidy.
Remember the round-trip test from Lecture 03? See a red door, turn away for ten seconds, turn back. A good world model shows the same red door. A weak one shows a blue door, or no door, because it only remembers what is in its context window. It's the same drift problem we saw in Lecture 07's long takes. Genie 3 holds visual memory for about a minute. That's the main thing to test when you try any world model: walk away, and come back.
The budget. At 24 frames per second you have 41.7 milliseconds per frame to read the action, run the model, decode and display. You already know the tricks: a KV cache so you don't recompute the past, and distillation from fifty denoising steps to a few. Those made Helios real-time in Lecture 07 and the avatars real-time last week. A world model is the same budget, except now your hand is on the controls.
CMU's comparison table. Read it left to right. GameNGen: one game, DOOM. Genie 2: general 3D worlds, but short and not real time. Genie 3: 720p, general, minutes long, real time. The biggest jump isn't image quality; it's the interaction horizon, from seconds to minutes. That's what turns a demo into a place you can spend time in.
And the limits, from the same lecture. Minutes, not hours. A limited, fixed set of actions. Multiple characters interacting is hard. Text and signs in the world are unreliable. And the last point leads to Story 2: it's video. There's no mesh, no 3D file. When the session ends, the world is gone. If you want to keep the world, open it in Blender, or walk a robot through it, you need geometry.
One more slide from CMU, as a bridge. Already in 2024 the Genie team trained the same recipe on robot videos with no action labels, and the latent actions became arm movements. Keep that in mind: if a model can imagine what happens when an arm moves, a robot can practise inside that imagination. That's Story 3.
Something useful for your projects. When you direct a world instead of a clip, think in four parts. The world: place, era, weather, light. The body: who you are, how tall, how fast, which camera. The events you can trigger while you walk. And the test: walk away and come back, and try to break it. For Assignment 3, your game, a world model is a fast way to sketch a level and feel it, before you build the parts that work in Unity or the browser.
Turn to your neighbour for three minutes. If a world only exists while someone is playing it, and disappears when the session ends, what is the artwork? The model? The prompt? Your sixty seconds of play? Or the screen recording you post afterwards?
Two ways to make a world. Draw the next frame, like Genie: anything can happen, water, crowds, storms, and it feels alive, but it forgets, and there's nothing to export. Or build the scene first and render it, like Marble, Atlas or HY-World: it's a file, consistent forever, you can open it in Blender, Unity, a VR headset or a robot simulator. But it's mostly static. Both camps are moving towards each other, and Atlas, which we'll see in a minute, is an attempt to have both.
Marble, from Fei-Fei Li's company World Labs. You give it text, a photo, several photos, a 360 panorama, or a video, and it gives you a world you can walk around. The output isn't video; it's a Gaussian splat file, .spz or .ply, which you can download. Version 1.1 in April improved lighting and added worlds that expand automatically. You can view it on a laptop, a phone or a VR headset with their open-source renderer, Spark.
In September World Labs announced Atlas, the model behind future versions of Marble. The idea: every input, text, photos, depth, video, goes into one shared spatial context, and each image is pinned to a camera position in 3D. From that context it can generate new views, up to a minute of 1440p camera-controlled video, depth maps, point clouds or splats. With only two or three ordinary phone photos it can rebuild a scene faithfully. It's in early access only.
World Labs' own numbers. Three quarters or more of raters preferred Atlas for camera-controlled video, against five video models including Seedance 2.5, which we saw in Lecture 07. Two or three photos for a faithful reconstruction. Twenty-four phone-video frames per room to simulate a robot driving through it, which links to Story 3. And a lower 3D reconstruction error than specialised open models. But these are company numbers, with no paper. Treat them as claims to test.
World Labs自己公布的数据：在相机控制视频上，75%到94%的评分者更喜欢Atlas，对手包括第七讲见过的Seedance 2.5。两三张照片就够重建。每个房间24帧手机视频就能模拟机器人在里面行驶，这连到第三个故事。3D重建误差也低于最好的开源模型。但这些是公司数据，没有论文，把它们当作待验证的说法。
31. The money: a chip company buys a world-model company
Now the money. In February, World Labs raised a billion dollars, from AMD, NVIDIA and Autodesk among others. At the end of September, AMD announced it would buy World Labs in an all-stock deal worth about 8.2 billion dollars, with Fei-Fei Li becoming AMD's chief scientist. Why would a chip maker buy a world-model company? Because worlds are compute: every generated frame, and every robot practising inside a simulated world, runs on chips. The deal hasn't closed, and nobody has said what happens to Marble or Spark.
If you want something open, Tencent's HY-World 2.0. From text or one image it builds a navigable 3D world, and from photos or video it reconstructs one. The pipeline is easy to read: make a 360 panorama, plan a camera path, expand the views, and then one network, WorldMirror 2.0, predicts depth, normals, cameras and Gaussian splats in one pass. It exports to Blender, Unity, Unreal, and NVIDIA's Isaac Sim for robots. Code and weights came out in April and May.
How does a pile of photos become a 3D world? Stanford's CS231A tells the story in two steps. Step one, NeRF, 2020: a neural network that answers a question. At this point in space, seen from this direction, what colour is it and how dense? To render a pixel, you march along the camera ray and ask the network many times. You train it until its renders match your photos. Beautiful results, but slow, because you ask the network millions of times per image.
Adapted from Stanford CS231A Lecture 17 "Gaussian Splatting for Novel View Synthesis" (Silvio Savarese & Jeannette Bohg; slides adapted from MIT 6.S980, Vincent Sitzmann).
Step two, 3D Gaussian Splatting, 2023. The key idea on this slide: most of space is empty air, so why ask a network about it? Instead, put millions of small, soft, coloured blobs, 3D Gaussians, only where something exists. Each blob has a position, a shape, an opacity and a colour. To render, you throw each blob onto the screen, like a splat of paint. That's the name.
Adapted from Stanford CS231A Lecture 17 "Gaussian Splatting for Novel View Synthesis" (Silvio Savarese & Jeannette Bohg; slides adapted from MIT 6.S980, Vincent Sitzmann).
Let's count. One splat has three numbers for position, seven for shape, three scales and a four-number rotation, one for opacity, and forty-eight for colour, because the colour changes with viewing angle and is stored as spherical harmonics, sixteen coefficients for each of red, green and blue. Fifty-nine numbers, about 236 bytes. A million splats is about 236 megabytes raw. A room has one to three million. Compressed formats like .spz, which Marble uses, are roughly ten times smaller. That's why splats now stream to a phone.
Why is it fast? Instead of marching along rays and asking a network, you project every blob onto the screen as a flat ellipse, sort them by depth, and blend them front to back. That's rasterization, the same thing graphics cards do for video games. At render time there is no neural network at all, just millions of coloured ellipses.
Adapted from Stanford CS231A Lecture 17 "Gaussian Splatting for Novel View Synthesis" (Silvio Savarese & Jeannette Bohg; slides adapted from MIT 6.S980, Vincent Sitzmann).
37. The result: from 0.07 to 100+ frames per second
And here is the payoff. High-quality NeRF rendered at about 0.07 frames per second. Instant-NGP got to about nine. Gaussian splatting: over a hundred. Real time is what turned splats from a paper into a medium you can walk around in VR. Same lesson as Lecture 07: speed changes what something is.
Adapted from Stanford CS231A Lecture 17 "Gaussian Splatting for Novel View Synthesis" (Silvio Savarese & Jeannette Bohg; slides adapted from MIT 6.S980, Vincent Sitzmann).
Training. You start from the rough 3D points that photo-matching software, structure from motion, finds in your photos. Then repeat: render, compare with the photos, nudge every blob's position, shape and colour. Every so often, split blobs that are too big, clone blobs where detail is missing, and delete ones that don't contribute. On one GPU, a room takes roughly twenty to forty minutes. Generated worlds like Marble and Atlas skip the photos: a model imagines the views first, then lifts them to splats.
Adapted from Stanford CS231A Lecture 17 "Gaussian Splatting for Novel View Synthesis" (Silvio Savarese & Jeannette Bohg; slides adapted from MIT 6.S980, Vincent Sitzmann).
39. Capture or generate: two roads to the same file
So there are two roads to the same file. Capture: take sixty to a hundred and fifty photos or a video, match them to find camera positions, optimise splats for half an hour. Faithful, but you need to be there. Generate: give text, one image or a few photos; a model imagines the missing views and lifts them to 3D. You need nothing but a prompt, but it invents what it can't see. Atlas blurs the line: the more photos you give it, the less it invents.
Here's why I think 3D is really back. This autumn, splats stopped being a research format. The Blender 5.3 alpha imports and renders splats natively, in EEVEE and Cycles. three.js, which many of you use for browser games, now has a built-in splat renderer. Khronos, the group behind glTF, has a splat extension. PlayCanvas shipped SuperSplat 3.0 for editing. Companies are streaming 4D splats of live sport. For your projects, that means a splat world can go straight into your normal tools.
If you want to try this tonight. Capture: walk slowly around the subject, sixty to a hundred and fifty photos or a one-minute video, even light, no people walking through. Train with a desktop tool like Postshot or Brush, nerfstudio's gsplat if you like code, or just a phone app like Polycam or Scaniverse. Clean it up in SuperSplat in the browser: delete the floating blobs and the sky. Then use it in Blender 5.3 or the browser. One tip: a splat keeps the light you captured, so shoot at the time of day you want the scene to feel like.
Three art prompts. First, a place about to disappear: a shop, your grandparents' flat, a street before redevelopment. A splat freezes a place the way a photo freezes a moment. Second, an impossible join: give Atlas or Marble two unrelated photos and let it invent what connects them, and look at what the model thinks belongs in between. Third, a level for your Assignment 3 game: scan a real corner of campus, then break it on purpose.
Three minutes. A photo freezes a moment; a splat freezes a place, walkable, from any angle. What should you be allowed to scan? Your room, fine. A shop? A temple? A museum gallery full of other people's art? And what about a stranger's home, reconstructed from photos they posted online, which Atlas says needs only two or three? Who owns the copy?
Story three. In Lecture 07 a robot learned one dance from one phone video. Tonight: robots learning many everyday tasks from watching people, and from simulators built out of video.
Where we left robots. In Lecture 07, the K-pop humanoid: estimate the 3D pose from a phone video, retarget it onto the robot's body, and train a controller in simulation. In Lecture 03: an image location and a robot movement use different coordinate systems. Copying a dance is retargeting one motion. Tonight is harder: picking things up, opening fridges, wiping tables, from video of people.
Under the hood, what is a VLA? Chelsea Finn's Stanford course shows the common recipe. Start from a vision-language model, which already knows objects and words from the internet. Attach an action expert, a small network that uses diffusion or flow matching, the same generation method from Lecture 03, to output continuous joint commands instead of pixels. Train on robot demonstrations, plus web tasks so it doesn't forget language, plus human video. It outputs a short chunk of actions at a time.
Why chunks? A robot arm running at fifty hertz has to decide every twenty milliseconds. If it decides one tiny step at a time, it hesitates and jitters, especially when the demonstrations disagree. So modern policies predict about a second of actions at once, execute part of it, then re-plan. It's the same idea as generating a whole clip rather than one frame: you get smoother, more coherent motion.
The classic problem with learning by copying. The robot imitates the expert, but never perfectly. One small error takes it to a situation the expert never demonstrated, so it has no idea what to do there, and makes a bigger error. Errors compound. It's exactly the shape of drift in long video from Lecture 07: train on clean data, then at test time you feed on your own mistakes.
And the big one: data. Language models had the whole internet. Robots don't. Teleoperation, a person steering the robot, gives precise data but slowly and expensively. Videos of people are everywhere, but a human hand isn't a robot gripper, and our bodies move differently. Finn's slide calls it the embodiment gap. Both of tonight's robot stories are about bridging it.
λ0, released yesterday by Alibaba's mapping company Amap. It's a VLA for a humanoid's whole body: walking, bending, carrying, using both hands. The robot is a Unitree G1 with five-fingered hands. What's new is the data: most of what it learned came from a dataset called HumanVerse, five hundred hours of people doing everyday things, with no robot present.
How do you make human video useful for a robot? You wear the robot's eyes. In HumanVerse, people wear a GoPro on the neck at the same height as the robot's head camera, a body tracker recording twenty-four joints, and hand tracking with twenty-one joints per hand. Then they just live: tidy, carry boxes, water plants. The team estimates it's about 3.4 times faster than collecting the same data by teleoperating a robot.
Training happens in three stages. One: learn how hands interact with objects from seven existing first-person human datasets. Two: the key step, translate HumanVerse into the robot's point of view and the robot's joints, so a person's reach becomes a robot's reach. Three: finish with a small amount of real robot data, six hundred demos, a hundred and fifty per task. At run time, the robot gets its camera image, an instruction and its body sense, and outputs whole-body motion.
The numbers. On four real tasks, the strongest baseline, Physical Intelligence's π0.5, succeeded 40 percent of the time. λ0: 62.5 percent, twenty-five out of forty. The most interesting part is the ablation. Remove stage two, the human-to-robot translation, and it falls to 22.5. Remove all human data, 15 percent. So the human video isn't decoration; it's most of the skill. Also note that 62.5 is far from good enough for a home. Keep that in mind.
A completely different route, from Google two weeks ago. Gemini 4 Argon isn't a robot policy; it's a general model that's very good at physical reasoning and code. You give it a robot camera feed, a video of a person, or even a hand-drawn sketch, and it writes a MuJoCo physics simulation of that scene. Then it writes control code, practises in the simulator and transfers the result. In one of their demos, simulation-generated data raised real-world success from 56 to 75 percent.
Four of their demos. [Click through.] A sketch of a humanoid throwing a ball becomes a simulation. A person's first-person video becomes a simulated twin of their kitchen. Argon steers a video world model, Gemini Omni, to spell its own name with Scrabble tiles, which is Story 1 and Story 3 meeting. And a cheap SO-101 arm, the kind you can buy for a few hundred dollars, gets faster at a pick-and-place task by practising: 59 seconds, then 23, then 17.
And last week, the demo that made the headlines. Argon watched an ordinary video of a person playing cornhole, the American bean-bag throwing game. It mapped the space, worked out the whole-body joint motion, wrote the simulation itself, trained the skill in simulation, and transferred it to Apptronik's Apollo humanoid. In a real trial, Apollo landed seven throws in a row. Google says this 'video-to-sim-to-real' pipeline turns weeks of manual simulation work into hours.
For context, the model we met in Lecture 03, now in motion. Gemini Robotics 2, from July, controls a humanoid's whole body, not just the arms over a table. An instruction like 'put the watering can in the green bin on the bottom shelf' means walking, picking, walking, placing. A separate reasoning model, ER 2, plans the steps and checks progress, and the VLA moves. It can adapt to a new robot body with typically fewer than two hundred examples.
Physical Intelligence, the startup behind the π models. π0.7 in April could be steered with more than words: you can say faster, or higher quality, or show a subgoal image. It folded laundry on a robot that had no laundry training data. And on the 13th of October they opened a beta Pi API: upload your robot's demonstrations, they fine-tune and host the model. Robot brains become a cloud service. Their older π0 and π0.5 weights are open, and that's what λ0 was compared against.
Under the hood, why isn't λ0's 62 percent, or anyone's 80, good enough? Finn's lecture makes the point clearly: VLAs trained only by imitation tend to plateau around 80 percent. A robot in a home needs 99 percent or more, because a robot that drops one cup in five is not a product. And copying can't make you better than your teachers. So the frontier is letting robots practise and learn from their own attempts, which is reinforcement learning.
Here is the evidence from the same lecture. Physical Intelligence's π-star-0.6 learned a value function, roughly a sense of 'is this going well?', and fine-tuned on what worked. Throughput roughly doubled compared with imitation alone, on real tasks like making espresso. And this is where world models come back. Practising on real dishes is slow and expensive. Practising inside a world model or a simulator built from video, like Argon's, is cheap.
So in 2026 there are three ways to teach a robot. Steer it: teleoperation, precise but slow, and Physical Intelligence's open models were pretrained on more than ten thousand hours of it. Show it: human video, cheap and everywhere, but a different body, as with λ0. Imagine it: simulation and world models, unlimited practice, but the sim-to-real gap from Lecture 07. Every serious system now mixes all three.
A game. One volunteer per pair is the robot: eyes closed, moves only when told. The partner gives one-sentence commands, like 'move your right hand twenty centimetres to the left', and the goal is to put a pen into a cup. Count the commands. Then let the robot open its eyes and try again with one instruction: 'put the pen in the cup'. What changed? Vision plus language is what lets one sentence replace twenty low-level commands. That's the V and the L in VLA.
Three minutes. If robots learn from first-person video of people's daily lives, whose bodies and whose homes become training data? HumanVerse paid people to record. But there are millions of hours of first-person video online already. Who should be paid or credited? What should never be recorded? And personally, would you wear the camera for a week? For how much?
Two camps. One generates the world: predict the next frame in pixels or splats. The advantage is you can watch the prediction, train robots inside it, and also sell it as video. Genie, Omni, Cosmos, Atlas. The other predicts in latent space: predict an abstract summary of what happens next, never the pixels. Why waste effort drawing every leaf and reflection, if what matters for acting is where the cup goes? That's Yann LeCun's position, with JEPA.
The strongest open version of the first camp is NVIDIA's Cosmos 3, from the end of May. It's one model with two towers: a reasoner, which is a vision-language model, and a generator that outputs video and robot actions. There's a 32-billion-parameter version for data centres and an 8-billion one for a workstation. NVIDIA released the weights, training code and datasets under a licence that allows commercial use. What is it for? Mostly making synthetic training data and testing robot policies before they touch real hardware.
The argument for pixels. A robot's camera sees pixels and depth, so to train or test it in imagination, someone must produce pixels and depth. Atlas does exactly that: from twenty-four frames of phone video per room, it generates what any robot would see along any path. Waymo uses the same idea for crashes it can't film. And pixels are checkable: you can watch what the model believes and catch it when it's wrong.
The argument against. In JEPA, you hide part of a video and predict the representation of the missing part, an abstract summary, not its pixels. Meta's V-JEPA 2 was pretrained on more than a million hours of video, then adapted with under 62 hours of unlabelled robot video into a model that can plan robot actions. LeCun left Meta and founded AMI Labs in Paris, which raised a 1.03-billion-dollar seed round in March to build this. His argument: a model that must draw every pixel wastes its capacity on details that don't matter for acting.
Here is what that looks like in practice, from Meta's V-JEPA 2 paper. You show the robot a goal image: the cup should end up here. The model imagines many possible action sequences, but only in embedding space; it never draws a single frame. It picks the sequence whose predicted embedding is closest to the goal's, executes one step, looks again and re-plans. That's model-predictive control, with a world model that never makes a picture.
CMU's slide puts the two losses side by side. The generative loss compares the predicted next observation with the real one, in pixel space. The JEPA loss compares the predicted embedding with the real embedding, in a learned space. The structure is the same, state plus action gives next state. The whole fight is about what you compare: pictures or meanings.
How would you know a world model really understands the world? Four tests. The round trip from Lecture 03. A counterfactual: push the cup instead of lifting it, does the outcome change in the right way? Long horizon: still consistent after ten minutes, an hour? And real action: does a robot that planned inside the model succeed in the real room? Only the last test works for both camps, which is why robotics has become the referee of this debate.
Debate. The motion: a video generator is a world model. Left side for, right side against. Two speakers per side, one minute each. Use tonight's examples: Genie, Omni, Atlas, Cosmos, JEPA, the robot results. Then we vote.
Some arguments to start from, if your side is stuck. For: Genie 3 already takes actions and keeps memory; Waymo uses it for real cars; Argon steers Omni to move objects, which means the model predicts consequences; and to draw the future correctly at all, you need some model of the world. Against: most video generators take no actions, they predict but don't respond; they forget and drift, as we saw in Lecture 07; a beautiful future can be wrong; and LeCun's point, the world is not its pixels.
Four stories, one line each. Add an action to a video model and it becomes a place, and the hard part is memory. Keep the geometry and the world becomes a file you can open, edit and sell. Robots now learn mostly from people and from imagined worlds, but still fail about one time in three. And whether a world model must draw the world at all is still open; robots will decide.
How tonight connects to your assignments, both due November 30th. For Assignment 3, your game: sketch a level with a world model or Marble, or scan a real place as a splat and load it into three.js or Unity. For Assignment 2, a splat or a world-model clip can be your set, just say clearly which tool made which part. And some final-project ideas: an explorable memory, a world you can only enter once, a robot that learns your gesture.
Three readings. One, World Labs' Atlas post: read the spatial-context part and the robotics part. Two, the λ0 project page: focus on HumanVerse, the three training stages and the ablation. Three, a TechCrunch piece on Genie 3 and why DeepMind thinks world models matter for general AI; read it critically, with tonight's debate in mind. Optional for the technical students: Google's Gemini 4 Argon robotics post.
Quiz 09, five short questions, same format as before: two technique, two art, one reading reflection. Due by the end of the semester, on the course site.
Next week we step back. Now that we can make worlds, voices and bodies, how do we judge them, how do we label them, and how do we decide what to release? Bring one piece you've made this term.
A toolkit, one row per job. For a world from a prompt: Marble online, or HY-World 2.0 if you have a big GPU. To try a playable world: Project Genie or Gemini Omni. To scan a place: Postshot, Brush, gsplat, or a phone app. Edit in SuperSplat. Use the result in Blender 5.3 or three.js. If you want to touch robots: openpi and LeRobot with an SO-101 arm, and MuJoCo for simulation. Check licences before commercial use.
A one-evening workflow for a walk-in level. Pick a place. Scan it or generate it. Clean it in SuperSplat and set the scale and floor. Load it into three.js with Spark, or Blender. Add a simple collider so you don't walk through walls, and one interaction. Then playtest: walk away, come back, try to break it. Roughly three hours in total. And as in Lecture 07, keep your source photos and export settings.
And a checklist before you submit a generated world. Consistency. Scale and physics. Which parts were invented by the model, and are they labelled? Rights: did you have permission to scan, and whose photos? Export details. And disclosure, which leads into next week.
