A beginner-friendly guide to building a world model in Python, letting it daydream its way through CartPole, and accurately measuring when the illusion collapses. The post How to Make Your First World Model from Scratch appeared first on Towards Data Science.

Part 1: A stick on a cart, and forty years of dropping it. You have a cart that slides along a rail. Balanced on top is a pole, hinged at the bottom, doing its absolute best to fall over, because that is what poles do. Your only power is to shove the cart left or right. Your job is to not let the stick fall down. If you've ever balanced a broom or a stick on your palm you already understand the whole problem, including the counterintuitive bit: you don't keep the broom still, you keep moving your hand underneath it. This is Cart Pole, the most famous toy problem in AI.

The game describes its world with four numbers: Cart position, Cart velocity, Pole angle, and Pole angular velocity. Two ways you can lose the game: the pole tips more than 12 degrees, or the cart slides more than 2.4 units from centre and off the rail. Survive 500 steps and you've won. (In the code those 12 degrees appear as 0.2095 radians, which is just another unit for angles. Computers prefer them. Nobody knows why they had to be so ugly.)

Why this silly little game is picked? That is because it's older than most of the field (after a 1983 paper by Barto, Sutton and Anderson) and so every new method gets tested on it first. Since four numbers and two buttons means every chart here is a line you can read with your eyes, so when something goes gloriously wrong later you'll see exactly why. And because every number here came from a script on one computer, written to a file in the repo so you can check I'm not making them up. We'll solve Cart Pole the standard way first, and it'll work. Then we'll ask that solution for something slightly different, watch it fail instructively, and only then build the thing this article is all about.

Part 2: How the normal approach plays it. Suppose you got good at Cart Pole by brute force with a notebook. Every time you hit some situation, e.g. pole leaning left while cart drifting right, you'd try a shove and note how the rest of the game went. Then playing is easy: look up your situation, read both entries, do whichever has the bigger number. You understand nothing about physics; you just need a big enough notebook. That is the whole idea behind the standard solution, and the only problem is the notebook. There are infinitely many situations, like the pole can be at 3.7 degrees or 3.7003, and you'll never see the same one twice. What you need is something that can guess the entry for a situation it has never seen, by noticing 3.7003 is a lot like 3.7. Which is what neural networks are for, so we replace the notebook with a small network: situation in, two scores out. That's a DQN — "deep Q-network", where "Q" is the letter somebody picked in the 1980s for "score of doing this action in this situation". It stuck. Four numbers in, two out. That's the entire brain. Training is a loop: play a shove (early on mostly at random, because flailing is a respectable way to gather data), write down what happened, nudge the network so its scores match the rewards that turned up, repeat. The word doing the heavy lifting is reward — the number the game pays out after each shove, and the only thing the agent is trying to collect. For ordinary Cart Pole it is one of the least dramatic pieces of code ever written: One point per step survived. Look at that line a moment longer than it deserves, because that single line is where the goal lives. Everything the agent will ever want is in there.