Creatures bred for speed grow really tall and generate high velocities by falling over. An evolved player makes invalid moves far away in the board, causing opponent players to run out of memory and crash. A game-playing agent accrues points by falsely inserting its name as the author of high-value items.

These bizarre exploits and dozens more can be found in the list of specification gaming behaviours, a document put together by Deep Mind Safety Research. Specification gaming is when an agent, like an AI, tries to succeed on a task by following the letter of the law rather than the spirit. Reward-shaping a soccer robot for touching the ball caused it to learn to get to the ball and vibrate touching it as fast as possible.

The list of specification gaming behaviours makes it clear just how tricky alignment can be. Even the simplest AI is lazy and alien, and will always be looking for a way to cheat. Some of the specification gaming behaviours are just creative solutions to the stated goal, like a four-legged robot learned to drop the ball into a hole in its leg joint and then walk across the floor without the ball falling out.

Some exploit the machinery of the simulation itself, like an evolved algorithm exploited overflow errors in the physics simulator by creating large forces that were estimated to be zero, resulting in a perfect score. Another common exploit is that when given the opportunity, agents will simply kill themselves. Death is the most terminal goal of all.