A developer set out to replace Minecraft’s handcrafted noise functions with a learned model that can “dream” voxel terrain. By coupling a Vector Quantized Variational Autoencoder with a transformer, the pipeline produces 44 chunks arranged in a 2 × 22 × 2 grid that resemble the game’s natural landscapes. The result shows procedural generation can be driven by neural networks rather than fixed algorithms.

Creating 3‑D data at this scale poses two classic hurdles: scarce high‑quality voxel datasets and exploding compute costs. The author harvested thousands of Minecraft chunks by scripting the game to teleport through a pre‑generated world and extracting region files, yielding a dataset where terrain continuity spans chunk borders. To tame class imbalance—air and stone dominate—the training loss was weighted by the inverse log‑frequency of each block type.

At the core of the system lies a VQ‑VAE with a 512‑codebook that tokenizes 3‑D space via 3‑D convolutions, then a GPT‑style transformer learns to generate sequences of these codes. By resetting unused embeddings during training, the model avoids dead codewords and produces coherent structures such as rivers and mountain peaks. The experiment proves voxel‑level generative modeling is feasible using readily available game data.