Can you find bugs in a compiler by generating random programs? I regularly check on Gleam’s changelog and issue tracker. I am very fond of this project and the people contributing to it. But every time I see an issue that relates to code generation or different outputs between the Erlang and Java Script, it nags me that there was no way to basically “compute all the Gleam programs”, run them and see if there are any issues.

My first attempt was prompting an LLM to find edge cases. It came up with bit array combinations, nested anonymous functions, and nested use patterns. Predictably, this approach did not yield many results. $20 bucks of tokens later, it found exactly one issue, which was reported and fixed right away. One is definitely more than zero. But there are plenty of issues with “LLM fuzzing”: it’s pricey, not deterministic, and a bit like pulling the lever on a slot machine.

Another idea was structure-aware fuzzing. Fuzzers generate randomized inputs to feed into a program. They can range from totally random scrambled bytes to highly structured grammar-aware ASTs. For example, this finding by zzuf in Firefox, where flipping some bits in an image file would result in a browser crash. But fuzzers have also uncovered real exploitable security flaws via buffer overflows.

Gleam is a particularly interesting candidate. It generates code for two targets: Java Script and Erlang. We can compare output for both targets and flag differences. It has minimalistic syntax, static types, and is written in Rust. Each aspect requires its own testing approach.