HeadlinesBriefing favicon HeadlinesBriefing.com

Go Runtime Bug: 32-Bit Embedded Linux Crash Analysis

Hacker News •
×

Our daily work usually revolves around Linux and security topics, deep down in the software stack. Still, more often than you might think, we end up debugging applications which live much higher up. Sometimes such a problem has its roots in the Linux kernel, sometimes elsewhere.

In this blog post we show how we found and fixed a bug inside the Go runtime. Recently a customer reported that an application written in Go crashes from time to time on one of their embedded Linux systems. The crash was always the same fatal error with the signature: runtime: netpoll: eventfd ready for 5.

At first we assumed that the application itself was buggy. But after inspecting the error more closely, it looked much more like an internal assumption in Go's netpoll mechanism no longer holds. The error message comes from netpoll() in src/runtime/netpoll_epoll.go: if ev.

Events != linux. EPOLLIN { throw("runtime: netpoll: eventfd ready for something unexpected") }. In this code path, the netpoll code expects EPOLLIN to be the only firing event, but it got 5, which is EPOLLIN|EPOLLOUT.

Why would epoll suddenly report more than EPOLLIN if the code asked only for EPOLLIN? Before digging deeper, we threw the error message into a search engine, hoping that somebody else had faced the same issue before. This led us straight to a report in the Go project's issue tracker: runtime: netpoll: eventfd ready for something unexpected. The issue describes exactly the fatal error we saw.

Also on a 32-bit ARM embedded Linux system! The reporters also noted that the crash happens in applications which run for a long time. That matched our customer's description, too. Bingo! The issue had been open and unresolved since March 2025.

The Go maintainers had also rejected one attempt to fix the problem. From the comments on the issue we learned that the fatal error only ever showed up on 32-bit ARM and i386 Linux systems, with all kinds of kernel versions. Some kernels were rather old, others recent.

Not a single reporter saw it on an x86_64 or arm64 system. Initially we suspected that epoll behaves differently on 32-bit ARM or i386. We ditched this idea quickly since epoll is generic core code in the kernel.

Why would it return a spurious event set only on 32-bit ARM or i386? Still, the fact that the error showed up only on 32-bit systems gnawed at us. As a next step we reviewed the epoll usage in Go's netpoll code. With the help of an LLM we went through src/runtime/netpoll_epoll.go, focusing on 32-bit pitfalls such as integer conversions.

The review revealed that Go's netpoll code uses the data field of struct epoll_event. Linux epoll can store an 8 byte cookie in the kernel and returns it as part of the firing event to user space. Applications use this cookie to attach metadata to an event, for example to tell different event sources apart. struct epoll_event { __poll_t events; /* ev.

Events in Go netpoll */ __u64 data; /* ev. Data in Go netpoll */ }; Deep inside the Go runtime, the main event handler needs to know whether an event belongs to an event fd or a socket fd. It decides by comparing ev.

Data to the address of its internal event fd object: if *(**uintptr)(unsafe. Pointer(&ev. Data)) == &netpoll Event Fd {...}.

Reading further through the code showed that ev. Data holds either a raw pointer to netpoll Event Fd or a tagged pointer to a per-socket object, poll Desc. The pointer tag is a counter, fdseq, which distinguishes recycled poll Desc objects.

So far so good. Mixing raw and tagged pointers in the same field looked fishy to us. But how this relates to the crash was not clear yet.

Inspecting how Go lays out tagged pointers in memory finally revealed the root cause. On 32-bit systems, a pointer occupies 4 bytes while the data field expects 8 bytes. This size mismatch causes truncation when storing a pointer value.

The upper 4 bytes get zeroed out during the conversion. When the runtime later compares the truncated pointer against the original event fd address, the comparison fails because the addresses no longer match. This explains why the bug exclusively affects 32-bit ARM and i386 Linux systems and why it manifests after the application has been running for some time, as recycled poll Desc objects trigger the mismatch.