Published September 1, 2026 |Updated September 2, 2026 In the beginning, there was mmap. It was convenient: it let us lazily read huge numbers of Arrow IPC files from disk without managing memory ourselves. It fit our file format perfectly — Arrow IPC’s layout is designed for zero-copy random access, and mmap gives you exactly that.

Then we deployed to production, ran real concurrent query loads, and mmap became a real problem. Our Workload At Conviva, we analyze trillions of events a day to pinpoint and diagnose end user experience. At the core of our architecture is an event and pattern analysis engine built on Data Fusion, Arrow, Rust, Rayon, and Tokio.

Raw events get transformed, encoded in a proprietary mostly-numeric format, and stored in the cloud. We copy them to local NVMe and read large (~3–5 GB) Arrow IPC files. We chose Arrow IPC for simplicity and speed — its memory and disk layouts are identical, so decode cost is minimal, and mmap gives us zero-copy reads natively supported by arrow-rust.

A typical query touches 6 columns across 8 batch files (one batch per file), ~1.6 GB per batch, ~13 GB total per day of data. The Test Setup Hardware: 192-core box, ~750 GB RAM. Two disk configs during the investigation: 2× NVMe LVM-striped (~5.5 GB/s fio ceiling) and 32× NVMe RAID-0 (~21 GB/s fio ceiling).

Kernel 5.15 during investigation, 6.x in production. The Production Symptom At lighter loads, mmap worked well — fast, serving queries from raw events in seconds. The trouble started under heavier concurrency.

Some latency increase under load is expected — more queries competing for the same CPU. But we saw p95s and p99s spike well beyond what linear scaling would predict, with rows scanned per core dropping sharply even after accounting for concurrency: OS page cache shrank — each pod consumed more memory as private allocations, less as shared cache A huge number of page faultsp95 spiked from ~30s to 150s+ under real concurrent load Adding pods made it worse, not better That pointed at mmap page-cache thrashing under memory pressure. Controlled Benchmark: 1 Pod vs. 4 Pods To isolate the effect, we ran a controlled test: 1 pod vs. 4 pods on the same host, same concurrent query load.

We expected 4 pods to win — more parallelism, better isolation. We were wrong. For 14-day queries — long enough to fill the page cache — 1 pod beat 4 pods by a real margin: 41% faster at max, >20% at p95.

The mmap page cache lives on the host and is shared across pods, so the 4 pods weren’t fighting each other for CPU — they were fighting for page cache. perf record on the same run showed 100% lock contention at the kernel level. The core issue: mmap’s page cache is implicit shared state. Every process on the host shares one cache, one lock hierarchy, one eviction policy.

No single pod controls the resource that matters most for read latency, and as concurrency rises, everyone’s slice of it shrinks. A Storm of Page Faults We didn’t want to just guess, so we dug into the mmap mechanics and the page fault stats. When you mmap a file, the application gets a region of memory backed by the file on disk.

Here’s what happens on access: The CPU touches a Virtual Memory Address (VMA) with no physical page attached, and throws a page fault. The kernel handles the exception: looks up the VMA, checks ownership, and acquires a lock, since another thread might be modifying or unmapping that virtual space concurrently. Linux 6.4+ has a fast per-VMA lock; earlier kernels fall back to the slower mmap_lock.

Once the kernel confirms the VMA is file-backed, it triggers a file fault — a minor fault if the bytes are already in warm page cache from read-ahead, or a major fault if it must trigger physical I/O. Recommended reading: mmap_lock scalability (LWN) and per-VMA locks (LWN, Suren Baghdasaryan’s design). Under heavy page cache contention, read-ahead runs out of room and major faults spike.

Here’s what that looked like — pidstat on one process during a stressful run...