HeadlinesBriefing favicon HeadlinesBriefing.com

Missing Values: Hidden Assumptions in Data

Towards Data Science •
×

A blank cell is often treated as a defect to fix rather than a fact about the world. Missingness reflects the process generating the data, not just an inconvenience. In clinical trials, dropouts aren't random—they're often patients whose treatment failed or condition worsened. Optional income fields are skipped when users prefer privacy. Churned users in experiments never produce the intended outcome, and their absence may relate directly to the treatment received.

Sensors aren't immune either. While they can't refuse to answer, their recording depends on operating conditions shaped by what they measure. A PM2.5 sensor estimates pollution via light scattering, but high particulate loading degrades performance and causes outages. This means gaps occur precisely during the high-pollution events the network was built to capture.

This pattern repeats across the Internet of Things: seismometers clip during earthquakes, strain gauges fail under heavy loads, and smart meters drop reports during peak blackouts.

Donald Rubin (1976) formalized three missingness mechanisms. Data are missing completely at random (MCAR) when the gap is independent of all variables. Missing at random (MAR) means gaps depend only on observed values, allowing unbiased estimation through proper conditioning. Missing not at random (MNAR) occurs when the gap depends on the unobserved value itself, making the absence informative about what remains hidden.