HeadlinesBriefing favicon HeadlinesBriefing.com

Build a Health Data Lake with DuckDB and Arrow

DEV Community •
×

Quantified Self enthusiasts juggle data from Oura, Whoop, and Garmin, but answering cross-device questions means wrestling with CSV exports and incompatible JSON. This guide builds a local data lake using DuckDB-Wasm, Apache Arrow, and TypeScript to unify disparate wearable data into a single, queryable store.

The core challenge is data normalization. Oura might report HRV as an average, while Whoop provides raw time-series. The pipeline acts as a translation layer, converting raw JSON into Arrow buffers for type-safe, high-speed ingestion into DuckDB's columnar storage, ideal for analyzing long-term health trends.

A strict TypeScript interface defines the unified schema, mapping each device's output to a common format. The final stack includes a Streamlit dashboard for visualization, letting you run complex OLAP queries locally. This approach keeps data private and fast, sidestepping proprietary vendor dashboards entirely.