HeadlinesBriefing favicon HeadlinesBriefing.com

MySQL CDC to BigQuery: Binlog vs Periodic Syncs

Hacker News •
×

Most MySQL-to-warehouse pipelines use scheduled SELECT-based syncs that compare current rows to previous snapshots. This approach misses deletes and intermediate updates between runs, and puts heavy load on production databases during large table scans.

Change Data Capture (CDC) reads directly from MySQL's binary log (binlog), capturing every INSERT, UPDATE, and DELETE as it occurs, in order, with complete row state. A CDC pipeline running hourly is fundamentally more reliable than a batch sync running every minute because it captures everything that happened, not just the latest snapshot.

CDC via binlog requires specific MySQL configuration: binary logging in ROW format with FULL row images (binlog_row_image=FULL), binlog_row_value_options not set to PARTIAL_JSON, a replication user with REPLICATION SLAVE, REPLICATION CLIENT, SELECT, RELOAD, and SHOW DATABASES privileges, unique server-id for each replication client, and sufficient binlog retention to cover downtime (default 30 days).

Managed platforms like Erathos handle snapshot mode selection, server-id collision avoidance, and recovery logic when binlogs are purged. Once CDC captures changes correctly, landing data in BigQuery is straightforward — each change event maps to a row operation. The priority is completeness over speed: a pipeline occasionally behind but never wrong beats one that lands incomplete data on time.