HeadlinesBriefing favicon HeadlinesBriefing.com

PostgreSQL to VeloDB Sync with Flink CDC

DEV Community •
×

Developers migrating data from PostgreSQL or Amazon Aurora to VeloDB can use Flink as a real-time synchronization engine. By enabling PostgreSQL's Logical Replication and creating a Replication Slot, Flink CDC captures change data for both initial full loads and incremental updates, ensuring consistency.

The process involves setting up an Aurora instance, creating a VeloDB warehouse, and installing Flink with the Postgres CDC Connector and Doris Connector. A sample job is submitted via Flink's CLI, which automatically creates target tables in VeloDB based on the source schema, handling schema mapping automatically.

Flink supports job submission across multiple environments, including Local, Standalone, Yarn, and K8S. The synchronization captures INSERT, UPDATE, and DELETE operations. To capture deletions, a specific PostgreSQL configuration like `REPLICA IDENTITY FULL` is required, ensuring complete data lineage during the migration.