HeadlinesBriefing favicon HeadlinesBriefing.com

Apache SeaTunnel Tuning Guide: Fix OOM with JSON

DEV Community •
×

A developer's 60GB machine faced repeated OOMs when scaling an Apache SeaTunnel pipeline from 'amzn_order' to 'amzn_api_logs'. Despite using STREAMING mode, the configuration triggered massive heap consumption due to large JSON columns, high parallelism, and batch sizes. The root cause was misunderstanding that streaming pipelines have multiple memory 'dams'—Debezium snapshot splits, mid-pipeline buffers, and Doris sink batches.

The solution involved reducing snapshot.split.size to 5,000 and doris.batch.size to 5,000 while keeping high concurrency. For initial loads, disabling exactly_once and 2PC stabilized memory. This SeaTunnel tuning case proves that row count is deceptive; byte size per batch is critical for stability.