HeadlinesBriefing favicon HeadlinesBriefing.com

Dataflow Model Retrospective: VLDB Test of Time

Hacker News •
×

Eleven years after the Dataflow Model paper, its authors reflect on what aged well and what didn't. The core ideas—event time primacy, the futility of waiting for completeness, and strong consistency—remain sound. However, the analytical interface was flawed: windowing and triggering were overemphasized, and triggers were an over-engineered solution to a problem users shouldn't face. The stream-centric view missed that streams and tables are just different access semantics of the same object.

Practical evolution came from the database world: SQL, incremental view maintenance, and materialized views with freshness contracts. The authors admit focusing too much on streaming mechanics rather than finishing what databases started—making analytical streaming complexity disappear.

The completeness principle split into two successful forms: watermarks for visible streams and snapshot-consistent refresh for hidden ones. The latter reached more users by demanding less. They generalize watermarks into declared constraints on change.

They also note the batch-versus-streaming debate was mostly semantic, low-latency demand bifurcates along OLTP/OLAP lines, and they adopt a new framing: leave in, leave out, push harder. Finally, they ponder streaming's eventual disappearance beyond analytics.