HeadlinesBriefing favicon HeadlinesBriefing.com

DataFusion for Billion-Scale Graph Analytics

Hacker News •
×

The author implemented graph map-reduce using Apache Data Fusion, offloading operations to disk and favoring bulk scans over random access. Data Fusion handles spillover, sort-merge joins, and aggregations, resulting in lightweight code. Tested under strict memory limits, the implementation successfully computed Page Rank on a graph with one billion edges using only 5 GB of memory and identified weakly connected components in a two billion edge graph using 10 GB.

This challenges the prior assumption that Apache Spark and Graph Frames are necessary for such large-scale graph analytics, suggesting a laptop is sufficient. While encountering issues like deadlocks and limitations with pre-sorting for sort-merge joins, the core functionality works. The Page Rank implementation is a Pregel-like approach, and the weakly connected components algorithm is based on "In-database connected component analysis."

The author notes potential optimizations, such as bucketing edges or fusing join and aggregation stages. The weakly connected components task, involving nearly four billion edges at its peak with 8 GB of memory, completed in 10 minutes due to the contraction process reducing edge count.