Agoda faced a critical challenge: millions of financial data points generated daily across sales, costs, revenue, and margins were being processed by separate teams using different pipelines. Each team—Data Engineering, Business Intelligence, and Data Analysis—built its own data pipeline to meet specific needs, creating a fragmented system where no single source of truth existed for financial data.

This fragmentation introduced serious problems. Duplicate sources meant multiple teams independently pulling from the same booking databases, wasting computational resources and creating synchronization issues. Inconsistent definitions across pipelines meant financial metrics could differ depending on which team produced them, undermining data reliability. The lack of centralized monitoring resulted in duplicated quality checks and made it difficult to identify issues when they occurred.

To solve these problems, Agoda developed FINUDP (Financial Unified Data Pipeline), a centralized system built on Apache Spark that processes all financial data from millions of bookings daily. The pipeline delivers both high data availability and quality through three key non-functional requirements: hourly data freshness with automated monitoring via GoFresh, automatic data quality checks that trigger alerts and halt processing on violations, and mandatory peer-reviewed designs with shadow testing before production deployment.