A new demo showcases the automatic deduction of a FlashAttention kernel using approximately 25 lines of SQL, expressing standard vanilla attention. This breakthrough relies on three portable SQL extensions designed for this purpose: xarray-sql, a library for efficiently representing petabyte-scale array datasets in the cloud as relations; ddx, a database extension adding automatic differentiation similar to JAX to standard SQL; and einfold, a modular query planner enabling fast tensor contractions in vanilla SQL. The developers propose that the relational model and algebra offer fresh perspectives on accelerator programming, suggesting that separating logical layers from physical layers in tensor programs is both more correct and scalable.
They frame this approach as a formalization of the "Bitter Lesson" in machine learning using relational algebra. Interested individuals are invited to join the project on Discord at xql.systems to explore the intersection of database theory and machine learning systems.
Source: Hacker News · Summarized by HeadlinesBriefing