You read that correctly, Pandas should go extinct. Not the cute fluffy things used for international diplomacy, but the Python Data Frame library. Why? Because Pandas' inefficiencies force you to adopt distributed querying systems before your workloads justify the added complexity. I posit that most workloads will never justify those systems, they are just well marketed 'silver bullets'.

To understand what I'm talking about we first must understand the typical adoption pathway for Pandas. Following it from left to right, you also see the typical adoption pathway for data analysis tools, and the cliff that Pandas' users experience beyond a certain data size. People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas' baroque API. The traditional answer at this point is to graduate to a 'real' (read: expensive) tool like Spark, Data Bricks, Snowflake, or Dask designed for Big Data ™.

The alternatives I propose, as alluded to earlier are Duck DB and Polars. In broad strokes, Polars is a Rust-based Data Frame library that feels familiar to Pandas, but differs in several im...