A new benchmark called Dolphin Bench tests whether an AI assistant's memory actually helps it complete tasks, rather than just recalling facts. Most memory tests ask an assistant about the past, such as which coffee shop someone likes. The authors argue that approach is too easy, because the question itself signals that a fact must be looked up. Dolphin Bench instead gives the assistant a job, such as ordering coffee, and checks whether it used the right fact. Each simulated person has about 500,000 tokens of chat history and 200 tasks, and every task is checked both with and without the relevant history. Results must also report total cost and time, not only accuracy. Dolphin Bench is run by Mem0, a company that sells a memory system, and its top score of 70.67 percent came from a Mem0 setup. The author did not reproduce Dolphin Bench and tested no company's product.
Instead, the author built a smaller test. A made-up project manager, Dana Whitfield, has a year of chat history of about 40,000 tokens. An assistant completed 30 tasks requiring one or two facts from that history and answered 30 plain questions about the same facts. Four memory designs were tried on two models, across 1,260 runs, with every run logged to Weights & Biases. The designs were: passing the whole history, passing a cheat sheet summarizing it, retrieving only the matching pages, and giving the assistant no memory at all.
One result stood out. With the whole history available, GPT-4o mini answered 97 percent of the questions correctly but completed only 58 percent of the tasks. Retrieval, a cheaper method, got 89 percent of the tasks right. The full-history approach cost 20 to 31 times more than looking up notes and did no better on the tasks.
The article closes with a short script that lets readers run the same four memory designs on their own history and tasks.
Source: Towards Data Science · Summarized by HeadlinesBriefing