Anna planned to sell her condo moved out in August 2024 and sought AI advice on closing timing for a tax break. An AI agent evaluated closing in October 2026 versus January 2027 and correctly concluded either date would satisfy the residence requirement for up to $250,000 in excluded gains. However, the agent left out a critical deadline.
Under standard rules, homeowners must have lived in the property for at least two of the five years before selling. Because Anna moved out in August 2024, she would stop satisfying this requirement around the end of August 2027. If the sale was delayed beyond that date, she could lose eligibility.
Across five attempts, Astra never identified the deadline. Opus identified it in all five attempts. The agent evaluated the proposed dates correctly but its advice left Anna unaware of a deadline that could affect a major financial decision.
This tests professional judgment even when the client does not know to ask. Surge AI's DAYJOB: Tax benchmark evaluates AI agents on professional tax assignments requiring specialized expertise and complex materials. Across 50 environments in eight tax practice areas, agents produce tax returns, workpapers, technical reviews, or client advice.
Claude Sonnet 5.5 leads with a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. The benchmark contains 1,789 workspace files with a median of 30 per assignment. Performance versus cost places Sonnet 5.5 and Opus 5.5 on the high-performance end of the Pareto frontier.
Source: Hacker News · Summarized by HeadlinesBriefing