HeadlinesBriefing favicon HeadlinesBriefing.com

MCP Server Audit: 33% Fail Agent Usability

Hacker News •
×

Teng Li scanned 36 popular MCP servers with mcpgrade, a Lighthouse-style scorecard. 11 of 36 scored D/F — including official servers from Mongo DB, Notion, Airtable, Todoist, GitHub, and firecrawl (57, 134 errors). The top 15 scored A, including Brave Search, Exa, Perplexity, Shopify, Apify, Airbnb, Figma, Tavily, Elastic, and Shrimp Task Manager.

Finding 1: an undocumented-parameter epidemic. Nearly every D/F server has a descriptions score of zero. Rule D004 — parameter has no description — dominates. firecrawl: 132 of 134 errors are undocumented parameters. Mongo DB and Airtable: 66 each. Schemas generated from Zod or OpenAPI strip .describe() calls, leaving models with only a name and type.

Finding 2: documentation discipline, not catalog size, drives usability. Shrimp-task-manager scored A/96 with 15 tools — every description distinct. Discipline doesn't scale by default.

Finding 3: compliance and usability are different axes. Archived Slack reference server scores A/97; actively developed commercial servers ship parameters with no descriptions.

Finding 4: static scores predict live confusion. On well-documented servers, tool-selection accuracy was 100%; on firecrawl it dropped to 84%. Refusal on out-of-scope tasks: 100% on small catalogs, 50% on firecrawl's 26 fuzzy tools.