Mesh LLM reframes large language model inference as a peer-to-peer problem. Instead of renting GPU time from a centralized provider, teams pool existing hardware — laptops, office workstations, closet servers — into a single OpenAI-compatible API at localhost:9337/v1. The 18 MB binary joins a mesh of iroh endpoints, each identified by a public key, with no central coordinator. Requests resolve three ways: execute locally, route to a peer already holding the model, or pipeline across multiple machines when the model exceeds any single GPU's memory.

The pipeline mode, called Skippy, partitions models by layer ranges — layers 0–15 on one node, 16–31 on the next — streaming activations over a dedicated QUIC ALPN (skippy-stage/2). The main mesh-llm/1 connection multiplexes gossip, HTTP inference tunnels, route queries, and plugin RPCs over bidirectional streams tagged by a leading byte. Two iroh relays provide NAT-traversal fallback, so nodes connect directly across the internet without manual port forwarding.

The catalog includes 40+ models up to a 235B mixture-of-experts giant. Plugins declare capabilities via manifest, exposing them over MCP, HTTP, and mesh events. A mobile client using iroh's Swift SDK is in development, with plans to adopt the emerging ACP agent standard. Private deployments simply restrict the gossip admission policy.

This architecture shifts the leverage point: organizations keep data on-prem, avoid API price escalation, and scale horizontally by adding commodity hardware. The trade-off is operational complexity — managing model distribution, version compatibility, and pipeline latency across heterogeneous nodes — but for teams with idle GPUs and privacy requirements, the economics are compelling.