HeadlinesBriefing favicon HeadlinesBriefing.com

MemStitch: Zero-Copy KV Cache Sharing for Multi-Agent LLM Inference

Hacker News •
×

GitHub project DaqulaLin/MemStitch (Context-Stitcher) introduces a zero-copy context bridging gateway for multi-agent GPU inference. In collaborative workflows, multiple agents often process identical long contexts sequentially, forcing redundant prefill phases and duplicating GPU activations.

The system uses PagedAttention and Context Topological Hashing to segment prompts into physical blocks mapped to cryptographic fingerprints. Zero-Copy Block Stitching maps Agent B's logical attention table directly to Agent A's physical GPU memory addresses, bypassing prefill entirely. A Zero-Trust Secure Gate enforces boundary control lists to prevent unauthorized cache access.

Benchmarks on a shared 200-page document show dramatic improvements: Agent B's TTFT drops from 1200 ms to 48 ms, a 25x speedup. GPU cache blocks allocated fall from 53 to 30, saving 43.4% VRAM. The gateway runs on localhost:8000 with an OpenAI-compatible API and real-time developer console.

Integration supports Python SDK decorators (@stitch_agent) and REST APIs via extra_body metadata for agent_id and session_id. Dynamic security policies manage cache-sharing permissions between agents through dedicated policy endpoints.