spinf-serve enables bulk classification and decision inference on a single GPU by reading content once and scoring multiple questions in the same pass. Designed for high-throughput use cases like analyzing millions of news articles, it avoids the inefficiency of re-reading content for each query. The tool supports an OpenAI-compatible API and runs on one 24 GB NVIDIA GPU with fp8 tensor cores (e.g., RTX 4090, H100).
It is not a chat server—nothing is generated unless requested. Instead, it returns probabilities over fixed answer sets (e.g., yes/no, labels), making it ideal for decision models. Built by Spinf for internal use, it is now publicly available under Apache 2.0.
Users can install via pip or Docker, and query using JSON-formatted scoring blocks. Each response includes option probabilities, floor probabilities (baseline without content), and residuals. The system supports placeholders to expand one template into many questions, and is optimized for throughput over single-request latency.
Source: Hacker News · Summarized by HeadlinesBriefing