Numbers, prices and tool versions are as of October 2026. This guide is for platform engineers new to inference. It explains how models and GPUs work together and how to run inference in production.
The guide covers the tools needed and how to grow into an inference engineering role. It builds on the work of others in the inference community. In an AI-forward organization, platform teams become inference teams.
Coding assistants cannot send source code to third parties, and frontier model bills increase month to month. Support tools also need internal data that hosted models cannot reach. Organizations want to run these models themselves, and open models now make that practical.
Hugging Face hosts more than two million open models, and the best compete with closed models. Deep Seek released R1 in January 2025, and it matched Open AI's o1 on math and coding benchmarks about a month after o1 came out. Running your own model controls latency, uptime, and data location.
At high volume, you can also cut costs by about 80%. The work suits platform engineers. The price of a fixed level of AI capability falls roughly 10x a year, mostly from how models are served.
Upbound, the company behind Crossplane and Modelplane, saw this firsthand. In its Project Champagne blog series, the team ran its own models and found that v LLM and scheduling worked. The time went to TLS, identity, timeouts, storage, IAM and observability.
An inference engineer picks a model, gets it onto the right GPU, keeps serving, makes it faster and cheaper, and gets paged when it breaks. That is SRE for a new workload, and most skills carry over. In August 2026, Chris Zeoli counted about 58 open inference roles on public careers boards, with no settled job title.
Inference 101 covers concepts in the order met. Most concern what the engine keeps in GPU memory or limits speed. An inference engine, like v LLM or SGLang, is the server that turns model weights into an API.
It loads a large dataset into memory...
Source: Hacker News · Summarized by HeadlinesBriefing