HeadlinesBriefing favicon HeadlinesBriefing.com

Splitting a Pipeline into MCP Services

Towards Data Science •
×

The usual way to build a multi-part pipeline is to put every part in the same process and call between them with function calls. It's the path of least resistance, and for a while it's genuinely fine until the parts stop being small enough to treat as implementation detail. Once each one is really a mini-application in its own right, with its own dependencies, its own failure modes, and its own release cadence, sharing a process with the others stops being convenient and starts being the thing that breaks first. That's the point we'd reached, and it's why we put a real boundary, an MCP server around each one instead.

Three separate failure modes hide in the monolithic approach: an unhandled exception in extraction takes risk scoring down with it, a dependency bump for the fraud engine can break the extraction engine's install, and shipping a bug fix to any single service means redeploying the whole application. The issues show up months after release, once each engine has grown enough real logic and dependencies that "just import it" stops being free.

The actual fix isn't MCP specifically, it's a real process boundary around each service so a crash in one doesn't touch the others, a dependency change in one doesn't ripple into another, and each can be deployed, scaled, and restarted on its own schedule. What MCP adds is a single, consistent way for any orchestrator to discover what a service can do and call it, without writing a bespoke integration per consumer per service.