NVIDIA NeMo Switchyard: model routing as a real primitive, or a pre-alpha wrapper?

What it is

NeMo Switchyard is an open-source model-routing library that sits between your agent code and the model providers. NeMo Switchyard is an open source model routing library for AI agents. The technology routes prompts to the most capable and efficient model for each step of an agent workflow automatically, based on specific needs.

Under the hood it's a proxy plus a decision layer. Clients keep their native OpenAI or Anthropic API format. Switchyard picks a configured backend, forwards the request in that backend's own format, and translates the response back. It offers both fixed heuristics and learned routers — NeMo Switchyard offers both tuning-free and tunable routers.

The next-work-session test

You're building a coding agent. Right now every step — file listing, diff summarization, planning, hard refactors — hits the same frontier model, and your monthly bill reflects it. In the next session, could you drop Switchyard in front of your existing OpenAI/Anthropic client and route the cheap steps to a smaller local model, escalating only the hard ones?

Directionally, yes. You keep your SDK code the same and point the base URL at a Switchyard server. You configure two or three backends and pick a stage/escalation router. The change is that model choice becomes config, not code — and you get Prometheus metrics on requests, errors, latency, tokens, and routing overhead so evals can actually compare cost vs. accuracy per step.

The catch below.

Pricing

The library itself is free and Apache-licensed. Apache 2.0 License, distributed via the Switchyard GitHub repo. There is no per-token fee to NVIDIA for using it — you pay whatever your upstream providers charge (OpenAI, Anthropic, your own GPUs). Hosting cost for the router itself is unverified; NVIDIA has not published a reference-server sizing guide that I could find, so treat "adds a network hop" as the honest baseline.

What we'd actually use it for

Two narrow, believable uses:

The Cognition claim — Cognition's implementation of staged routing in its Devin Desktop coding agent achieved near-frontier performance at a 28% lower cost — is again a partner data point, not a promise for your workload.

Limits

The big one, straight from the maintainers: Switchyard is pre-alpha software that is evolving rapidly. The API and algorithms are expected to change significantly before we reach v1.0. They also explicitly say experimental, not for production. If you build critical routing logic against 0.2, expect to rewrite it.

Other honest limits:

• Self-hosting ≠ private. Self-hosting Switchyard does not make remote inference local. Prompts still reach every configured upstream, and a router becomes a privileged point in the data path. You now have a new component to threat-model.
• Vendor numbers are vendor numbers. As one analyst put it, NVIDIA claims Switchyard reduces costs to one-third compared to relying solely on Opus, assuming consistent workload patterns — an assumption that rarely holds in real agent traffic.
• Latency overhead is real but under-documented. There's a Prometheus metric for routing overhead, which tells you they know it matters; there's no published p50/p99 budget to plan against.
• Tunable routers need your data. The learned routers only pay off if you have labeled traces of which model handled which step well. If you don't, you're on heuristics.

Try it if

• You run a multi-step agent and can point to specific steps that are obviously overpowered for a frontier model.
• You already have an eval harness and want to measure cost/accuracy tradeoffs across models without rewriting client code.
• You're comfortable pinning a pre-alpha dependency and tracking breaking changes.
• You want to keep OpenAI/Anthropic SDK compatibility while adding a local Nemotron or open model as a cheap tier.

Skip it if

• You're shipping to production this quarter and can't absorb API churn before v1.0.
• Your agent is single-model and single-step — the routing math won't beat just picking the right model once.
• You can't add another network hop to your critical path, or another privileged service to your security review.
• You don't have traces to train a tunable router and the tuning-free heuristics don't match your task mix — you'll just be paying overhead for a coin flip.

The interesting thing here isn't the cost numbers. It's that NVIDIA is treating model routing as a first-class, open, SDK-shaped primitive instead of a closed feature of a gateway. That's the part worth watching — the technical blog post lays out the routing architecture in more detail. Just don't confuse "primitive" with "stable."