The LLM gateway market, honestly.
We mapped every router and gateway we could find — who sells routing, who gives it away, and where the gap is. Compiled October 2026. Savings figures are each vendor’s or paper’s own claim: unverified, and not comparable across rows.
Routing as the product
These sell model selection itself, not a gateway with routing attached.
| Product | Model | Claimed savings |
|---|---|---|
| Martian | Hosted proxy, passthrough + fee | 20–97% lower cost at “maintained quality” |
| Not Diamond | Recommender, not a proxy — keys never transit | Up to 10× lower cost; up to +25% accuracy |
| Neutrino AI | Hosted | — (prices at 3% of AI spend) |
| Requesty | Hosted / hybrid | — (flat 5% markup) |
| Unify · AIRouter · Respan | Hosted | — |
Gateways — routing is a feature
Routing here is rules, load-balancing and failover, not quality-aware model choice.
| Product | Owner | Notes |
|---|---|---|
| Portkey | Palo Alto Networks (acq. May 2026) | Core AI gateway of Prisma AIRS; conditional routing |
| OpenRouter | OpenRouter, Inc. | Passthrough pricing; auto mode; reported Stripe talks |
| Helicone | Mintlify (acq. Mar 2026) | Rust gateway, Apache-2 core + hosted tier |
| Cloudflare / Vercel | Cloudflare, Vercel | Free / zero-markup; bundled as loss leaders |
| Kong · Bifrost · Braintrust | Kong, Maxim AI, Braintrust | Self-host / embeddable / hosted beta |
Provider-native routing — free
The strategic threat: three major platforms now route between their own tiers at no charge.
| Router | Vendor | Charge |
|---|---|---|
| Bedrock Intelligent Prompt Routing | AWS | None beyond model tokens |
| AI Foundry Model Router | Microsoft | Passthrough |
| Auto Routing (GPT-5 Auto) | OpenAI | None beyond model tokens |
Open-source & research
Routers you can self-host, and systems with published numbers — all on benchmarks, not your workload.
| Project / system | What it is | Reported |
|---|---|---|
| lm-sys/RouteLLM | Learned router over strong/weak model pairs | 35–85% on benchmarks |
| vLLM semantic-router | Classifier-based routing, typed policy DSL | — |
| BerriAI/litellm | Gateway over 140+ providers; difficulty routing | — |
| MixLLM (research) | RL / bandit router | 97% of GPT-4 quality at 24% of cost |
| Hybrid LLM (research) | Query-difficulty router | ≤1% quality drop at 20% cost advantage |
Where the gap is
- No one shows your-traffic evidence. Only two products even claim measured savings — both hosted, neither drawn from the buyer’s own requests.
- Open-source routers report benchmark numbers (MT-Bench, RouterBench), not your-workload numbers.
- No gateway ships an evaluation harness. Nothing in the category can prove the cheaper model was good enough.
- No frontier judge on the request path. Published cascades lean on tiny encoder models; no one runs a strong LLM as the judge where it matters.
Routing is the 7th lever.
Ordered by what actually cuts the bill, routing is the 7th lever — behind caching, budget caps, batch tiers, fallbacks, prompt compression, and output control. Those first few are mechanical and carry no quality risk; the invoice simply falls. Routing is the one everyone markets, the smallest real contributor, and the only lever that can silently degrade output — which is exactly why nobody has had to prove it.
The whole market enforces or charges. Almost nobody proves.
That’s the gap we’re building into the gateway — every control proven on your own traffic, not a public benchmark.