Image: The New Stack
Why it mattersA team deciding what to put between its application and its model vendors has one more account of how LiteLLM behaves on upgrade, and one more plugin shape to copy or avoid.
A team that gave up on its toolchain after one bad upgrade is now shipping an open-source replacement. Respan released its Rust AI gateway on 20 October, two years after it started wrapping LiteLLM in more than 100,000 lines of its own code.
The company's Hacker News launch post, cited by Amanda Caswell at The New Stack, says a routine LiteLLM upgrade in January 2026 caused intermittent connection failures. Respan rolled the upgrade back and, as the company put it, "we still haven't figured out the cause." Since then the team has pulled in upstream LiteLLM fixes one at a time rather than taking a release.
A small core with compiled plugins
The new gateway keeps its core small and moves routing, model providers, MCP tools, realtime voice and security controls into plugins, a pattern the company borrows from Nginx. Developers can write plugins against the public Rust interface without changing the core. Respan says its Mistral provider takes 22 lines of code, and a quirk such as Groq reporting token usage in a different field can be handled on its own.
Plugins are compiled into the gateway, so adding one means maintaining the code and rebuilding the binary. They also run inside the gateway process, where they can read model requests and other sensitive data. One Hacker News commenter argued that compiled crates should not be called plugins, because they force developers to maintain a build pipeline and bring supply-chain risk. Respan's founder agreed and said developers can at least inspect the source they are compiling.
Plugin order matters. The gateway runs plugins in the order they are registered, and one can answer or reject a request before the others run. Putting a cache in front of a security check lets a cached response bypass that check.
What the gateway can do that a wrapper cannot
When a request comes in over HTTP or WebSocket, the plugins decide which model handles it, can change the response, make additional calls, or stop the request before it reaches a provider. When both sides use the same format, the gateway leaves the original request and its provider-specific fields alone. It can also translate between protocols: an application written against OpenAI's Realtime API can connect to Gemini Live without changes to the client, Respan says, with the caveat that some features may not carry over.
LiteLLM is porting its own gateway to Rust at the same time, one route at a time, with the Rust core switched off by default and any request it cannot handle falling back to Python. The current beta does not run every hot path in Rust yet, including authentication and rate limiting. LiteLLM has also published benchmarks that put its Rust gateway at about seven times less overhead and nine times less memory than Bifrost, numbers from LiteLLM's own tests on its own benchmark harness.
Where this sits for a reader
Respan's gateway is pre-alpha. Developers can download a prebuilt binary from GitHub and run it without the company's paid platform, under Apache-2.0. The repository had 42 stars at the time of writing, four weeks after it was created, which puts the project in the category of a public bet rather than a tested dependency. The useful part for a team already running LiteLLM is the account of how an upgrade broke a two-year wrapper, and the question of what a plugin interface should let a developer do without touching the core.
Source
Respan spent two years building on LiteLLM. One upgrade made it start over. by Amanda Caswell at The New Stack. Primary: respanai/respan-gateway on GitHub and the Hacker News launch thread.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
