Image: ElevenLabs
Why it mattersA team that was running speech recognition, a language model and a text-to-speech model as three services can now point its existing OpenAI Realtime code at one endpoint and let the vendor handle the handoffs.
An app wired to OpenAI's Realtime API can now point at another company's endpoint by changing the hostname and the model name. ElevenLabs launched Eleven Realtime on Tuesday, with two models, Mini and Max, that bundle speech recognition, reasoning, turn-taking and voice generation into one managed service, the company said on its blog.
Both models use a cascaded architecture internally, but a developer who signs up cannot pick or configure the underlying components. ElevenLabs is taking over the voice-agent harness: turn detection, interruption handling and orchestration all sit on its side. In return, the developer gets a single endpoint, a configurable prompt, a choice from the company's library of 17,000 voices, and support for over 90 languages.
The numbers the vendor gives
ElevenLabs posted benchmarks run by Artificial Analysis. On Big Bench Audio, which puts 1,000 spoken reasoning questions to a model, Max answered 98.8 percent correctly and Mini 75.9 percent. On a tool-use test, Max completed 87.1 percent of tasks against Mini's 60.3 percent. Time to first audio averaged 0.57 seconds on Max and 0.62 seconds on Mini. These are the vendor's figures and the report has not been reviewed by a third party.
The company recommends Max for agents that make frequent or complex tool calls, and sets Mini as the default. Mini scored 95.8 percent on a turn-taking test against Max's 94.1 percent, with the ranking reversed from the reasoning scores above.
What the OpenAI compatibility means in practice
The migration guide says Eleven Realtime speaks the same protocol as OpenAI's Realtime API over the same WebSocket transport, so an existing app can keep its OpenAI SDK and swap the endpoint and model. There is a cost. Eleven Realtime supports WebSocket only, so an app built on WebRTC has to change transport first. On WebSocket, the SDK does not handle audio for you: a developer captures and plays 24 kHz PCM16 audio themselves, which the WebRTC path used to do automatically.
The Realtime API is also separate from ElevenAgents, the company's platform for voice agents. Phone and WhatsApp channels, knowledge bases and monitoring all sit on the ElevenAgents side. The company says teams that need those should build on ElevenAgents from the start.
Trade-offs between the two models
Mini and Max can transfer calls between each other, but neither can transfer calls to or from an agent in Custom mode, where a developer still picks the underlying models. Mini does not support Zero Retention Mode or HIPAA configurations. Max supports both, in the US and the EU, but not in the India or Singapore data residency environments.
Max costs more per token than Mini. ElevenLabs says that for workloads where Mini takes several attempts to finish a task, Max's stronger tool use can make it cheaper to run. Both models are billed at the standard ElevenAgents per-minute rate plus token charges.
What a team gives up in exchange is diagnostic depth. ElevenLabs runs the models and the orchestration, so a developer no longer sees whether a slow turn came from the speech model or the language model. ElevenLabs did not respond to The New Stack's request for comment by publication.
Source
Introducing Eleven Realtime: a simpler way to build natural voice agents and ElevenLabs wants to run your voice agent. Here's what developers give up. by Amanda Caswell at The New Stack.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
