Rate limits, retries and latency in production

The first production incident with a decision model is almost never the model being wrong. It is a 429 at the start of the hour when a batch job and the live traffic share one key, or a timeout that nobody set, leaving a request waiting on a decision it could have made by default. This page is the operations checklist we run before a Jev integration goes live.

The published limits

TypeSafe's models page gives two limits for jev-1.13.0: 250,000 tokens per second and 1,200 requests per minute, with a note that they can be adjusted [1]. The same page gives the context limit: 64k tokens per request, of which 32k is the ceiling for the state plus the longest question [1].

Those two limits interact with request shape. At 1,200 requests per minute, a product doing one Jev request per user message can serve 20 messages per second on one key before it reaches the request limit; the token limit at that rate is 12,500 tokens per request, which a state of a few thousand tokens never approaches. A batch job that screens a backlog is the opposite: it will reach the request limit long before the token limit, so it needs its own rate limiter in code, below 1,200, and ideally its own key so it cannot use up the capacity the live traffic needs.

The errors and what each means

The API reference lists four error codes [2]. A 401 means the key is missing or invalid, and a retry will not help. A 422 means the request body failed validation: a question without instructions, a Choice with more than 255 options, a state over the limit. A retry will not help with that either, and it should fail the request loudly in development, because it is a bug. A 429 means "You have exceeded your rate limit". A 529 means "TypeSafe is temporarily overloaded".

For 429 and 529 the docs say to retry "with exponential backoff instead of retrying immediately" [2]. Exponential backoff is a wait that doubles after each failed attempt (half a second, then one, then two), usually with a random fraction subtracted so that many clients retrying at once do not all retry in the same instant. That random fraction is called jitter.

What the SDK does by default

TypeSafe's Python SDK, the client library that wraps the API (the web service your code calls), documents its RetryPolicy [3]. The defaults: max_retries of 2, so three attempts in total; retryable statuses 408, 429 and every code from 500 to 599, which covers 529; backoff_initial of 0.5 seconds, doubling each time; backoff_max of 5.0 seconds; backoff_jitter of 0.25, which subtracts up to 25 percent from each delay at random; and a timeout of 30.0 seconds as the total budget for all attempts and delays in one call. The policy respects a Retry-After header when the server sends one, and it retries connection errors and timeouts unless told not to. The SDK's constants page gives the default per-request HTTP timeout as 10.0 seconds and the default model as jev-latest [4].

Two of those defaults need changing for a request-path decision. A 30-second budget is right for a batch job and wrong for a user waiting on a route: the request path needs a short budget with fewer retries, and the batch path can keep the long one. And the default model alias, jev-latest, moves when TypeSafe releases a new version [1]; versions, drift and monitoring is the argument for pinning.

The JavaScript SDK, @typesafe-ai/sdk on Node 20 or newer, reads the same TYPESAFE_API_KEY [5]; we have read its retry defaults only in the Python documentation, so confirm the JavaScript values in its own reference before relying on them.

Latency, and what to set the timeout to

TypeSafe reports 70 to 500 milliseconds end to end [6]. Openlayer, running its jevals guardrails through a gateway, measured a p50 of 244 milliseconds and a p95 of 371 milliseconds per request on its own benchmark [7]. A p50 is the time that half of requests complete within; a p95 is the time 95 percent complete within, so one request in twenty takes longer. TypeSafe's own cookbooks report similar figures: 0.16 to 0.31 seconds for a wide Choice over 182 options and 0.09 to 0.12 seconds for a narrow re-check [8], 0.27 seconds for 13 questions against a 53,777-character document [9], and 111 milliseconds per call in the self-consistency cookbook [10]. Those are the vendor's measurements on the vendor's tasks, and your own p95 from your own region and request shape is the number to set a timeout against.

The timeout rule we use: measure your p95 for a week, set the request-path timeout at twice it, and set the retry policy on that path to one retry with a budget that fits inside the user-facing deadline. A route that has to answer inside a second cannot afford the SDK's default 30-second budget, and a timeout that expires should be treated the same as a 529: use the safe default.

Because questions run in parallel, adding questions to a request barely changes its time [11], so latency is decided by the state size and the round trip rather than by the number of decisions. That is the reason to batch.

The safe default when Jev does not answer

Every decision on the request path needs a written answer to "what happens if the model does not reply in time". The answer is the action that costs least to undo, chosen per decision and coded before launch.

For a route: send the request to the general queue and mark it unrouted, so a person or a later retry can place it. For an input screen: hold the message for review rather than passing it, if the product can tolerate the delay; pass it with a flag and screen the output if it cannot. For an output screen: hold the reply. For a tool-call gate: confirm, never run, because a gate that lets calls through when it cannot decide is no gate. For an extraction: leave the field empty and flag it. For an eval grade: mark the check as not run rather than passed.

The point is that the default is never "act as if the model said yes". TypeSafe's confidence page says of the low band, "Don't guess" [12], and an absent answer is the lowest confidence there is.

Record every use of the default with the reason (timeout, 429, 529, connection error), because a rise in that count is the earliest sign of a limit being reached or an outage starting.

Batching to cut requests

The request limit, the latency and the cost all improve when questions share a request, and TypeSafe's parallel questions cookbook measured it. Thirteen questions (eight Nouls, two Choices, three Scores) against the 53,777-character Wikipedia article on the GDPR, five repeats each way on jev-1.12: one batched call cost $0.000497 and took 0.27 seconds; thirteen single calls cost $0.006090 and took 2.71 seconds. TypeSafe reports that as 12.2 times cheaper and 10.0 times faster, and reports a standard deviation of 0.0 across the repeats for 11 of the 13 questions, with the other two under 0.01, so batching neither shifted the answers nor added variance [9]. The saving comes from sending the document once instead of thirteen times, because the document dominates every request.

For a request-path product the rule is one request per message with every question that message might need: the route, the screen, the score, asked together and filtered in code, which is TypeSafe's speculative fan-out pattern [13]. For a batch job the rule is one request per document with every question about that document. The limit is the 32k-token ceiling on state plus the longest question [1].

Batching has one cost: a 429 or a timeout on a batched request loses every answer in it, so the safe default has to be defined for the whole request.

A launch checklist

Separate keys, or at least separate limiters, for live traffic and batch jobs. A rate limiter in code below 1,200 requests per minute on every batch path. Retry on 429 and 529 only, with backoff and jitter, and never on 401 or 422. A per-path retry budget: short on the request path, long on batch. A timeout at twice the measured p95 on the request path. A safe default per decision, coded and logged. Batching on by default, with the state under 32k tokens. Alerts on the rate of 429s, 529s, timeouts and defaults used. A pinned model version rather than the alias.

Where Reveneau fits

Reveneau runs its own eval suite through Jev with the batch-side settings above (a long budget, a code rate limiter, one request per check group), which is part of why the suite runs ten times faster on our own suite than it did with a language model as the grader; Evals with Jev describes the setup. In the products we build, every request-path decision ships with its timeout, its safe default and its alert, written down beside the question. What it costs and how long it takes to release an AI agent covers where this operations work sits in the plan, and why observability matters more with generated code is the argument for the alerts.