Together AI lets coding agents write GPU code for its inference service

Together AI says coding agents are now writing the GPU code behind its live inference service. The company announced Together Compiler this week. The system generates and tests kernels for open-source models. Together says the compiler serves trillions of tokens a day in production.

The company is a cloud inference provider for open-source models. Its post names five models that the compiler has already tuned. Together reports up to 2.6x higher throughput on Kimi K3 and GLM 5.3. On Llama 1B at batch size 1, Together says the compiler beats open-source megakernels by 1.7x. On MiniMax M3.1-Flash the kernels reach 1,111 tokens per second. On DeepSeek v4.1-Flash they reach 1,205. Together calls these the best figures for low-latency serving today.

What is new

The main technical change sits in the test loop. Together calls the piece a persistent engine. The inference engine starts once and can hot-reload any candidate forward pass. A new kernel can be swapped in without a restart.

Together says engine startup drops from 467 seconds to 9 seconds. That is a 52x cut in setup time for each test. The post reports a 252x higher experiment throughput compared with baseline agent systems.

How long it takes

Together says the compiler reaches those speed-ups in under 24 hours of search. The post was written by Reyna Abhyankar, Dylan Lim and Simran Arora.

The post also mentions results from four fields outside language models: weather modelling, PDE solvers, time series and biology. Together says three of those four models beat a Claude Code baseline, without giving the numbers.

Gaps in the announcement

Together does not say which model or agent writes the kernels. The post does not describe safety checks or verification before a new kernel reaches production. It does not explain who, if anyone, reviews a kernel before Together puts it live. A reader who wants to copy the pattern will have to ask.

Together's post also does not say how much the compiler costs to run, or how often it runs. Every number in this item is Together's own, measured on models and workloads Together chose.

The result follows a slow year for machine-written GPU kernels. Together also runs a benchmark called ParallelKernelBench, cited in the post's sidebar. That benchmark reported that frontier models solved under a third of its multi-GPU test cases. Together's answer was to tighten the search loop so a weaker model can keep trying faster.

The lesson for teams building AI products is narrow. GPU kernel tuning has one measurable goal and a clear way to test each try. Most product work carries neither. Teams waiting for a coding agent to help with performance code should pick a task of the same shape. One clear goal, a fast test for each try, and one number that moves.

Source

• Together AI blog, Together Compiler, the primary source.