Google releases a C++ API for writing on-device ML models by hand

A team releasing an AI feature on a phone or a laptop has two ways to make it fast. One way is to tune the compiler that converts the model. The other is to rewrite the hot path by hand in each backend's native code. Google just added a third.

Google published the LiteRT Tensor API on its developers blog on 19 October 2026. InfoQ wrote it up on 3 November. The authors are four Google engineers: Ping Yu, Quentin Khan, Chintan Parikh and Lu Wang. The code is on GitHub under google-ai-edge.

What it is

LiteRT Tensor API is a C++ library. A developer describes a model's compute graph by calling operators on Tensor values: FullyConnected, Transpose, Reshape, BatchMatMul, Softmax and more. The same C++ code then runs on every backend LiteRT supports. The backends are CPU (through XNNPACK), GPU (Metal on Apple, OpenCL and Vulkan on Android, WebGPU in browsers) and NPU. Buffers pass between pre-processing, inference and post-processing without a copy.

Google names Apple MLX and llama.cpp as the inspiration. The gap it fills is the one most on-device ML projects hit. A converted .tflite file is easy. A custom kernel rewritten for each platform is painful. Hand-authoring the graph once in a portable C++ API sits between the two.

What the demo shows

The blog post includes an interactive demo. It authors Meta's Segment Anything Model 2 in the Tensor API. Gemma 4 E2B handles the natural language prompts. Google states two numbers on that demo. On a MacBook Pro, the C++ version runs about 1.5 times faster than Apple MLX in fp16. That is measured over the image encoder and the mask decoder. Pre-processing and post-processing add about 0.34 milliseconds per frame. The output masks match Meta's PyTorch reference pixel for pixel.

Google also says an AI coding agent moved the same model from under 10 frames per second to over 30. That was at 512 by 512 on WebGPU. The figure is Google's own, on an agent Google ran. The post does not name the agent or the number of runs.

What is not there yet

Google calls out four work-in-progress items. Direct ML Drift backends. NPU-friendly operations. Automatic full-graph quantization. And Python authoring with .tflite export. The current API is C++ only, so a researcher who writes models in Python has to wait or cross over. Dynamic-shape support is also still being widened.

The post is a developer article, not a product commitment. There is no deprecation notice on the existing LiteRT converter path. Teams already using .tflite do not have to move.

This is a real third option for a team that hit a ceiling with converted models. It also saves the team from writing three native versions by hand. The deal is accepting a C++ graph authoring layer in the build. If you already keep native code on these platforms, that cost is small. If your model pipeline is Python end to end, watch this one, do not act on it this week.

Source

Native model authoring for edge with LiteRT Tensor API, Google Developers Blog