Hugging Face Transformers 5.23 exports a model once for four runtimes

Taking a Hugging Face model to production used to mean one export path per runtime. A team wanting fast inference on GPU, CPU and edge wrote three of them. Hugging Face released Transformers 5.23 on 13 November. One exporter now writes to four runtimes from the same call. The release also adds seven new model families and drops a few audio dependencies.

One call, four runtimes

Every export now returns an ExportedModel. The Hugging Face team said it supports ONNX Runtime, OpenVINO, PyTorch Dynamo and ExecuTorch. The exported object can be called like the original model. It can also be pushed to the Hub. The four runtimes cover different targets. ONNX Runtime and Dynamo serve GPU and CPU. OpenVINO is tuned for Intel CPUs. ExecuTorch runs on mobile and other edge devices.

The release notes give one benchmark. ONNX Runtime with the TensorRT execution provider ran EmbeddingGemma2 7.6 times faster than eager PyTorch on an A100. The same run was 1.4 times faster than torch.compile. On a Xeon Platinum 8570, OpenVINO ran the same model 4.1 times faster than eager. It was 2.2 times faster than torch.compile on that machine. The numbers are Hugging Face's, measured on one model.

Seven new model families

Inkling2 is a multimodal model. It has 594 billion total parameters and 17 billion active, and takes text and image inputs. LFM3 is the third Liquid Foundation Model. It uses gated short convolutions with grouped-query attention. LFM3-VL is its vision-language variant. Dots 3 Note Preview is a mixture-of-experts model that takes text, image, video and audio in one pass. IQuestCoder is a code model line at 7, 14 and 40 billion parameters, with the same architecture as Llama. InternLM3 is an 8-billion-parameter instruction-tuned model, also Llama-shaped. JEV is a decision head: a linear layer on the last token, soft-capped the same way a language model head is, and marked experimental.

What breaks

The ep_router expert-parallel style is gone. The [audio] extra no longer pulls in librosa and phonemizer. Wav2Vec2ProcessorWithLM is deprecated, along with the pyctcdecode and kenlm dependencies. Default processor options moved from the call site to the processor class. Rope deltas no longer appear in ModelOutput. A team upgrading from an older 5.x release should read the breaking-changes section before pinning.

Teams run Hugging Face models on more than one target. The exporter is now one call, not a fork per runtime. The runtime still matters. The numbers above are one benchmark on one model, so a team should run its own before committing. What changed is that the surface for doing so is in one place.

Source

Hugging Face Transformers v5.23.0 release notes. Also the exporters documentation.