Hugging Face Transformers 5.17.0 adds six new model architectures, including a 780B mixture of experts

Hugging Face published Transformers 5.17.0 on 2026-09-09. The release adds first-class support for six new model architectures and rewrites how vision rotary position embeddings are computed, a change flagged as a breaking one for anyone maintaining custom vision models.

What is new to load

The largest addition is HYV4, listed under the Hy4-Preview checkpoint. It is a 780-billion-parameter mixture-of-experts language model that activates 49 billion parameters per token. Each MoE layer holds 256 routed experts and one always-active shared expert, and every token is routed to 8 of them. The context window is 1 million tokens. The architecture combines Multi-head Latent Attention, DeepSeek Sparse Attention, gated MLA with learnable attention sinks, and Independent Hyper-Connections that replace the plain residual path with parallel residual streams.

VibeVoice is a next-token diffusion approach to long-form multi-speaker speech synthesis, aimed at podcasts and audiobooks. NeoMME is a family of 260M and 800M multimodal-native multilingual encoders from H Company that processes text tokens and raw image patches in a single bidirectional Transformer without a separate vision tower. NeoMME-Retriever, fine-tuned on top of it, produces multi-vector embeddings for visual document retrieval.

Fun-ASR-Nano is an 800M end-to-end speech recognition model from Alibaba DAMO Academy's FunAudioLLM team, covering Chinese, English and Japanese with 7 Chinese dialects and 26 regional accents, plus native punctuation output and hotword customisation. KimiLinear is Moonshot AI's hybrid linear attention architecture built around Kimi Delta Attention, a refinement of Gated DeltaNet where every key channel gets its own forget gate; every fourth layer keeps a full-attention block that reuses DeepSeek-V3's Multi-head Latent Attention. Canary-1B-v2 is NVIDIA's fast multilingual ASR and speech-to-text translation model, pairing a Fast Conformer encoder with a Transformer decoder that picks its task from a prompt prefix.

The breaking change

Vision rotary embeddings, both 2D and 3D, have been folded into a single implementation in modeling_rope_utils.py. Hugging Face says any custom vision model that relies on attention-layer-level or model-specific RoPE grid interleaving has to be migrated to the centralised computation. The rest of the release is additive, with the usual round of generation, cache, and kernel fixes.

Two generation-side changes matter for production use. The decoder no longer synchronises the accelerator on every decode step, which reduces per-step overhead in throughput-bound inference. And generate no longer unconditionally downloads remote hub files during generation, which matters in restricted networks and in air-gapped setups.

For a team standardised on Transformers, six new families becoming loadable through AutoModel.from_pretrained in one upgrade is the useful part; the vision RoPE change is the one to test against before the upgrade lands in a production pipeline.

Source

• Release v5.17.0, Hugging Face on GitHub, 2026-09-09