Hugging Face releases Transformers MIC, a shrunk-model test

A team that runs a model-backed feature already knows the test problem. Full integration tests load a real checkpoint on real hardware. Running every test on every pull request would need a fleet of GPUs busy all day. A scheduled run finds a regression after it is already in main. Hugging Face published Transformers Model Integration Contracts on 3 November to close that gap. A contract packs a shrunk model, a runnable example and a reviewed output into one object. A CPU runner checks it on each change.

The bug the small model found

Hugging Face says the design was proven on a real regression. On 30 July 2026 the team enabled PyTorch's scaled dot-product attention for T5. A side effect let UMT5's decoder see future tokens during training. Nothing crashed. The bug stayed in main for almost two months. A small UMT5 fixture flagged the problem on every commit between 30 July and the fix on 28 September. Hugging Face reverted the fix on a test branch. The contract failed with log-probabilities 1.22 off the baseline. Training loss was about 0.20 off. The CPU runner step took 13.4 seconds, not counting setup.

What is in a contract

Each contract has three parts. A fixture is the shrunk model, built from a real checkpoint. It keeps every kind of layer the real model has, with fewer and narrower ones. The tokenizer and processor files are kept byte for byte. A usage path is a runnable Markdown example. It generates text, runs a training step or whatever the model is used for. Baselines are the outputs recorded on a known-good build, reviewed by a human. The contract holds if a new build produces the same outputs on the fixture.

The shrink numbers

The experimental UMT5 fixture is 66 MB. The source checkpoint is 1,227 MB. Llama 4 Scout drops from 108.6 billion parameters to 44.7 million. Its vision stack goes from 34 layers to 2. The language stack goes from 48 to 4. Hugging Face publishes the fixtures on the Hub. Each contract is pinned to a frozen revision and known package versions. A later run can then be reproduced.

Random weights or trained weights

The team tried two ways to build the weights. Random weights with a fixed seed is cheap. Overfit weights trained on a few examples make the model respond to real inputs. The random Whisper fixture caught 18 of 24 injected faults. The overfit fixture caught 24. Hugging Face says the study used synthetic faults. No real regression was found that only the overfit version catches. Both versions missed an assisted-decoding bug that only the full Whisper checkpoint exposed.

Where it stands

As of 1 November, the MIC job in the Transformers CI reports failures but does not block pull requests. The team says the open questions are measured next. How many real bugs does it catch on new changes? How many false alarms does it send? How much review does a baseline change take? Transformers supports more than 750 model types. The suite has 88,000 CPU tests on main. A daily run adds over 200,000 GPU tests. MIC coverage is still partial. A team can add a contract for a model that is not yet covered.

Source

Transformers Model Integration Contracts, Hugging Face, by Tarek Ziadé. Repository: huggingface/transformers-mic.