AI NewsModels & agentsAnnouncement
NVIDIA fits a 74B agent model into 24GB of GPU memory
NVIDIA released Nemotron 3.5 Bolt, a 74B parameter agent model packed into 24GB of VRAM through 2-bit quantization. It runs on a GeForce RTX 5090 with llama.cpp, Ollama or vLLM.
Image: NVIDIA
Why it mattersAn agent model this size on a single desktop card lowers the floor for building a local, offline coding or browser agent, so a team can test one without renting a GPU per developer.
Running a 74 billion parameter agent model has needed a datacenter card. NVIDIA says its latest Nemotron release puts one on a single desktop GPU.
The Nemotron 3.5 Bolt 74B-A9B 24GB model card on Hugging Face states that the model activates 9 billion of its 74 billion parameters per token and fits in 24GB of video memory through what NVIDIA calls IQ2 quantization. The model is a Mamba-2 and transformer hybrid with a vision encoder, and the card lists a context length of up to 256,000 tokens. It is published under the OpenMDW 1.1 licence, which NVIDIA states permits commercial use and modification.
What runs it
The model card lists supported GPUs including the NVIDIA Ampere A100, Hopper H100-80GB, Blackwell, RTX Spark, DGX Spark, and the GeForce RTX 5090. NVIDIA states the quantization uses IQ2-XXS at 2.0625 bits per element on the expert layers, NVFP4 on the Mamba layers, and Q8_0 on the embedding and output layers. A separate BF16 release carries the model at full precision for teams that want to run or quantize it themselves.
NVIDIA lists six inference backends on the card: vLLM, SGLang, Docker Model Runner, Hugging Face Transformers, Ollama, and llama.cpp. A GGUF variant is published alongside the main repository for the llama.cpp path.
What NVIDIA says about accuracy
The model card gives Bolt's own scores on eight benchmarks, as NVIDIA's figures: 50.98 on Terminal-Bench 2.1, 86.43 on GPQA Diamond, 50.86 on WebArena with vision, 66.75 on BrowseComp, 52.20 on MCP-Atlas, 41.02 on SciCode, 64.06 on AA-LCR, and 28.47 on GDPVal-AA-V2.
The accompanying report from The New Stack adds a comparison, also from NVIDIA, that Bolt scores 51.0 on Terminal-Bench 2.1 against 44.4 for Alibaba's Qwen3.6-35B-A3B and 37.2 for Google's Gemma4 26B-A4B, both tested in BF16 rather than the mixed precision Bolt uses. These are NVIDIA's own numbers against models it did not train, so a buyer should run the agents it cares about rather than trust the chart.
What the 24GB figure actually covers
The quoted 24GB is for the packed model only. Spare memory for the KV cache, a vision encoder on an image input, and the browser or terminal the agent drives all come on top of that number. NVIDIA's vLLM example uses an FP8 KV cache to keep the KV cache size down and a drafter model called LiLiCorr for speculative decoding, both of which cost more memory in exchange for throughput.
A larger sibling, Nemotron 3.5 Super VL at 120 billion parameters with 12 billion active, launched the same day and targets DGX Spark hardware rather than a single desktop. A smaller Bolt variant, the E4B at 4GB, is scheduled for release later this month.
A team building a local coding or browser agent can now test one on hardware a single developer owns. The practical move is to download Bolt from Hugging Face, serve it from vLLM or Ollama, and judge it on the jobs it would actually be used for.
Source
NVIDIA Nemotron 3.5 Bolt model card, NVIDIA Nemotron 3.5 Super VL developer blog, and The New Stack
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
