AI NewsInfrastructureAnnouncement

Together AI releases Storage Accelerator, a free cache for GPU clusters

Together AI released Storage Accelerator, a cache on cluster NVMe drives that fetches each model file once per cluster. It is free on Together GPU Clusters.

AI News

Editorial2 min read

LinkedInX

Why it mattersA team training on 200 nodes pays egress 200 times for one checkpoint. A cache that fetches once per cluster cuts that bill to one.

Together AI released Storage Accelerator on 19 November. It is a cache on the NVMe drives of a GPU cluster. The cache serves repeated reads of the same model files. Together says every node in the cluster fetches a file once at most. The feature is free on Together GPU Clusters.

What Together measured

The numbers in the post are Together's own. Warm reads run at up to 3.4 GB per second per node. Together calls that about 20 times faster than cross-cloud origin access. A 2 GiB object pulled from Cloudflare R2 takes 13 to 15 seconds. From the cache it reads in 0.6 to 0.7 seconds. Small files between 5 KB and 2.4 MB see 60 to 80 times lower latency. Cold reads, when the cache has to go to the origin, are within 5 percent of a direct download.

Together frames the egress saving as a per-cluster multiplier. A 70 GB checkpoint fetched by 200 nodes downloads once from the object store. That is 200 times less egress for that one file.

What you have to change

Together says the integration needs no SDK change. A user sets two environment variables on the cluster. The existing boto3, transformers and CLI workflows route through the accelerator. It supports S3-compatible stores (S3, Cloudflare R2, Together's own T4) and Hugging Face Hub as origins.

Who this is for

Teams that run training or long batches of inference across many GPU nodes on Together's clusters. The cache helps the exact workloads that reach for the same checkpoint from every node at once. Distributed fine-tuning. Multi-node reinforcement learning. Large eval sweeps. Serving the same model from a pool of replicas.

The post names one case where the accelerator does not help. A single-node workload that reads each file once has a hit rate of zero. The overhead is small.

Teams on another cloud or on self-hosted GPUs do not get this directly. The post gives enough detail about the architecture that a team could build its own cache in front of S3. Teams on Together's clusters set two environment variables and the warm-read graph changes.

A cache between GPUs and object storage changes what a cluster costs. Egress on repeated reads is one of the hidden bills in distributed training. A tool that cuts it to one fetch per cluster is worth checking against a real workload. Take a job where 200 nodes need the same 70 GB file. Together's numbers put the egress bill down by a factor of 200.

Source

Introducing Together Storage Accelerator, Together AI.

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.

Share
LinkedInX
Start a project