Shopify cut its Sidekick agent's prompt from 6,000 to 1,500 tokens

Shopify's engineering team, in a post by Cody Mazza-Anthony and Paige Vegna published on 19 August, described how it uses "gisting" to compress the system prompt on its Sidekick GraphQL agent. InfoQ picked the post up on 3 September, which is when it started circulating to a wider audience.

What was done

Sidekick is Shopify's shopping assistant. The GraphQL agent inside it turns natural-language merchant questions into GraphQL against Shopify's schema, and its system prompt had grown to roughly 6,000 tokens per request. Every request paid the cost of that prompt, and on dedicated GPUs a long prompt sets a limit on how many requests the servers can carry.

Shopify replaced the 6,000-token prompt with about 1,500 learned "gist" tokens, a 4:1 reduction. The team says the compressed prompt produced no drop in prediction quality on the agent's workload.

How the technique works

A gist token is a special token added to the model's vocabulary. The team trains the embeddings for those tokens by knowledge distillation. The model is run twice per training example: once with the full prompt, to get teacher logits, and once with the gist tokens in place of the prompt, to get student logits. The gist embeddings are updated to minimise the KL divergence between the two, so the model behaves as if it had seen the full text. The rest of the weights stay frozen.

Once the embeddings are trained, they are written straight into the model's embedding matrix, and the gist tokens are registered as special tokens in the tokenizer. The model then serves normally, and inference requests substitute the gist token string for the original prompt. Shopify credits the approach to a 2022 paper by Wingate, Shoeybi and Sorensen on prompt compression by contrastive conditioning.

The production numbers

Shopify measured the serving impact at 350 requests per minute. Median time to first token dropped from 438ms to 354ms. Median end-to-end request latency dropped from 6.8 seconds to 4.2 seconds. Throughput rose from 20.2 to 23.4 queries per second. Together, the team says these gains let it serve the same GraphQL traffic on 14 percent fewer GPUs.

The team also names limits worth reading before copying the approach. 4:1 was the ratio at which quality began to degrade on Shopify's domain, and it says other domains will land elsewhere. Initialising the gist embeddings from the mean of the corresponding prompt chunks, rather than random noise, cut the starting loss by a factor of seven. And averaging the training loss across the batch, rather than per response token, was needed to stop the model from hallucinating on longer answers.

Prefix caching helps once, gisting shortens every request

Prefix caching already handles the case where many requests share the same prompt tokens verbatim, but it only helps once and only for exact matches. Gisting is a training-time cost that reshapes the prompt itself, so every request thereafter is short. For a team running an agent with a several-thousand-token system prompt on paid dedicated hardware, Shopify has now published concrete numbers on both sides of the trade-off: what it costs to distil, and what a production workload gets back in latency, throughput and GPU count. Those numbers replace the "it depends" that this kind of decision has had to rely on.

Source

• Gisting: cutting inference costs and latency with learned prompt compression, Shopify Engineering by Cody Mazza-Anthony and Paige Vegna, 19 August 2026
• Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens, InfoQ by Sergio De Simone, 3 September 2026