Three Claude models shopped Shopify's catalog and picked different winners: Reveneau

Why it mattersA brand cannot control which model a buyer's agent runs. Each one reads the same listing differently, so one listing has to earn three decisions.

A brand that sells on Shopify now has three different buyers reading each listing. Slobodan Manic ran 1,980 agent picks through Shopify's Shop MCP connection. Three Claude models, Haiku 4.5, Sonnet 5 and Opus 5, each read the same twenty-product list for one search. Each picked a different winner. He wrote up the method and the numbers on his blog, No Hacks.

Shopify opened Shop MCP a few weeks earlier. An agent can search Shopify's catalog through it. The agent reads the product field a merchant fills in. The agent can buy on the shopper's behalf. The agent decides which product the shopper sees first.

What each model picked

Manic ran six product searches, 50 picks each per model. He ran two more sets to test the questions behind the first results. Each model saw the same twenty rows: title, store, price, star rating, review count, a summary and three features.

For coffee beans, Haiku and Sonnet picked the dark roast with the most reviews. Opus picked a medium roast 48 of 50 times. Opus called it the "safest crowd-pleaser." For a water bottle, Haiku picked a nobl Steel Bottle 45 of 50 times. Sonnet picked it 39 of 50. Opus always picked a different bottle, an Owala FreeSip. Opus called that one "the safest default." For a yoga mat, Haiku picked POPFLEX. Sonnet and Opus both picked Liforme.

Opus wrote the word "safe" or "safest" in 222 of 300 picks across the full set, Manic reports. The three models read the same list and reached three different answers from it.

Review counts the shopper never sees

The nobl bottle's catalog listing showed 447,221 reviews in the agent's view. The product page on the store shows 6,537. A wireless earbud listing read 34,394 reviews. The merchant fills in one field. The shopper on the web page reads another field. The agent only sees the first one.

The models leaned on the count. Haiku picked the bottle with the higher catalog number 45 of 50 times. A listing that overstates reviews, by typo or by test import, picks itself.

Order moved picks

Manic ran eight skincare searches twice. In the second run he moved the top product from the first row to the last row. The product's text did not change. Haiku's picks of that product dropped from 28 to 11.

Order should not matter in a scored list. Manic shows it does, and that two of the three models shift their answer when the row moves.

What a constraint does

Manic then asked for sunscreen under $25 for oily skin. The $48 product that led the open search dropped out. All three models picked an under-$25 option in 298 of 300 picks. A specific constraint overrides the default. A vague ask returns the default. The default is the model's own idea of safe.

The author's own practical reading sits at the foot of the piece. A merchant should check how a product shows in Shop MCP. A merchant should write the catalog field in the words a shopper would say. A merchant should keep every claim in the listing true. Manic writes that a listing claim is now a claim the agent makes for the buyer. The buyer picks the model. The merchant does not. One listing now has to earn three readings.

Source

Primary research: Slobodan Manic, No Hacks
Reporting: Slobodan Manic, Search Engine Journal
Shop MCP announcement: Shopify changelog

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.