The AI Shopping Agent Paper, Explained

AI tools such as Agent mode in OpenAI's Atlas browser and Google's Project Mariner can open a store page, compare the products on it, and add one to the cart for the buyer (page 2, Introduction). When an agent makes the choice, the buyer may never look at the other products on the page, so a product the agent passes over loses the sale without a person ever considering it. A team from Columbia Business School, Yale University, and the AI company MyCustomAI built a test store to find out which products these agents choose and what changes their choices.

Our researchers read What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commerce, a paper by Amine Allouah and Josué D. Figueroa of MyCustomAI, Omar Besbes and Yash Kanoria of Columbia Business School, and Akshit Kumar of Yale University, and wrote this report to explain its findings in simple terms and what they mean for brands and sellers. The paper was first posted to arXiv on August 4, 2025, and we read the version posted on December 17, 2025. The authors ran thousands of randomized shopping trips with AI agents from Anthropic, OpenAI, and Google, changing product positions, prices, ratings, review counts, and labels such as "Sponsored" to see how each change moved the agent's pick (pages 1 to 3, Abstract and Introduction).

In the following sections, we explain how the experiment worked, what the authors concluded, how well the agents handled simple choices, which products they bought, how much a product's spot on the page mattered, how the agents treated sponsored and recommended products, how they weighed price against ratings and reviews, what a rewritten product title did, and the limits the authors point out themselves.

The Main Experiment

The authors asked four questions. Can AI agents follow simple instructions and make obviously correct choices? Which products do they buy when the decision is left to them? How do they respond to a product's position on the page and to labels the store adds? Can a seller win more sales by rewriting a product title (pages 2 and 3, Introduction)?

The test store

The authors call their test system ACES, short for Agentic e-CommercE Simulator. It has two parts. One is a mock store that shows a search results page of eight products in a grid of two rows and four columns, with each product's title, price, star rating, number of reviews, and any labels visible on the page. The other is an AI agent that opens the store in a web browser, takes a screenshot of the page, and picks one product to buy. There are no product detail pages, so everything the agent uses to decide is in that one screenshot (pages 10 to 12, Section 2).

The store carried eight product categories: fitness watches, iPhone 16 Pro covers, mousepads, office lamps, staplers, toilet paper, toothpaste, and washing machines. Each category held eight real products the authors found on Amazon, mixing well-known and lesser-known brands (page 10, Section 2).

In most tests the agent received the same general request, which asked it to act as a personal shopping assistant, pick a good product in the category for a typical buyer, and explain the choice. The agents ran at their lowest practical reasoning settings, since a real store has to answer quickly and cheaply (pages 11 and 12, Section 2 and Figure 4).

The six AI models

The main tests used the newest model from each company in August 2025 and the successor that was newest on December 1, 2025. Some of the simple-choice tests also included older models, such as GPT-4o and Claude Sonnet 3.5 (page 3, Introduction; page 43, Appendix B).

What they changed and measured

• Market share. For each category, the authors ran 1,000 shopping trips with the eight products shuffled into random spots and counted how often each product was bought. They treat that share as the product's market share in a world where agents do the buying (page 13, Section 4.1).
• Position and labels. In a second set of 500 trips per category, they shuffled positions again and also randomly added "Sponsored," "Overall Pick," and "Only X Remaining" labels and randomly raised or lowered prices, ratings, and review counts. Because each change was random, a statistical model could estimate how much each one moved the agent's choice by itself (pages 15 to 17, Section 5.1 and Table 1).
• Seller rewrite. A separate AI agent, acting for the seller, rewrote one product's title in each category using sales results from 200 earlier trips, and the authors reran the same 200 trips to measure the change (pages 21 and 22, Section 6.1).

What the Paper Concluded

The authors concluded that AI shopping agents now handle simple comparisons well, while their buying is shaped by strong preferences that differ from one model to the next and shift when a model is upgraded. In their words, "agentic markets are volatile and fundamentally different from human-centric commerce" (page 1, Abstract).

They list five main results. An agent's preference for certain spots on the page is large and depends on the model. An "Overall Pick" label raises a product's chance of being picked by a wide margin, while a "Sponsored" label lowers it. Agents weigh price, ratings, and reviews in sensible directions but by very different amounts. These patterns hold when the agent reads a plain product list, when its instructions change, and across model generations. Small edits to a product's title can shift sales, and a model update can move demand even when nothing in the store changes (pages 29 and 30, Conclusion).

The authors call for regular auditing of shopping agents and suggest that model developers publish profiles of how their agents choose, including position preferences and responses to labels, so stores and sellers can compare models and releases (pages 30 and 31, Conclusion).

Can the Agents Make Obvious Choices?

Before studying preferences, the authors checked whether agents could find the right answer when only one existed. Instructions such as staying under a budget or choosing a stated color were followed almost every time, and the worst result was Claude Sonnet 3.5 missing the requested color 13.5% of the time (page 43, Table 7). The harder tests set out identical products except for one small difference. In one, a single product was 1% cheaper. In another, a single product had a star rating 0.1 higher. A rational buyer picks that product every time (pages 12 and 13, Section 3).

Re-charted from Allouah et al. Table 9.

• Older models often missed small differences. When one product was rated 0.1 higher, GPT-4o picked a different product 71.7% of the time and Claude Sonnet 3.5 did so 57.3% of the time. Claude Sonnet 4 and GPT-4.1, the newest models in August 2025, missed it 28.7% and 15.1% of the time (page 44, Table 9).
• The December 2025 models almost never failed. Claude Opus 4.5 and GPT-5.1 had no misses on the rating test, and Gemini 3 Pro Preview missed 0.3% of the time. On the 1% price cut, Claude Sonnet 3.5 missed the cheaper product 63.7% of the time, compared with 1.0% for GPT-5.1 and none for Gemini 3 Pro Preview (pages 43 and 44, Tables 8 and 9).
• When agents did fail, their explanations included choosing the "standard" price as the safer option, treating the lower price as a display or pricing error, and calling the difference too small to matter (page 44, Appendix B.1.2).
• By December 2025, mistakes on single-number comparisons had mostly disappeared. The rest of the paper studies choices where no single right answer exists.

Which Products the Agents Bought

Across 1,000 shuffled trips per category, each model settled on a clear set of favorites, and the favorites differed by model. The chart shows the fitness watch category, where two Fitbit watches took most of the sales (page 5, Figure 1).

Re-charted from Allouah et al. Figure 1.

• Claude Sonnet 4 picked the Fitbit Inspire 45.2% of the time, while GPT-4.1 and Gemini 2.5 Flash picked it 24.4% and 25.5% of the time. After the upgrade to Claude Opus 4.5, the Inspire's share rose to 77.4%. After the upgrade to GPT-5.1, it fell to 5.9%, and the Fitbit Versa took 88.9% (page 5, Figure 1).
• The WHOOP watch was never picked by any of the six models, and a lesser-known watch the paper lists as Watch2 reached at most 0.1% (page 5, Figure 1).
• Staplers were even more concentrated. Amazon Basics took between 77.0% and 94.2% of sales across the six models, and the Arrow stapler was never picked (page 5, Figure 1).
• The same swings appeared in other categories. GPT-4.1 picked the Mikeke iPhone 16 Pro cover 62.6% of the time, while GPT-5.1 picked the ESR cover 95% of the time and Mikeke 5% of the time (page 14, Section 4.2).
• The authors describe a model update as a demand shock for sellers. A product can be the favorite of one model version and lose that place in the next, with no change to the product or the store (page 14, Section 4.2).

How Much a Product's Spot on the Page Mattered

To separate the effect of position from everything else, the authors fit a statistical model called a conditional logit to the randomized trips. It estimates how much each feature of a listing raises or lowers the chance it is picked while holding the other features constant. The chart applies those estimates to eight identical products and shows the chance each spot is picked (pages 16 to 20, Section 5.2, Tables 2 and 3, and Figure 7).

Re-charted from Allouah et al. Figure 7.

• Every August 2025 model favored the top row, and each preferred different columns. GPT-4.1 favored the top left, where an identical product was picked 37.4% of the time. Claude Sonnet 4 favored the second and third spots of the top row, and Gemini 2.5 Flash leaned toward the third and fourth columns (page 17, Section 5.2; page 20, Figure 7).
• For Claude Sonnet 4, moving a product from the bottom right corner, where it was picked 4.5% of the time, to the second or third spot of the top row raised its chance about five times (page 18, Section 5.2).
• Upgrades changed the pattern. GPT-5.1 preferred the bottom row, with the bottom right corner at 21.0%, nearly the reverse of GPT-4.1. Gemini 3 Pro Preview's top-row spots ranged from 17.0% to 26.8%, and its bottom-row spots ranged from 2.0% to 3.1% (page 20, Section 5.3 and Figure 7).
• Position still mattered when the agent read a plain ranked list of products with no screenshot. In that test, Claude Sonnet 4 picked the first listed product 30.2% of the time and the last 4.1% of the time, about a sevenfold gap (pages 27 and 28, Section 7.2.2 and Figure 11).
• Telling the agent to ignore position did not remove the effect. For Claude Sonnet 4 and GPT-4.1, the position estimates did not change in a statistically significant way (page 28, Section 7.3.1).

How the Agents Treated Sponsored and Recommended Products

The store randomly placed a "Sponsored" label on one to four products, an "Overall Pick" label on one product, and an "Only X Remaining" label on another. The chart shows what the first two labels did to a product that would otherwise be picked 10% of the time, for the three August 2025 models, on the store page and in the plain product list (page 15, Table 1; page 18, Section 5.2; page 27, Section 7.2.2).

Re-charted from Allouah et al. Sections 5.2 and 7.2.2.

• On the store page, a "Sponsored" label lowered a 10% product to between 7.9% and 8.9%. An "Overall Pick" label raised it to 24.3% with Claude Sonnet 4, 19.9% with GPT-4.1, and 42.6% with Gemini 2.5 Flash (page 18, Section 5.2).
• Both effects were larger in the plain product list. "Sponsored" dropped the product to 1.8% with GPT-4.1, and "Overall Pick" lifted it to 72.7% with Gemini 2.5 Flash (page 27, Section 7.2.2).
• For the August 2025 models, the "Only X Remaining" label had a weak negative effect or one too small to tell apart from zero (page 18, Section 5.2).
• The December 2025 models showed the same pattern, with a penalty for "Sponsored" and a large reward for "Overall Pick" (page 19, Table 3; page 20, Section 5.3).
• The authors point out that a paid ad still buys a seller a better spot on the page. Their result means that at the same spot, the label itself costs the product some of its chance of being picked (page 18, Section 5.2).

How the Agents Weighed Price, Ratings, and Reviews

All six models preferred cheaper products, higher ratings, and more reviews, though by different amounts. The authors converted each effect into a price change. A positive number is how much a seller could raise the price after gaining that feature and keep the same chance of being picked. A negative number is how much the seller would need to cut the price to make up for it (pages 18 and 19, Section 5.2; page 47, Tables 10 and 11).

• For GPT-4.1, a rating 0.1 higher was worth a 67.3% price increase, nearly twice its value to Claude Sonnet 4 and more than twice its value to Gemini 2.5 Flash. Claude Opus 4.5 valued the same change at 80.6% (page 47, Tables 10 and 11).
• GPT-5.1 was the one model for which a top-row spot had negative value. A seller moved to the top row would need to cut the price 22.2% to keep the same chance of being picked, which matches its preference for the bottom row (page 47, Table 11).
• The authors estimate price elasticity, which measures how strongly demand falls as price rises, at between −1.6 and −2.2 for the August 2025 models. They note this is close to estimates for human shoppers choosing among brands and products (page 18, Section 5.2).
• When the instructions told the agent on the store page to ignore position and prioritize price, a 10% price cut raised a product's odds of being picked by about 81% for Claude Sonnet 4, 165% for GPT-4.1, and 101% for Gemini 2.5 Flash. The "Overall Pick" effect fell to near zero for Claude Sonnet 4 and GPT-4.1, while position effects stayed large enough to matter (pages 28 and 29, Section 7.3.1).

What a Rewritten Product Title Did

In the last experiment, a seller-side AI agent received a screenshot of the store, its own product's details, and the sales results of all eight products across 200 trips. It was asked to suggest a better product title using only real product features. GPT-4.1 wrote the new titles for the August 2025 buying models, and GPT-5.1 wrote them for the December 2025 models. Only one product per category changed, and the authors reran the same 200 shuffled trips (pages 21 and 22, Section 6.1 and Figure 8). In the chart, the line through each bar shows one standard error.

Re-charted from Allouah et al. Table 4.

• Five of the six buying models gave the rewritten product a statistically significant average gain, from 3.66 percentage points with Claude Sonnet 4 to 14.89 points with GPT-5.1. With Gemini 3 Pro Preview, the change was 0.32 points, which was not significant (page 22, Table 4).
• Most single rewrites made no measurable difference. The change was significant in 33% of the category and model pairs, and in those pairs the gains were large (page 22, Section 6.2).
• The office lamp produced the biggest gains, up to 80.4 points with GPT-5.1. Its original title began "Floor Lamps for Living Room," and the word "Office" was cut off on the store page, so agents looking for an office lamp passed it over. The seller agent moved "Office" to the front (pages 23 and 24, Section 6.2 and Figure 9).
• Other gains came from adding missing details, such as "2-Ply," "Septic-Safe," and "Unscented" for toilet paper (15.5 points with GPT-5.1) and "Fitness" for a watch first listed as a GPS running watch (13.0 points with Claude Opus 4.5). The rewritten titles were not significantly shorter than the originals (page 24, Section 6.3).
• Some rewrites backfired. The mousepad rewrite lost 12.5 points with Gemini 3 Pro Preview, and the stapler rewrite lost 3.0 points with Claude Opus 4.5 (page 23, Figure 9).
• The authors suggest sellers could show different titles depending on what the agent is shopping for, presenting the same lamp as an office lamp to one agent and a living room lamp to another. They call this query-conditional SEO (pages 24 and 25, Section 6.3).

Limits the Authors Note

• Each trip covered one choice from a single results page, with no product detail pages and no browsing between pages (page 12, Section 2).
• The agent received a general request with no personal preferences, so the results describe each model's default behavior before any personalization (page 9, Related Literature).
• The study did not compare agents with human shoppers. The authors note that some patterns resemble human behavior, while the large gaps between models suggest that most models differ substantially from people (page 30, Conclusion).
• In the seller test, only one seller per category rewrote its listing, and it made a single attempt (page 22, Section 6.1).
• The plain-list test left out product images, which the authors describe as a deliberately simplified stress test (page 26, Section 7.2.1).
• The study describes what agents do without explaining why. The authors point to post-training as a likely factor, since the move from the preview release of Gemini 2.5 Flash to its final release flipped its preference from the bottom row to the top row (page 6, Introduction; page 30, Conclusion).

What This Means for Brands in 2027 and Beyond

For a brand that sells online, this paper shows that an AI agent's purchases depend heavily on which model does the shopping. The same store page produced very different sales from one agent to the next, and a single upgrade moved a Fitbit watch from 24.4% of GPT-4.1's picks to 5.9% of GPT-5.1's. The store's own labels carried large weight as well. In the authors' price terms, an "Overall Pick" label was worth a price increase of between 61.5% and 168.9% across the six models.

The authors write that "the scope for Generative Engine Optimization (GEO) is substantial and that sellers must continuously adapt product listings to remain attractive to AI agents" (pages 30 and 31, Conclusion). Their seller test supports that view. A rewrite that put the right word first raised a lamp's share by as much as 80.4 points, while other rewrites changed nothing or lost ground.

So if you sell through stores that AI agents shop, test your product titles with the agents your buyers use, put the words a buyer would search for at the start, and test again after each major model release.