Visa has published the first round of results from its Agentic Commerce Benchmark, and the findings are a sobering reality check for anyone ready to hand their credit card to an AI assistant. Across 22 models drawn from 12 different labs, the agents failed 42.8% of test runs — a pass rate of just 57.2% on tasks that simulate real-world shopping decisions.
- Wide performance spread: The strongest model passed 82.2% of attempts; the weakest managed only 28.9%.
- Consistency is the real problem: Each model got three attempts per scenario, and all three passed in only 34.2% of cases.
- Failures still cost money: In 54% of failed runs, the AI went ahead and made a purchase anyway.
How the benchmark works
The test is built around a deliberately messy premise: shoppers rarely know exactly what they want. A customer might say they need a television but omit their budget, the formats they care about, or whether they want HDR10+ for family videos and gaming features. The AI must extract those hidden requirements through conversation, then decide whether to buy — or to walk away if nothing fits.
To block lucky guesses, a model only passes if it uncovers enough information to justify its decision. The first round covered 15 shopping challenges, including a TV, a laptop and a personal care appliance. Visa says the benchmark is publicly available and consumers can attempt it themselves.
The data reveals a clear pattern: models that asked better questions performed better. The five models most effective at surfacing hidden buying factors averaged a 70.2% pass rate, while the five weakest in that skill passed just 44.4% of the time. Price offered only partial protection — the five most expensive models per run passed 67.1%, against 50.2% for the five cheapest.
The failure mode that matters
The headline number is the 42.8% failure rate, but the more troubling statistic is what happened after those failures. In more than half of them, the agent completed a purchase that either ignored the customer’s stated needs or was made before those needs were fully understood.
“AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, Global Head of Growth at Visa. “Visa Agentic Commerce Benchmark is yet another way Visa is working to boost standards across the industry.”
Birwadker also acknowledged to Fortune that the industry is not yet where it needs to be: “We’re not there yet from a trust or technology perspective. But we’ll soon get into a world where consumers can trust and delegate.”
That gap between capability and caution is the core design flaw. A human shopper who is unsure tends to hold back; these agents are biased toward closing the transaction. Until a failed run reliably means “I won’t buy” rather than “I’ll buy something that doesn’t fit,” delegating a card to an AI remains an act of faith.
Infrastructure is racing ahead of reliability
The benchmark lands as payment networks and platforms build out the plumbing for agent-driven purchases. Mastercard has been issuing virtual cards designed for AI agents, and Google has been rolling out shopping features that let users buy across multiple stores from Gemini. The rails for agentic commerce are being laid faster than the agents themselves can be trusted to use them.
Adoption data explains the urgency. Visa’s Trust Index, released in September 2026, found that 72% of U.S. consumers had used an AI assistant, but only 23% trusted AI to handle payment transactions on their behalf. Visa has already integrated its payment capabilities into ChatGPT and joined Mastercard and Ant International on identity checks for agents.
Visa says the results do not mean AI agents are unfit for commerce — the top models each cleared more than eight in ten attempts. Rather, the company frames the benchmark as a roadmap. It plans to add more scenarios and product categories, and to extend testing into checkout, payment and order placement. The benchmark has been shared with leading model providers for feedback.
The significance of this study lies partly in its author. Visa profits from every transaction, so it has every incentive to champion agentic commerce. When a company with that alignment says the technology is not ready, the warning carries more weight than any external critique. The 42.8% failure rate is a hurdle the industry can engineer past; the 54% of failed runs that ended in a purchase is a design problem that must be fixed before consumers will ever feel comfortable letting an AI spend their money.
Sources & References