Our Take
A 69-person internal pilot with $100 budgets proves nothing about real market viability, but the unnoticed agent quality gaps matter.
Why it matters
If users can't tell when their AI agent performs poorly in negotiations, early adopters of agent commerce could lose money without knowing why.
Do this week
Procurement teams: audit any agent-mediated purchases monthly to detect performance gaps before deploying at scale.
Anthropic ran four agent marketplaces with real money
Anthropic created Project Deal, a classified marketplace where AI agents represented both buyers and sellers in actual transactions. The pilot involved 69 company employees, each given $100 in gift card budgets to trade real goods with coworkers.
Agents completed 186 deals totaling over $4,000 in value (company-reported). Anthropic ran four separate marketplace versions: one "real" environment where deals were honored post-experiment, plus three additional variants for comparative study.
The key finding: users represented by more advanced models achieved "objectively better outcomes" than those with weaker agents, but users couldn't detect this performance gap. Initial instructions given to agents had no measurable impact on sale probability or negotiated prices.
Invisible agent gaps create unknown losses
The unnoticed performance disparity reveals a critical blind spot in agent-mediated commerce. When AI agents negotiate on behalf of humans, the human principals lack visibility into whether their agent performed competently.
This creates an information asymmetry problem: if one party deploys a more capable negotiation agent, the other party suffers worse outcomes without realizing it. Unlike human negotiations where poor performance is often apparent, agent quality differences remain hidden from end users.
The small scale and controlled environment limit broader conclusions. A 69-person internal test with $100 budgets among coworkers differs substantially from open market conditions with higher stakes and unknown counterparties.
Audit agent performance before scaling
Organizations considering agent-mediated purchasing should establish performance monitoring before deployment. The Anthropic experiment shows users cannot reliably detect when their agents underperform, making objective measurement essential.
For procurement teams, this means tracking outcomes across different agent configurations and comparing results to human-negotiated baselines. Finance teams should consider the legal implications: if an agent makes a poor deal, determining liability becomes complex when the human principal couldn't reasonably detect the agent's inadequate performance.
The experiment's scope limits its practical applicability, but the core insight about invisible performance gaps applies broadly to any automated negotiation system.