An eight-member team led by Yijun Pan from Alibaba Group’s Accio division, working in collaboration with Yale University, built a simulated marketplace and assigned fifteen leading language models the roles of store owners. Each model received starting capital of $80,000 and thirty simulated days to grow it. Average final wealth ranged from just under $21,000 for MiniMax M2.5 to more than $188,000 for Gemini 3.1 Pro, representing an almost ninefold difference. More than half of all simulations ended in a loss, and only four models managed to preserve their starting capital in every single trial. The authors published the study on the arXiv platform.
Why test AI specifically on commerce
Previous tests of autonomous agents mainly measured whether a model could fix a bug in code or successfully navigate a web application. In such cases, there is always a single correct answer, and the surrounding environment does not change while the agent is working. Real-world business, however, behaves differently. The authors identified four characteristics that make it a tough challenge: market signals are imprecise and partly contradictory, merchants must invest money before they know the outcome, demand and costs change unpredictably, and obligations to authorities and customers persist regardless of how successful the business is.
Testing in real-world operations was therefore not possible. A model error would mean losing real money, confusing customers, or violating regulations. The main problem, however, would be that two different assignments could never be compared under exactly the same real-world conditions.
A marketplace built on real Alibaba data
The supplier catalog is based on real listings from the Alibaba.com platform. The arena contains 965 tradable offers from 831 anonymous suppliers and includes 135 types of goods. Each offer retains its country of origin, unit price, minimum order quantity, stock level, delivery time, and stated quality. The authors obtained tariff rates from the World Bank database, while seasonal demand was simulated using retail statistics from the U.S. Census Bureau, Eurostat, and China’s national statistics office. The thirty-day calendar in the simulation represents one real year, allowing it to include China’s 618 shopping festival and Singles’ Day, Vietnam’s Tet, the Brazilian Carnival, the Super Bowl, Halloween, Black Friday, Christmas, and Ramadan with Eid al-Fitr.
The competitive environment consists of sixty scripted sellers divided into ten behavioral types—from price leaders who remain below the market median and discount sellers to merchants who raise prices by half during peak shopping periods.
The agent has access to more than sixty tools covering the entire business cycle, as well as its own working directory where it can write scripts and notes. It decides when to end the current day. The simulated world then advances by exactly one day: customers make purchases, competitors adjust their prices, shipments move toward their destinations, and ongoing costs are deducted.
Models’ trading results
The strongest models more than doubled their capital, the middle group earned little, and the weakest ended with less money than they had received at the start. However, even the winner could not match a human-designed strategy. The best of the manually constructed approaches, which continuously recalculates demand estimates and uses them to coordinate pricing, advertising, inventory replenishment, and capital allocation, brought final wealth to more than $436,000, twice the average achieved by the best model.
More interesting than the ranking itself is how the models ran their businesses. Gemini 3.1 Pro behaved like a premium seller. It maintained an average margin per order above fifty percent and accepted selling a smaller share of its inventory. Its own pricing program calculated costs separately for each shipping route and undercut competitors only when doing so left it with a margin of at least fifteen percent. By contrast, the GPT-5.6 Sol and Opus 4.6 models focused on sales volume, slightly sacrificed margin, and sold more than 93 percent of their goods. Fable 5 emerged from the evaluation as the most balanced seller, combining a healthy margin with the ability to capture actual demand effectively.
The weaker models failed in two opposite ways. Qwen 3.7 Max spent only 22.3 percent of its starting capital and missed most opportunities. MiniMax M3 purchased more, but with an inventory turnover of 0.36, its capital became tied up in goods that were not selling. MiniMax M2.5 finished with a margin of 12.1 percent and sold just under 40 percent of its inventory. In one log, its pricing script assigned each product a single price for all markets, even though the model itself had recognized that costs differed by route. The result was disastrous: 98 of 142 orders fell below the actual purchasing and transportation costs.
Penalties and problems
The test took regulatory obligations seriously. The agent must first determine which permits it needs for a given market, apply for them sufficiently far in advance, and wait until it receives them before entering the market. For every sale made without a permit, the system imposes a fine that rises to as much as $2,500 for repeated violations. MiniMax M2.5 accumulated an average of 22.7 violations and fines exceeding $51,000, while DeepSeek V4 Pro recorded 17.4 violations and nearly $40,000 in fines. By contrast, Fable 5 and GPT-5.5 did not incur a single fine.
Customer support reshuffled the rankings even more significantly. Opus 4.8 converted 84 percent of inquiries into orders, Fable 5 succeeded in 81 percent of cases, and Qwen 3.8 reached 74 percent. The winning Gemini 3.1 Pro, however, managed only 57 percent, leaving a large share of inquiries completely unanswered or responding to them inadequately.
One log from the GPT-5.5 model. On the nineteenth day, it found itself with no cash and approximately $90,000 tied up in inventory, while two of its products attracted minimal interest. The model immediately stopped its existing script, liquidated the unsellable inventory, and wrote a new program. In it, the model restored prices with a safe margin, passed shipping and customs costs on to buyers, and disabled advertising. According to the authors, what matters is not that the model identified the unsellable inventory, but that it was able to translate its assessment immediately into actions that restored the business to working order.
However, the authors themselves acknowledge the test’s limitations. It measures only how an agent decides to act, not whether it can operate a real business, send offers through external platforms, or communicate through an actual email inbox. Moreover, the entire benchmark is limited to international business-to-business trade (B2B), while real-world business encompasses a much broader range of industries.
