In the AI Showdown, Only Half of the Models Delivered the Winning Playbook

Firmulate — Four AI Models Ran the Same Company Through Its Worst Week. Only Two Finished the Job. — Live on firmulate.com.

Imagine a high-stakes game where several AI players face the same critical moments—crises, temptations, and tough decisions. Only some make the winning move, and the difference isn’t in their chatter but in their resolve under pressure. This real-world experiment reveals what truly separates a successful AI from just a good talker.

The Crucible Experiment: Testing AI in the Hot Seat

Last July, the firmulate.com live experiment put four leading AI models through a grueling test: managing a small software company’s worst week. This wasn’t a mere chat demo; it was a full-blown business simulation involving real money mechanics, customer crises, and ethical dilemmas. The goal? See which AI could make the right decisions, avoid manipulations, and actually close a critical deal worth €55,000.

The Models and the Stakes

The models ranged from the latest GPT-5.6, scoring 95 in the Crucible League, to Fable 5, with a score of 77. All were tasked identically: to diagnose problems, resist social engineering, and sign off on the deal based solely on their own analysis. The stakes were high—every decision was versioned and auditable, mimicking real-world management challenges, including fake CEO messages and reporter tricks designed to manipulate.

What the Models Saw—and Did Not See

Remarkably, all four models identified every crisis and refused all manipulation attempts. They showed integrity and awareness, refusing to be duped. But here’s where the story gets interesting: only two managed to complete the job and sign the deal. The other two, despite correct diagnosis, left the critical contract unsigned, leaving millions on the table.

Why the Difference Matters

The key revealing detail was buried two document references deep in the company’s files—information that would have confirmed the deal’s validity. Models that read the company’s documents thoroughly won the deal at full price, worth over €4,583 in monthly recurring revenue (MRR). The models that failed to dig deep missed this opportunity, costing the company potential revenue.

Beyond Chat: Testing True Business Capabilities

This experiment underscores a vital reality: chat demo scores are misleading. Being able to generate convincing dialogue or respond quickly doesn’t automatically translate into executing complex, multi-step business decisions. The real test is whether an AI can read relevant documents, stay disciplined, and follow through—especially under pressure.

Learning from the Live Company

The live simulation involved a company with 13 synthetic employees managing €105,000 in monthly burn against only €2,300 in MRR. It operates with over 680 self-learned rules and versioned daily to adapt to new challenges. This setup demonstrates the importance of discipline and thoroughness, qualities that only two models demonstrated convincingly during the experiment.

AI in Strategy and Decision-Making for Small Business Owners: Affordable AI Tools to Evaluate Ideas, Model Outcomes, and Set Priorities (AI Productivity for Small Business Owners Book 10)

As an affiliate, we earn on qualifying purchases.

The Takeaway for Business Leaders

This experiment isn’t just about AI prowess in a lab. It’s a mirror for real-world enterprise: if your AI support or decision systems do not demonstrate the ability to finish what they start—reading relevant documents, resisting manipulation, closing deals—they may look good but fail when it counts. The true strength of an AI isn’t just its ability to identify problems; it’s its capacity to execute solutions reliably under pressure.

To see the experiment in action, visit firmulate.com/live. You can also explore plain-language results, try a quiz, or run your own business wargame against a read-only export at firmulate.com.