Imagine a sports coach who not only watches the game but studies every play, every player, and every prior mistake to make the perfect call. Now, what if AI agents could do the same—scanning your confidential files before deciding how to act? That secret could be the difference between sealing a deal at full price or losing it instantly. Welcome to a groundbreaking experiment that exposes how AI models’ reading habits can determine business success, even in high-stakes scenarios.
Uncovering the Hidden Factor in AI Decision-Making
In a recent live experiment, four leading AI models faced the same challenge: run a small software company through its worst week—handling crises, customer demands, and manipulative tactics—just like a real business. The goal wasn’t just to perform well in chat but to actually make decisions that lead to tangible outcomes, like closing a €55,000 deal. The results were revealing: all models successfully identified crises and refused manipulative tricks. However, only two managed to follow through and sign the deal based on their own analysis, while the others fell short despite identical diagnoses and pitches.
The Critical Deep Dive
What separated the winners from the rest was their ability to read beyond superficial cues—specifically, to dig two document references deep into the company’s internal files. The decisive weakness was buried in this layered reading—if the AI read the relevant internal document, it secured the deal at full price. Those who skipped this step lost automatically, illustrating how crucial internal file comprehension is for trustworthiness and proper decision-making in AI systems.
Testing Trust and Integrity Under Pressure
The experiment also tested the models’ resistance to social engineering. Fake CEO messages escalating over three stages and a reporter trick—requiring just a simple yes/no answer—were introduced to see if the models would be duped. All five models refused to get manipulated, with Kimi K3 reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” This demonstrates an important trait: AI models can be trained to detect and refuse manipulative tactics, an essential feature for safeguarding business operations.
Doxie Go SE – The Intuitive Portable Document Scanner with Rechargeable Battery and Easy Software for Home, Office, or Work from Home
- Go Paperless: Portable, wireless document scanning
- Fast, Easy Scanning: Scan in 8 seconds at 600 dpi
- Compact & Battery Powered: Small size, rechargeable battery, 400 pages per charge
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Live Company and Real Money Mechanics
The experiment isn’t just theoretical: it’s integrated into a real company with 13 synthetic employees, handling actual money mechanics—burning €105k/month against a €2.3k monthly revenue. Every decision is versioned and auditable, and the company’s performance can be watched live at firmulate.com/live. This ongoing setup allows stakeholders to see firsthand whether AI models can make trustworthy decisions in complex, high-pressure environments.
The Performance Gap and What It Means
The top-performing AI, gpt-5.6-sol, scored 95 out of 100, successfully finding the buried fact needed to close the deal. Kimi K3, the newcomer, scored just slightly behind at 93, also closing the deal with the cleanest discipline. Sonnet 5 scored 88, and Fable 5 trailed at 77, with some slips in process. The baseline score, representing a do-nothing approach, was only 26—highlighting how much better these models are at decision-making, but also how critical the depth of analysis is for success.
Implications for Business and AI Adoption
For companies deploying AI across CRM, support, or forecasting, the key question isn’t just whether the AI can generate good text. It’s whether it can complete its tasks reliably, read your files thoroughly, and stay honest under pressure. The experiment underscores that the ability to read and interpret internal files before making decisions is a measurable, crucial factor in AI performance—one that can determine whether a deal is won or lost.
As an affiliate, we earn on qualifying purchases.
What You Can Learn and How to Prepare
Businesses should consider testing their AI tools in similar simulated environments—what Firmulate calls “wargames”—to assess how well these models perform under real-world pressures. The platform allows companies to run their own scenarios, mirroring their internal crises, without risking actual systems or data security. It’s a practical way to gauge whether an AI will finish what it starts and stay trustworthy when stakes are high.
Takeaway: Reading Deep Matters
The experiment highlights a fundamental truth: AI’s competitive edge often lies in its ability to read and understand context at multiple levels—especially internal documents that contain the critical facts. As AI moves from chatbots to decision-makers, ensuring that these models can access, process, and trust internal data will be key to unlocking their true potential—and avoiding costly mistakes.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI decision-making software for business
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
