Your AI-Built Ops Tool Passed Its Own Tests. Can Staff Trust It?

Your team has built a useful internal tool with AI. It works in the demo. The tests pass. Someone wants to send the link to everyone on Monday.

The missing question is not whether AI wrote good code. It is whether the tool keeps the promises your business needs it to keep when something goes wrong.

A purchase request can look approved without reaching the finance system. A retry can create two requests. A helpful change can remove an approval rule that made testing inconvenient. None of those failures has to appear in the happy-path demo.

Better evidence is useful. It is not the acceptance decision.

In its September 11 Cognition example, OpenAI describes Devin testing an iPhone game and returning a simulator recording alongside passed checks and areas left untested. That is useful evidence: reviewers can inspect behaviour and see the limits of the test. It is a vendor demonstration, not proof that an unrelated business workflow is ready for staff.

Dan Luu's agent-testing study illustrates a different limit. In its Rust/Zstd evaluations, naming a testing technique often led to superficial use of that technique rather than checks that caught the important failures. The study does not establish that every AI-written test is weak. It does show why the label on a test strategy is not enough.

A recording proves what happened in that run. A passing test proves that a particular check passed. Neither decides whether the right business rule was checked in the first place.

Write the business rule before reviewing the result

Consider a fictional purchase-approval tool. Staff submit requests, a manager approves larger purchases, and an approved request creates a record in finance. This is an illustrative acceptance exercise, not a Tessera client case.

The ops owner defines the rule: requests up to and including $500 may proceed without manager approval; anything above $500 needs approval from an authorised manager. Requesters cannot approve their own above-limit requests. One request must produce at most one finance record.

Those expectations belong to the business. The builder can implement and automate the checks, but cannot quietly change the limit, permit self-approval or redefine success to make the tests pass. A policy change is a separate decision.

Keep a short acceptance sheet with the starting state, action, expected business result and evidence to inspect. Record actual results separately. This makes disagreement visible instead of burying it inside a green test report.

Four scenarios worth checking before Monday

1. An ordinary request reaches the real destination

Submit a $200 request as an ordinary staff user. The tool should create one finance record containing the correct amount, requester and request identifier. The interface should show the corresponding status.

Inspect the destination record, not just the success message. Start in a sandbox where available, then verify the actual integration path with a controlled, authorised record before wider release. A mock response cannot establish that the live connection works.

2. The approval boundary survives an inconvenient case

Check $500 and $500.01, then try an $800 request. Under this example's rule, the latter two must wait for an authorised manager. A requester attempting self-approval must not advance them.

After the manager approves, check that exactly one finance record appears and that the approval history names the approver. Also decide what happens if someone changes the amount after approval. An approval for $800 must not silently authorise a later $1,500 request.

The important evidence is the enforced boundary, not whether the approval button looks disabled. Have the technical reviewer check that an alternate route cannot bypass the same rule.

3. A second click does not become a second purchase

Submit the same request twice, including a retry after a slow response. The expected outcome is one request with one downstream record, not two purchases and a warning to be more careful next time.

Agree how the tool distinguishes a retry from a genuinely new purchase. Two legitimate requests can have identical amounts and descriptions. Blocking every similar-looking entry solves one problem by creating another.

4. A failed update stays visible and recoverable

Make the destination temporarily unavailable in the test environment. The request must not claim completion. It should retain the approved details, show an actionable pending or failed state, and identify who handles recovery.

Then test the awkward case: finance accepted the record but the response never reached the tool. Before retrying, the system must establish whether the record already exists. Otherwise the recovery action can create the duplicate that the normal path avoided.

Restore the connection and follow the recovery procedure. Verify one destination record and a consistent final status. A written recovery plan that nobody has exercised is still an assumption.

Use the results to choose pilot, repair or release

These four scenarios are a starting point for this workflow, not a universal certification. Access controls, sensitive data, audit history and any other consequential business rules still need appropriate review.

• Keep it in a controlled pilot when critical behaviour is untested. Limit users and permitted actions, and keep the established process authoritative. Do not ask staff to enter the same live purchase independently in both systems.
• Repair the bounded gap when the failure is understood. Fix the duplicate handling or approval check, then rerun the affected scenarios and any connected rules. One failed check does not automatically justify a rewrite.
• Release to staff when the required scenarios pass, remaining limitations are explicitly accepted, and someone owns operation and recovery. Record who made that decision and which version they accepted.

Agree the stop condition too: for example, an unexplained duplicate or approval bypass pauses new submissions. Decide who can pause the tool, how pending requests are reconciled, and how staff return to the previous process without losing or repeating work.

Independent judgement does not mean manual testing forever

The business owner does not need to inspect every line of code. They need to define acceptable outcomes and review evidence with someone capable of checking the implementation and integrations. That reviewer should be able to challenge both the software and the tests.

Once the rules are settled, automate the repeatable checks. Keep the acceptance expectations outside the builder's freedom to redefine them. Our control-plane guide covers the broader operating boundaries; this exercise makes one specific release decision concrete.

The goal is not to distrust the person who built the tool. It is to make its limits visible before the rest of the team depends on it.