For weeks every status light was green. Then I opened the folder and found nineteen fresh files on one machine, three fossils from May on the other—and the fossils were winning.
That failure changed how I test AI agents. The short version is this: test the outcome the agent was supposed to produce, not the activity it performed along the way. A running process, a completed tool call, or a 200 response can all be real while the system is still doing the wrong thing.
Here is the failure that taught me the difference, followed by the five checks I now use before I trust an agent in production.
The stale machine won
I keep one AI agent's memory synced across two machines—a MacBook I work on and a Mac mini that runs the same agent in the background. Same brain, two bodies. A daemon on each machine is supposed to watch the memory folder and push changes to the cloud every thirty seconds, so whichever machine I touch, the other catches up.
The sync infrastructure looked healthy from the process view. The mini's daemon was running. Its last push was seconds ago. The cloud was accepting its updates. If you had asked me whether sync was working, I would have said yes.
Then I compared the files.
The MacBook had nineteen memory files, the newest written the previous day. The Mac mini had three, the newest from May 30. It was two months stale.
The part that took me an embarrassingly long minute to understand was that the mini was winning. Its daemon was running and dutifully pushing those stale files to the cloud every thirty seconds. The MacBook—the machine with the current work—had a sync daemon that was not running at all.
The machine with nothing to say was talking constantly. The machine with everything to say was silent. The cloud believed the mini.
Every check I had looked at was reporting a true fact. The process was alive. The request succeeded. The push was recent. None of those facts answered the question that mattered: was the newest state moving in the right direction?
Why AI agent tests pass while the system fails
“Is the daemon running?” is a real question with a real answer. It is also the wrong test for data convergence.
A process can happily do the wrong thing forever. An AI agent can call every expected tool, produce a plausible answer, and mark its task complete without creating the required result. From the process's point of view, that is success: it ran, made calls, and received responses. From the user's point of view, the work failed.
I call this a check that can't fail: a check that is structurally incapable of returning bad news about the outcome you care about.
These checks are more dangerous than obvious failures because they buy the wrong conclusion. A red light gets investigated. A green light that cannot turn red sits on the dashboard and makes you more confident while the real state quietly drifts.
How to test AI agents in production
The exact assertions will differ by product, but the method is consistent.
1. Define the outcome before the workflow
Write down the state that must be true when the agent finishes. “The agent ran” is not an outcome. “The approved invoice exists in the ledger with the correct amount and source document” is.
If you cannot state the expected result independently of the agent's steps, you cannot build a useful test for it.
2. Make the check prove that it can fail
Ask what realistic failure would turn the test red. Then create that failure in a controlled environment.
Stop a worker. Give the agent stale input. Deny a permission. Return a successful API response without the expected state change. If the test stays green, it is measuring activity rather than correctness.
3. Separate liveness from correctness
Liveness checks still matter. You should know whether a worker is running and whether an API responds. Just do not confuse those signals with proof that the work is correct.
For the sync system, the liveness check was “did the daemon push?” The correctness check was “does the newest file on the source match the newest file in the cloud?” I needed both, but only the second one could catch the dangerous failure.
4. Test the boundaries between tools
Agent failures often live between individually healthy components:
- The model chose the right tool, but the tool wrote to the wrong account.
- The API returned
200, but the intended record was never created. - The review completed, but it evaluated an older revision.
- The agent reported completion, but the required artifact was missing.
Test the handoff, not only each component. Read the resulting state from the system that owns it.
5. Verify an artifact, not the agent's status
An agent saying “done” is evidence of intent, not evidence of effect.
For a coding agent, inspect the commit and run the test. For a research agent, open the cited source. For a deployment agent, query the live endpoint and assert the behavior that changed. For a financial agent, reconcile the resulting record against the source document.
The final test should observe the world after the agent acted.
An AI agent testing checklist
Before trusting an agent workflow in production, ask:
- What exact state should exist when the task succeeds?
- Can the test detect a plausible but incorrect output?
- Have we deliberately made the test fail?
- Are liveness and correctness reported separately?
- Do we test the handoffs between the model, tools, data stores, and reviewers?
- Does the final assertion read from the system of record?
- Can the agent's completion message disagree with the verified outcome?
- Do we preserve enough evidence to explain a failure later?
If several answers are no, more dashboards will not solve the problem. The system needs a better definition of success.
How this changes AI agent evaluation
Agent evaluation often starts with answer quality: Was the response accurate? Did the model follow the instructions? Those questions matter, but production agents also change external state. Their evaluation has to include whether that state changed correctly.
That means measuring at least three layers:
- Decision quality: Did the agent choose a reasonable action?
- Execution quality: Did the tools perform the requested operations?
- Outcome quality: Is the final state correct in the system of record?
The third layer is the one most likely to be missing—and the one a customer, operator, or investor ultimately pays for.
Replace green lights with claims you can disprove
The fix for my sync failure was simple once I understood it: start the MacBook's daemon and compare the freshest source file with the freshest cloud file. That assertion can fail because it measures the outcome rather than the motion.
The larger lesson is the one I keep. I no longer trust a check until I can say how it would break. If I cannot describe the failure that turns it red, it is not a test. It is decoration.
That may sound like a narrow reliability lesson, but it connects to the larger reason I build these systems. The question that drives my work is not “What can AI do?” It is “What can humans do with AI that was not possible before?”
Reliable agents expand the size of the problems a person can take on. They let a small team operate systems, investigate questions, and build products that once required far more time and organizational weight. Testing is not about restraining that possibility. It is what lets a person safely reach for it.
We build, operate, and evaluate AI-agent systems across the Derivative portfolio. If you need to know whether an agent workflow is genuinely ready for production—or whether a company’s demo survives contact with real operations—we can help you test the outcome, not just the pitch.