How I Would Evaluate an AI Agent That Moves Money

Consider an agent in payroll operations. A scanned court order arrives as a PDF. The agent reads it, extracts thirty-odd fields, decides which of several withholding rules applies, writes a garnishment record, and schedules a deduction from someone's next paycheck. Payroll providers, Deel among them, have begun publishing production examples of exactly this genre, and the pattern of their write-ups is consistent: the extraction was the quick part, and most of the engineering went into making the workflow safe.

That pattern is the subject of this note. When an agent's output can change a payment, a filing, or a record of fact, the question "how accurate is the model?" is the least informative question you can ask about it. It is not unimportant. It is upstream of almost everything that actually goes wrong.

What follows is the evaluation architecture I would build for a system like that. It is written as a standalone framework. The observations behind it come from eighteen months of operating one agent system with real side effects, plus five years of shipping applied models at a payments company, and I will say so where an observation is from a sample of one.

Three instruments, not one

The first thing to separate is three things that get called "the benchmark" interchangeably.

A model benchmark asks whether the model produced the right interpretation of an input. Did it read the order correctly, pick the right jurisdiction, extract the right amount. This is the benchmark most teams build first, because it is the one the tooling makes easy.

A workflow benchmark asks whether the system, with the model inside it, reached the right end state through a safe path. Exactly one record written. No duplicate deduction on retry. Escalated when it should have. Recoverable when something timed out halfway.

A business-outcome benchmark asks whether the deployment produced the result the business wanted, net of everything: hours saved, errors avoided, errors introduced, cost of review, cost of the incidents nobody predicted.

These three do not reduce to each other. A model with 99% extraction accuracy can sit inside a workflow that double-books 2% of deductions on retry, and that workflow can sit inside a deployment that saves eight thousand hours and generates one regulatory letter that costs more than the hours were worth. You need all three, and you need to stop reporting the first one as if it were the third.

The rest of this note is mostly about the second instrument, because it is the one that is usually missing.

Eleven surfaces

I think of a consequential-agent evaluation as eleven separate surfaces. Each has a distinct question, a distinct way to measure it, and a distinct way to fail. Averaging across them hides exactly the failures you built the evaluation to find, so the first rule is: score them separately, compose them deliberately.

1. Semantic correctness

Did the model understand the input correctly?

This is the familiar one. Field-level extraction accuracy, classification accuracy on the decision-relevant attributes (which rule applies, which entity, which effective date), measured against a labeled corpus.

Two things make it harder than it looks in this domain. First, the labels have to be produced by people who can actually be wrong, which means you need inter-rater agreement before you trust the gold set. Second, "correct" is sometimes undefined: an ambiguous order with two defensible readings is not a labeling error, it is a case that belongs in surface 5.

2. Action correctness

Given a correct reading, did the system perform the right operation?

Separate this from semantic correctness on purpose. A system can understand the input perfectly and still call the wrong tool, pass the right value to the wrong parameter, or apply a correct rule to the wrong account. Measure it by holding interpretation fixed (replay with the gold interpretation injected) and scoring the resulting action against the expected action.

Doing this decomposition tells you where to spend. If semantic correctness is 97% and action correctness is 91%, your problem is not the model.

3. State integrity

Did the workflow reach a valid state exactly once?

Every consequential workflow should have an explicit state machine, and the evaluation should check that each run traversed it legally and terminated in a valid terminal state, once. Count runs that end in no terminal state, runs that end in two, and runs that reach a terminal state through an illegal transition.

This is the surface my own operation taught me most about. A build pipeline I run once rewrote 40% of a production tool overnight because it was handed a job without the existing source bundled in. It reached a "valid" terminal state. The state was valid and the outcome was a disaster, which is why state integrity is necessary and not sufficient, and why surface 10 exists.

4. Idempotency

What happens on duplicate delivery or retry?

Deliver the same input twice. Deliver it twice concurrently. Crash the agent after the write and before the acknowledgment, then retry. The expected result in every case is one record, one deduction, one notification. Measure the duplicate rate directly; do not infer it from the happy path.

The engineering answers here are well known (compare-and-swap writes, idempotency keys, explicit timeout states). The evaluation point is that they have to be tested as a scored surface, because they are the kind of thing that works in staging and fails under a message-queue hiccup at 2 a.m.

5. Escalation calibration

Does the system know when not to act?

Build a corpus of cases that should be escalated: ambiguous jurisdiction, conflicting orders, amounts outside plausible ranges, documents that are not what they claim to be. Score two rates: how often the system escalates when it should (recall on hard cases) and how often it escalates when it should not (false-alarm rate, which is a real cost because every escalation consumes a reviewer).

Then plot them against each other and decide, explicitly, where on that curve the business wants to sit. That choice is a policy decision and it should be written down as one.

6. Human override

Can a person stop or alter the path cleanly, at any point?

Measure three things. Time from the override signal to the system actually halting. Whether the halt leaves the workflow in a valid state (see surface 3). And whether an override at each stage of the state machine is possible at all; there is usually at least one stage where it is not, and that stage is where your incident will happen.

I would also measure the override rate in production as a standing metric. In my own system, roughly 15% of high-risk actions get denied at the approval gate. Most of those are timing; a few are genuine catches. If that rate ever drops to zero, either the system has become very good or the reviewer has stopped reading, and the evaluation has to be able to tell which.

7. Provenance

Can every consequential output be traced to evidence?

For every write, the system should be able to answer: which source document, which fields, which model output, which rule, which approval, and when. Score this as a coverage rate over consequential outputs, and audit a sample by actually walking the chain backward. A provenance record that exists but cannot be followed by a person in under a few minutes is not provenance, it is a log.

8. Recovery

What happens under timeout, partial failure, or stale state?

Inject the failures deliberately. Kill the downstream API mid-call. Return a 200 with a malformed body. Serve the agent a record that was true an hour ago and has since changed. Score whether the system detects the condition, whether it recovers to a valid state, and whether it tells anyone.

Stale state deserves its own line. In a system with durable memory, the worst failure is not forgetting; it is acting on outdated information with full confidence. The agent does not know its memory is stale. The evaluation has to find that out for it, which means freshness and source-of-truth checks are part of the scored surface, not a nicety.

9. Distribution shift

What happens on jurisdictions, forms, and workflows the system has not seen?

Hold out entire slices rather than random rows: a state, a court, a document template, a client. Measure the degradation across every surface above on the held-out slice, not just semantic correctness. The important finding is usually not that accuracy drops; it is that escalation calibration drops with it, so the system gets worse and less aware that it is worse at the same time.

10. Regression

Does a prompt, model, or tool change silently degrade behavior that used to be correct?

Keep a replay corpus of real past cases with their verified outcomes. Every change to a prompt, a model version, a tool, or a rule reruns the corpus and diffs the behavior on every surface above. Gate the release on it.

This sounds obvious and is almost never done for prompts, because prompt changes feel like editing a document rather than shipping code. They are shipping code. My own rule after the build-pipeline incident was that no build ships without the existing behavior bundled in and re-verified. The same rule applies to a model upgrade.

11. Cost and latency

What reliability improvement are we buying per unit of cost?

Every surface above can be improved by spending more: more model calls, more review, more retries, more verification. The evaluation should report the cost per case alongside each reliability metric so that the trade-off is a decision rather than an accident. A system that is 0.3 points more accurate and three times as expensive per case has a business-outcome benchmark that may well have gotten worse.

Composing the score

Having eleven surfaces is useless if the dashboard shows one number. Here is how I would compose them.

Do not average. A mean across surfaces lets a strong extraction score mask a weak idempotency score, which is precisely the mask you built the evaluation to remove.

Weight by consequence. Define, per workflow, the cost of a failure on each surface. A duplicate deduction costs more than a slow one. A missed escalation on a conflicting order costs more than a false alarm. Those weights are a policy statement and should be reviewed by the people who own the consequences, not just the people who own the model.

Report floors, not just means. For each surface, the metric that matters is usually the worst slice, not the average. Report the worst-performing jurisdiction, template, or client alongside the aggregate.

Separate the three instruments. Model benchmark, workflow benchmark, business-outcome benchmark. Three panels. Never let the first stand in for the third.

The harness

Concretely, I would build:

• A replay corpus of real, anonymized past cases with verified outcomes, versioned, growing with every production escalation that gets resolved.
• Adversarial fixtures for surfaces 3, 4, 6, and 8: duplicate delivery, concurrent delivery, mid-call failure, stale upstream state, override at every stage.
• Held-out slices for surface 9, chosen by jurisdiction and template rather than by row.
• A shadow lane for new models and prompts: run them against live traffic, write nothing, diff against the production path and against held-out human decisions. Nothing graduates from shadow until its error profile is known, not just its accuracy.
• A regression gate that reruns the corpus on every change and blocks release on any surface moving the wrong way beyond a stated tolerance.
• A standing production panel showing escalation rate, override rate, duplicate rate, provenance coverage, and cost per case, with the worst slice named.

None of this is exotic. Most of it is plumbing. The discipline is in treating each surface as a first-class metric with its own fixtures, rather than treating "accuracy" as the metric and everything else as engineering hygiene.

What this framework cannot guarantee

I want to be precise about the limits.

The observations behind this note are from operating one agent architecture with one operator and a particular set of tools and models. The failure classes I weight most heavily, inference treated as permission, stale state acted on with confidence, valid-but-wrong terminal states, are the ones that happened to me. Another system will have others, and the right response to that is to extend the surfaces, not to trust these eleven as complete.

The framework also cannot tell you where to sit on the escalation curve, how much to spend per case, or what the consequence weights should be. Those are business and policy decisions. What the framework does is make them visible, so that they get made on purpose.

And it cannot make the model the hard part again. In every consequential-agent system I have run or read about, the model was the easy part. The hard parts were state, control, recovery, and the willingness to measure the things that are inconvenient to measure.