Process Metrics for AI Agents: Measuring Execution Quality Beyond Task Completion

Agent reliability now depends on measuring how an agent reaches its result, because a correct final answer can conceal a duplicated tool call or a wasted step; it can also conceal a completion the agent asserted without writing it back.

Gartner projects that over 40% of agentic AI projects will be canceled by the end of 2027 because of escalating costs or unclear business value, or because risk controls are inadequate (Gartner, 2025). Escalating cost and inadequate risk controls leave evidence in the agent’s trajectory, such as duplicated calls and actions outside the agent’s authority; unclear business value becomes measurable once cost per successful task is compared with the human cost per case.

What are process metrics for AI agents?

Process metrics for AI agents measure how an agent reaches its result: the plan it formed, the tools it selected and the arguments it passed, the steps it took, the tokens it consumed, and the policies it respected.

The core set covers plan quality, plan adherence, reasoning coherence, tool selection accuracy, argument correctness, step efficiency, loop rate, latency per step, error recovery rate, token usage, cost per successful task, hallucination rate, and policy adherence rate. Each one is a ratio computed over the recorded trace, so each one can carry a threshold.

Only 10% of the 545 executives HFS Research surveyed use agent-native measures to judge autonomous-system performance (HFS Research, 2026). The bot-era KPIs that fill the gap, throughput and pass/fail, describe the result and say nothing about the path.

A claims agent that sets the correct reserve after running the policy lookup three times passes an outcome test and fails a process test. The process test predicts cost per case and latency under load. It also helps anticipate the next production incident.

Process metrics vs. outcome metrics

Task completion alone hides tool errors that returned plausible data and wasted steps that inflated cost without changing the result. Process metrics also expose false completion where the agent reported success with no write-back, bringing the total to three failure classes.

In a Know Your Customer (KYC) review, an agent can clear the alert while calling the sanctions list with a truncated name or re-running the same adverse-media search four times; it can also mark the case closed before the case management system accepted the disposition.

The comparison below sets the two measurement families side by side:

Outcome metrics compared with process metrics

How process metrics work: The core metric layers

Process metrics work as five layers that each read a different slice of the trace, from the plan the agent formed to the policy limits it respected, and a single failed case usually needs all five to explain. The layer names below match the metric names that evaluation libraries publish, which keeps dashboards comparable across tools.

Reasoning metrics

Plan quality scores whether the plan the agent wrote in its reasoning would complete the task; plan adherence scores whether the executed steps followed that plan; reasoning coherence scores whether each step follows from the step before it.

DeepEval computes Plan Quality as an alignment score between the task and the plan, and Plan Adherence as an alignment score between the task, the plan, and the execution steps (DeepEval, 2026). Both default to a passing score of 1 when the agent wrote no plan, so a governance dashboard must report plan presence as its own field or a planless agent will look fully compliant.

In underwriting, a plan that skips the loss-history check scores low on plan quality even when the agent later binds the policy within appetite.

Action metrics

Tool selection accuracy and argument correctness are scored separately because an agent can select the correct policy-verification service and pass the wrong policy number, and the second error corrupts a system of record without raising an exception.

Arize Phoenix packages Tool Selection as a correct-or-incorrect score and Tool Invocation as a separate check on arguments, formatting, and hallucinated fields (Arize Phoenix, 2026); DeepEval scores Tool Correctness deterministically and Argument Correctness with an LLM-powered judge. A prior authorization agent that queries the correct clinical criteria service with the wrong procedure code returns a plausible denial, and only the argument check catches it.

Execution metrics

Step efficiency, loop detection, latency per step, error recovery rate, and reasoning thrash measure how much work the agent spent reaching its result. In the OSWorld-Human study of desktop agents, 16 agents took 2.7 to 4.3x more steps than necessary to complete the same tasks (OSWorld-Human, 2026). Loop detection flags a repeated identical call, meaning the same tool name, arguments, and output as an earlier call, or an action-error cycle the agent repeats without changing its approach.

Reasoning thrash is the reasoning-only version of a loop: the agent rewrites its plan across consecutive turns without executing a new action. Latency per step splits wall-clock time by span, which separates a slow mainframe lookup from a slow model call. Error recovery rate is the share of tool errors after which the agent still completed the task without a human.

Resource metrics

Teams use token usage and cost per task to price the execution layer and connect a trace to the ROI case. Kapoor et al. recommend reporting input and output token counts separately so a cost can be recomputed at current prices rather than frozen at the price on the day of the run (Kapoor et al., 2025). The cost of a loop compounds because each turn carries the full history forward, so prompt size grows with turn count.

Cost per successful task is the ratio to report, because failed runs consume budget and never appear in an outcome count.

Guardrail metrics

You measure hallucination and safety as process metrics by expressing them as rates rather than incident counts; apply the same approach to policy adherence. Hallucination rate is the share of outputs containing a claim the retrieved evidence does not support; policy adherence rate is the share of actions taken within the agent’s defined authority; safety rate is the share of runs with no flagged unsafe action.

Ragas scores topic adherence with the same precision and recall structure, where precision counts answers that stayed on approved topics over all answers given and recall adds the refused queries the agent should have answered (Ragas, 2025).

Record one hallucination event when a prior authorization agent cites a clinical criterion absent from the payer’s policy; set the threshold against the rate per 1,000 cases.

How do you calculate each process metric?

Each process metric reduces to a ratio over the trace, and the formulas below follow published library and benchmark definitions, with the matching unit named because that unit changes the score. The formulas are listed in the order they appear in a typical review:

• Step efficiency: Reference steps divided by actual steps, where the reference is the shortest trajectory an operator or a prior successful run needed. The Arize Phoenix cookbook computes it as optimal path length over path length with the minimum observed path as the optimum (Arize Phoenix, 2026), and the SPL measure multiplies the ratio by binary success so failed runs score 0 (Anderson et al., 2018).
• Tool precision: Matched calls divided by matched calls plus extra calls.
• Tool recall: Matched calls divided by matched calls plus expected calls the agent never made; F1 is 2 times precision times recall over their sum. Ragas ToolCallF1 matches on the pair of tool name and arguments, and a library that matches on names alone will score the same trace higher because wrong arguments stop counting as misses (Ragas, 2025).
• Argument correctness: Calls with correct parameters divided by total tool calls, scored by an LLM judge in DeepEval (DeepEval, 2026).
• Loop rate: Repeated identical calls divided by total calls. AgentBench’s diagnostic treats any two responses with a Rouge-L similarity of 0.8 or higher within the last 10 rounds as a repetition signal (AgentBench, 2024).
• Error recovery rate: Errors followed by unaided task completion divided by errors encountered.
• Cost per successful task: Total cost of all runs, failed runs included, divided by successful tasks. Cost-of-Pass states the same ratio as the expected cost of 1 correct outcome, cost per run divided by success rate (Cost-of-Pass, 2026); some benchmarks average over successful runs only, so state the denominator convention on every dashboard.

The illustrative trace below shows the arithmetic on an auto physical damage claim, where the reference procedure calls retrieve claim, verify policy, check coverage, read adjuster notes, set reserve, and write back, in that order:

1. Retrieve claim (matched)
2. Verify policy (matched)
3. Verify policy with the same policy number (exact repeat)
4. Check coverage (matched)
5. Fetch claimant history (not in the reference)
6. Read adjuster notes (matched)
7. Set reserve (matched)
8. Write back (matched)

Step efficiency is six reference steps over eight actual steps, or 0.75. Tool precision is six matched over eight calls, or 0.75; recall is six over six, or 1.0; F1 is 0.86. Loop rate is one exact repeat over eight calls, or 0.125.

If the run costs C and the agent closes four of every five such claims, cost per successful claim is C divided by 0.8, or 1.25 C, and the two extra calls sit inside C on every run. These figures are illustrative and come from the trace above rather than from a benchmark.

How do you capture these metrics with traces and spans

You capture process metrics by instrumenting the agent loop so every run emits a session, a trace per task, and a span per model call or tool call, then scoring those spans with deterministic checks and calibrated LLM judges.

Capture units: A session groups the traces for 1 case or conversation; a trace records one task attempt from trigger to result; a span records one unit of work inside it, with start time, end time, status, and attributes.

The OpenTelemetry GenAI semantic conventions name the spans: invoke_agent for an agent run, execute_tool for each tool call with gen_ai.tool.name required and the call arguments and result opt-in, and gen_ai.usage.input_tokens and gen_ai.usage.output_tokens for token counts on the agent span (OpenTelemetry, 2026).

Convention stability: Every GenAI attribute group still carries Development status with none marked Stable, so pin the convention version your instrumentation emits and plan for attribute renames.

Capture runs either through the framework’s own callbacks, which in LangSmith’s case already record tokens, cost, error, status, and latency per run (LangSmith, 2026), or through an OpenTelemetry exporter that any backend accepting GenAI spans can ingest. Instrument at the step boundary in either case, because a span without the tool name and arguments cannot feed the action metrics afterward.

Scoring splits by metric type:

• Tool matches, loops, step counts, and token sums come from span data deterministically.
• Plan quality and reasoning coherence need an LLM judge reading the trace, as does LLM-judged step efficiency.

Wang et al. found that calibrating a judge with human annotation on 20% of items matched or exceeded average human alignment (Wang et al., 2024).

Position swapping and reference-guided judging reduce judge bias further, and comparisons across runs must hold the evaluation mode constant because a judge-free scoring path produces numbers on a different scale.

Multi-agent and human-in-the-loop process metrics

Multi-agent deployments add delegation accuracy and inter-agent latency, along with a separate handoff failure rate; human-in-the-loop deployments measure escalation and intervention, along with fallback to manual processing. Both sets need a human baseline from the same queue to carry a threshold.

Delegation accuracy is tasks routed to the correct agent over tasks delegated.

Inter-agent latency is the time between one agent’s final span and the next agent’s first span.

Handoff failure rate is handoffs where the receiving agent lost or ignored the handed context over total handoffs.

The MAST taxonomy groups those handoff failures under inter-agent misalignment, including conversation reset, information withholding, ignored other agent’s input, and reasoning-action mismatch (MAST, 2025). Each mode is visible in a trace when the receiving span’s inputs omit fields the sending span emitted.

Attribution after the fact is unreliable. In the Who&When study of annotated multi-agent failures, the top automated method identified the failure step in 14.2% of cases (Who&When, 2025), so handoff metrics must be computed from span data at run time rather than reconstructed from logs during an incident review.

Human-in-the-loop metrics use three ratios:

• Escalation rate: Cases the agent routes to a human over cases attempted.
• Intervention frequency: Human edits or overrides, including rejections, over agent actions reviewed.
• Fallback rate: Cases that revert to the manual process after the agent started over cases attempted.

No reviewed standard sets a numeric target for any of the three. The regulator and analyst sources reviewed also provide no numeric targets. NIST’s AI RMF notes that the frequency and rationale with which humans overrule AI output may be useful to collect (NIST, 2023), and Singapore’s IMDA agentic AI framework defines human override rate and warns that a low rate may signal rubber-stamping while short review times may signal automation bias (IMDA, 2026).

Thresholds therefore come from the observed human baseline. Set the escalation ceiling at the referral rate experienced adjusters or analysts show on the same queue. Set an intervention floor above zero so a reviewer who never overrides is checked for rubber-stamping. Treat a fallback rate that rises week over week as process drift rather than agent variance.

Which tools compute which process metrics

DeepEval and Ragas package named agent metrics, LangSmith packages trajectory matching, Arize Phoenix packages tool-level judges, and Langfuse packages trace filters. The table maps each metric to what each tool ships natively:

Process metric coverage by evaluation tool

The definitions come from each tool’s current documentation: DeepEval lists Task Completion, Step Efficiency, Plan Adherence, Plan Quality, Tool Correctness, and Argument Correctness as its default agent metrics (DeepEval, 2026); LangSmith documents its four deterministic match modes (LangSmith, 2026); Langfuse documents code evaluators that expose tool_calls with a 2-second limit and no network egress (Langfuse, 2026). Ragas has deprecated its legacy metric module in favor of the collections API, so new pipelines should import from there.

The public benchmarks score outcomes and record trajectories as a side product:

• AgentBench: Researchers score success rate and F1 across eight environments. They also score step success and record five finish reasons, including Task Limit Exceeded, which is the closest thing to a packaged loop signal; token usage is not reported as a metric (AgentBench, 2024).
• SWE-bench: Researchers score the share of instances resolved, meaning the patch applies and all tests pass (SWE-bench, 2023); reasoning traces became a submission requirement on 7/29/2024 (SWE-bench, 2024), and a proposal to add tokens, cost, or time per task was deferred.
• GAIA: Researchers score a quasi-exact match on the final answer, and the authors state that they do not evaluate the trace leading to that answer (GAIA, 2023); the leaderboard accepts an optional reasoning_trace field.

Use the benchmarks to compare models and scaffolds before selection. Production process metrics come from your own traces on your own queues, because no benchmark reproduces a mainframe policy lookup or a payer’s prior authorization criteria.

How do process metrics connect to agent ROI

Process metrics connect to ROI through cost per successful task: failed runs still consume budget, and the comparison is against the human cost per case on the same queue.

Failed-run costs: The SWE-agent evaluation found that failed runs cost more than successful runs, with a mean of $2.52 and 21 steps per failed run (SWE-agent, 2024). A loop metric is a cost metric read early.

The gap between AI investment and AI return is where that cost lands. McKinsey’s “The State of AI 2025” reports that 39% of respondents attribute any EBIT impact to AI, with most of those reporting under 5% (McKinsey, 2025). Compare cost per successful task with the human baseline at the volume the queue carries, with escalations and fallbacks counted on the agent’s side of the ledger.

The missing baseline:

• Standard operating procedures describe the intended steps.
• Event logs record only what instrumented systems write.
• Process mining reaches end-to-end depth only after integration into every system of record while still missing work that never produces an event.
• Task mining needs no integrations but captures individual tasks rather than the end-to-end process.

For a claims queue that spans digital intake and a mainframe policy system, then requires spreadsheet reconciliation, teams need a human baseline that covers the full workflow. The agent must beat the human step count and cost per case while meeting the exception-rate baseline.

Skan AI closes that gap by building the Context Graph of Work from desktop-level observation of process activity across mainframes, VDI, legacy desktop applications, web applications, spreadsheets, and office tools, pairing end-to-end process depth with zero-integration capture that needs no backend connectors or application changes.

Privacy by design: Skan PRISM applies pixel-level redaction inside the customer environment so raw screen data stays within customers’ control and the unit of analysis stays on the work.

Skan AI Intelligence converts those observations into end-to-end process flows and execution metrics, including processing times and cost per case, which teams use as the human baseline for step efficiency and cost per successful task. Automation Explorer scores the same observed work through separate RPA Opportunity Score and Agentic Automation Score outputs, so teams can assign deterministic steps to a bot and judgment steps to an agent before any trace exists.

Agentic operating procedures (AOPs) encode the observed workflow steps, decision rules, exception handling, escalation thresholds, and success criteria as machine-usable execution rules, so plan adherence and policy adherence are scored against the procedure operators follow rather than against a generic prompt. Skan AI Agents executes approved work within that defined authority, and the observation-to-agent pipeline evaluates each agent against the human baseline before production and benchmarks its performance afterward.

When an adjuster reverses an agent’s reserve recommendation, record the reversal as an intervention event in the trace and, after approval, update the AOP so the agent uses the corrected rule on the next case. Skan AI Blueprint maps operational activity and produces the prioritized roadmap tied to measurable business outcomes, so teams can start the program on queues where the baseline already exists.

Fund the observation layer before the agent, and prioritize queues where the agent writes back to a system of record, because write-back confirmation is the process metric that turns a reported completion into a closed case.