Training AI Agents: Logs vs. Observation

AI agents built from event logs and standard operating procedures fail in production because those inputs omit the exception paths, spreadsheet workarounds, and cross-application handoffs that regulated work runs on. Gartner projects that “over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls” (Gartner, 2025). The agents that reach production and stay there learn from observed work, where the exceptions are visible.

A backend event log records a transaction when an instrumented system commits it. It does not record the adjuster who opens a policy in a mainframe session, checks an email attachment, and keys a reserve into a spreadsheet before the claims system receives a single update. Desktop-level observation captures that sequence, and the sequence is where an agent’s decision logic has to come from.

How to train AI agents on real work: The short answer

How to train AI agents on real work reduces to five ordered steps: capture observed work, separate rules from judgment, choose the cheapest context method that holds, connect with bounded permissions, and evaluate on real cases before widening authority.

Treat the agent as a new hire who learns from the cases your operators handle rather than from the manual they were handed on day one.

The steps run in this order because each one consumes the output of the one before it:

1. Capture the real work: Collect workflow traces and closed cases. Add current SOPs, then convert the inputs into examples and evaluation cases.
2. Break the workflow into deterministic code and AI judgment: Write a function for every step a rule can decide, and reserve the model for steps that need interpretation.
3. Choose the context method: Start with instructions and skills, add retrieval and memory when the agent lacks knowledge, and fine-tune only for output consistency.
4. Connect the agent to production systems safely: Grant per-tool permissions and sandbox execution. Require human approval before writes to systems of record.
5. Evaluate on real tasks and roll out in stages: Run the agent in shadow mode against a human baseline, then supervised, then autonomous within defined authority.

Steps 1 and 5 depend on the same asset, an observed baseline of how operators complete the work today. Without it there is nothing accurate to decompose in Step 2 and nothing to measure the agent against in Step 5.

What training an AI agent actually means

Training an enterprise agent rarely means changing model weights; it means supplying a language model with instructions, tools, memory, and a loop, then engineering the context that loop reads at every turn. The model is the smallest part of the system to change.

An agent is a language model that calls tools in a loop. Each turn it reads its context, selects a tool or produces an answer, observes the result, and repeats until an exit condition arrives: a structured output, a maximum turn count, an error, or an escalation. Instructions tell it what to do, tools let it act on a CRM or a core system, and memory carries state within a session and, where configured, across sessions.

Context engineering and fine-tuning solve different problems:

• Context engineering selects the smallest set of high-signal information the model sees per turn: the procedure for this case type, the retrieved policy clause, the case data, the tolerance threshold.
• Fine-tuning adjusts weights on curated examples to change tone or format and improve output consistency. Provider documentation from Microsoft describes it as changing behavior rather than adding fresh knowledge.

Set expectations with your sponsors accordingly. No provider documents that fine-tuning teaches a model how your prior authorization team handles a missing clinical note; that knowledge enters through context, which means it has to exist in a form the agent can read.

Why logs fall short and observation grounds agents

Event logs record what instrumented systems commit; observation records what operators do between those commits, including the exception handling and handoffs that break agents built only from logs and documents. The gap is structural, so adding more logs narrows it without closing it.

The one multiple-case study that measured this gap directly found it large. Beerepoot, Šinik, and Reijers observed six professional-services staff and calculated that the main system’s event log would have covered 32.5% of client-related time, and none at all for one participant (Beerepoot et al., 2024). The sample is small, and the remaining time sat in other applications the log never saw.

What sits outside the log is the material an agent needs most. Workaround studies document procurement completed by phone and bypassing the ERP, and spreadsheets maintained because the system of record could not hold the required fields.

A workaround carries intent, and intent is rarely registered as an event, so a model trained on the log learns the sanctioned path and none of the detours operators take to finish a case. Processes dependent on tacit knowledge are especially exposed when agents rely only on formal documentation.

The two legacy approaches split the problem:

• Process mining reaches end-to-end depth only after integration into every system of record, and it still misses work that never reaches an event log.
• Task mining needs no integrations but captures individual tasks rather than the end-to-end process, so an agent gets detail on 1 screen and nothing on the handoff to the next.

Skan AI closes that gap at the data layer with a Context Graph of Work. It builds the graph from observed process activity, so the graph holds the dependencies, cross-system handoffs, documented-process gaps, and exception patterns that neither a log nor a task recording contains on its own.

Logs vs. observation for agent training data

Teams choosing between event logs and desktop-level proccess observability must assess coverage and exception visibility, then integration effort, to determine whether an agent sees the work it will be asked to perform. The comparison below describes the two data sources at category level:

Event logs and desktop observation as agent training inputs

Use the table as a sourcing decision per process. A loan origination workflow that runs entirely inside one modern platform may be well served by its logs; a Know Your Customer (KYC) review that moves between a screening tool, a document viewer, a core banking session, and email will not be.

Step 1: Capture the real work as SOPs, examples, and traces

Before anyone writes a prompt, capture screen-level workflow traces and closed cases with their outcomes. Add the current SOPs, then convert these three inputs into labeled examples and held-out evaluation cases. The SOP states intent; the traces and cases state fact.

Each input serves a different purpose:

• Workflow traces: The observed sequence of applications, steps, decisions, and handoffs for each case type, including the paths operators take when the standard path fails.
• Closed cases and tickets: The input and final outcome, with the human decision that produced it. These become the expected result an evaluation case checks against.
• Current SOPs: The documented procedure, useful as the control to compare observed paths against and to identify where the document and the work diverge.

Annotate each trace with its outcome and whether it followed the standard path or an exception. Mark where the operator paused for judgment. Split the annotated set in two: examples the agent will see as instructions or retrieved context, and a held-out set it never sees, reserved for Step 5.

Privacy by design governs the capture. Keep the unit of analysis on the work rather than the worker, exclude email content and keystrokes from capture, and redact protected information before any data leaves the environment. Skan PRISM, an on-premises privacy and information security gateway, applies pixel-level redaction so raw screen data stays inside the customer-controlled perimeter.

Step 2: Break the workflow into deterministic code and AI judgment

Separate every step a rule can decide from every step that needs interpretation, write the first set as code, and give the model only the second. An agent that re-derives a tolerance check with a language model adds cost and variance to a decision a function makes correctly every time.

Take a property claim arriving for coverage verification and initial reserve. The operator checks the policy in a core system, confirms coverage dates against the loss date, screens for a duplicate claim, reads the adjuster’s first-notice notes, judges whether the attached photos support the stated damage, and sets a reserve from a table keyed to peril and severity. Decomposed, the steps fall into two groups:

• Deterministic: Policy lookup, coverage-date comparison, duplicate screening, and reserve calculation from the table. Each has defined inputs and one correct output.
• Judgment: Interpreting the free-text notes, deciding whether the attachment supports the claim, and apportioning liability when two parties are named. Each depends on reading context and applying experience.

If you can write a function for the step, write the function. The judgment steps are where the captured traces matter most, because teams can see in the examples how operators resolved ambiguity and use those resolutions to define the decision rules and escalation thresholds the agent runs against.

Then, score each step on rule density and exception frequency, then assess its judgment requirement. Route the high-rule, low-judgment steps to deterministic automation regardless of which platform executes them.

How do you choose between instructions, RAG, and fine-tuning?

Start with instructions and skills, add retrieval and memory when the agent needs knowledge it does not have, integrate tools when it must act, and fine-tune only when output consistency still fails. This is Step 3, and the order follows cost and auditability.

Compare the methods by cost and data requirements. Then identify the failure each method fixes.

Context methods for enterprise agents by cost, data, and fit

Teams must now account for provider availability when deciding whether to fine-tune. OpenAI’s deprecations page states that from 7 May 2026 organizations that had never run fine-tuning can no longer create jobs, with further restrictions through January 6, 2027 (OpenAI, 2026). Teams that planned on fine-tuning as their path to domain behavior need a context-based path instead.

For regulated work, teams must also assess auditability. They can read and version a procedure supplied through instructions or retrieval, then show it to a reviewer; they can’t inspect a behavior baked into weights to see which rule it applied to a given claim.

Step 4: Connect the agent to production systems safely

Connect the agent through its own identity with the minimum permission per tool, and execute inside a sandbox with short-lived credentials. Require human approval before any write to a system of record or any irreversible action. Authorization belongs in code, never in the prompt.

The controls that hold in practice are specific:

• Per-tool scopes: Allow read-only database queries and no send or delete rights on email. Limit API permissions to the minimum needed for creating or updating records.
• Per-action authorization: A policy engine checks each action in the context of the specific user and case, so the model cannot talk its way past a limit.
• Sandbox and credentials: Isolate execution and deny network access by default, with allowlists. Use a proxy to inject credentials the agent never sees.
• Human-in-the-loop gates: Mandatory approval for high-impact actions, with the agent failing closed when review times out.

Regulators have written the oversight requirement into law for some of these workflows. Under the EU AI Act, Annex III treats creditworthiness assessment and life and health insurance risk assessment as high-risk. Article 14 requires that natural persons overseeing such systems can detect anomalies and can override outputs or interrupt the system. Article 12 requires automatic event logging over the system’s lifetime (EUR-Lex, 2024).

The connection problem in those institutions is rarely the API itself. Core policy and claims platforms, mainframe sessions, VDI-delivered desktops, and point-to-point interfaces between them mean that extracting event data or building connectors into each system of record takes months of IT work before an agent can see 1 case.

Observation and capture. Skan AI, an agentic AI platform, takes the observation route to that problem. Skan Virtual Assistant, a lightweight desktop sensor, captures process activity across mainframes, VDI, legacy desktop applications, web applications, spreadsheets, and office tools without backend connectors, application changes, or system downtime; IT still provisions the Skan VA and reviews its privacy configuration. That pairing of end-to-end process depth with zero-integration capture is what separates the approach from process mining, which requires integration into each system of record, and from task mining, which requires none but stays at the task level.

Skan AI builds the Context Graph of Work from those observations: a continuously updated property graph connecting people, applications, tasks, decisions, policies, and outcomes through the sequences and governance constraints that link them.

Skan AI Intelligence converts the same observations into end-to-end process flows, variant analysis, execution metrics, and drift detection, which supplies the decomposition in Step 2 and the human baseline Step 5 measures against.

Governed execution. Agentic operating procedures (AOPs) are machine-usable representations of that observed work: workflow steps, decision rules, exception handling, policies, escalation thresholds, execution rules, and success criteria.

Skan AI Agents executes approved work against AOPs and the Context Graph of Work. Agent Studio sets scope, permissions, boundaries, escalation paths, and controlled launch conditions. The control and governance layer records inputs, outputs, actions, decisions, and rationales for case-level review.

That sequence from observation to governed execution is the observation-to-agent pipeline (O2A pipeline). It carries policy controls into the rollout, alongside evaluation against human baselines and performance benchmarking.

Step 5: Evaluate on real tasks and roll out in stages

Build the evaluation harness from the held-out real cases in Step 1, define pass criteria against the observed human baseline, and widen the agent’s authority only as it clears each stage. An agent that was never measured against the people doing the work cannot be shown to match them.

A harness has four parts:

Task and trial: A task is one real case with an expected outcome. A trial is 1 attempt, started from a clean environment.

Transcript and grade: A transcript is the full record. The grade compares the final state to the expected one. Prefer deterministic graders that assess the outcome rather than the path, and allow partial credit where an operator would.

Run multiple trials per task, because a single pass hides variance. Anthropic’s evaluation guidance gives the arithmetic: an agent with a 75% per-trial success rate clears all 3 out of 3 trials only about 42% of the time (Anthropic, 2026). Set regression suites near 100% and let capability suites start low; a regression suite that drops signals that a prompt or procedure change broke something that used to work.

Roll out in three stages, each with an exit criterion:

• Shadow mode: The agent receives replicated production cases and returns no actions; graders compare its decisions with the operator’s.
• Supervised: The agent proposes each action and a person approves, edits, or rejects it before execution, with every reversal logged.
• Autonomous within authority: The agent executes approved case types end to end, escalates on defined thresholds, and stays under continuous monitoring.

Gartner forecasts that by 2027, 40% of enterprises will demote or decommission autonomous agents because of governance gaps discovered only after production incidents, and names uniform, binary governance as the root cause (Gartner, 2026). Staged authority per case type is the alternative to treating every agent as either locked down or fully trusted.

Benchmark against observed baselines, not estimated ones: cost per case, cycle time, exception resolution time, and the share of cases resolved without escalation as your operators perform them today. When observation shows the process has drifted, re-run the harness before the agent encounters the change in production.

Which frameworks should you use to build agents?

Choose the framework by who will maintain the agent: n8n for teams that want a visual canvas with optional code, CrewAI for Python multi-agent builds, LangGraph for stateful long-running orchestration, and Microsoft Agent Framework rather than AutoGen for new Microsoft-stack work. None of them supplies the operational context from Step 1.

The four projects differ in license, positioning, and the skill level they assume:

Agent-building frameworks by license, positioning, and fit

Confirm each project’s current release and license terms on its repository before you commit.

What every framework provides is execution scaffolding: the loop, tool calling, state management, and approval middleware. What none provides is the content of the loop, meaning which exception paths exist in your claims intake, what the variance tolerance is on a PO match, and who approves a reserve above a threshold. Teams that treat the framework as the answer end up hand-coding business logic from interviews, and that logic changes faster than engineers can update it.

Worked example: Training an agent to process invoices

An accounts payable agent built from observed traces, a defined procedure, scoped ERP tools, and a shadow-mode evaluation against the AP team’s own outcomes shows the five steps end to end. The work below follows 1 invoice type: a PO-backed supplier invoice at a large insurer’s shared-services center.

Observed traces show what arrives and where it stalls. An invoice lands in a shared mailbox as a PDF; an AP analyst keys the header into the ERP, which runs a three-way match against the PO and the receipt; a price variance above tolerance places the invoice on hold; the analyst opens the contract in a document repository, checks the agreed price in a spreadsheet the team maintains, and emails the buyer; the case waits until the buyer replies, then the analyst releases the block.

The documented SOP covers the first three steps and none of the detour, which is the part that consumes the cycle time. Decomposition separates the steps cleanly:

• Deterministic: Header extraction, the three-way match, tolerance comparison, and duplicate screening.
• Judgment: Reading the contract clause and judging whether the supplier’s price explanation holds. The agent also decides whether to escalate to procurement.

Routing. The team scores rule density and exception frequency, then judgment requirement, to route the match and tolerance steps to deterministic automation and variance resolution to the agent. Automation Explorer produces that split as separate RPA Opportunity Score and Agentic Automation Score outputs.

Procedure. The team then encodes the procedure as an AOP. It encodes the ordered steps, the tolerance threshold as a decision rule, the exception paths observed in the traces (no PO on file, suspected duplicate, price variance, missing receipt), the escalation condition for each, and the success criterion that an invoice is correctly matched or correctly routed.

Skan AI Agents executes against that AOP with the Context Graph of Work as its model of the handoffs between AP and procurement and their connections to receiving.

Permissions. The team connects tools with scoped permissions: read access to PO and receipt data, plus contract data. The agent can draft the buyer email for approval but has no authority to release a payment block without a person approving the action during the supervised stage. The team records every input, output, action, and rationale per invoice for review.

Evaluation. The team evaluates the agent in shadow mode on the held-out invoices from Step 1. Graders compare the agent’s match decision and routing with the analyst’s recorded decision. Passing requires agreement with the human decision on standard cases and correct routing on each exception type, with no false release of a block.

For reference, a peer-reviewed deployment at a multinational electronics distributor processed 76% of roughly 80,000 invoices with low or no manual intervention over more than a year of production (Tater et al., 2022); your own observed straight-through rate is the baseline the agent has to meet.

The iteration loop runs on reversals. When an analyst overrides the agent’s recommendation to release a hold because the supplier’s explanation referenced a superseded contract, the approved override updates the AOP with a new decision rule: check the contract version date before accepting a price explanation. The team uses Skan AI Blueprint to map operational activity and prioritize a roadmap tied to measurable business outcomes as the agent takes on more invoice types.