What Skews Task Data in Process Intelligence and How to Correct It

A business case built on skewed task data fails twice: in the CFO review, where no savings number traces back to observed work, and in production, where the automation meets ways of working the data never showed. McKinsey’s “State of AI 2025” reports that only 6% of companies see meaningful EBIT impact from AI investments (McKinsey, 2025). When the task record over-represents the clean, system-logged path and misses the work that happens in email and spreadsheets, as well as terminal screens, the roadmap funds the wrong processes and the committed savings erode quarter by quarter.

What is skewed task data in process intelligence

Skewed task data is a process record in which a small share of cases or variants accounts for most captured events, while other work steps appear thin, sampled, or not at all. The skewed task data process intelligence teams work from has two senses, and operations leaders meet both.

• Statistical or sampling skew: The record’s distribution is lopsided. A few process variants dominate the event count, so every average and model, including each ROI estimate, tilts toward them.
• Engine-level skew: Borrowed from distributed data processing, task skew and partition skew describe a workload in which one slice of the data, such as one application’s log or one team’s sessions, runs far larger than the rest. The processing job stalls on that slice, and analysts see its shape in the summary metrics.

Apache Spark, Hadoop MapReduce, Databricks, and AWS Glue all show engine-level skew, where one overloaded task or partition holds up the whole job however fast the others finish. Adaptive Query Execution (AQE) in Spark 3.0.0 and later and in Databricks flags a partition only above both a multiple of the median size and a byte threshold, while AWS Glue compares the longest task with the median. Repartitioning a skewed task’s remaining work is one documented mitigation, but these compute-scheduling defaults miss partitions below either AQE threshold and are not validated for business process data.

Both senses show up in one claims queue. Picture a property and casualty insurer whose auto physical damage claims run end to end inside the claims platform, while complex liability claims stall in adjuster email and a reserving spreadsheet. Adjusters also use a mainframe policy lookup. The auto claims hold most of the logged events and dominate every discovered model; the liability claims leave a few system events and most of their work unrecorded.

Concentration of this kind is the usual shape of a process record. Van der Aalst and Carmona’s opening chapter of the “Process Mining Handbook” states that most processes have a Pareto distribution, with 80% of cases described by 20% of process variants, and that the remaining cases often hold the majority of performance and compliance problems (Springer, 2022).

What skewed task data looks like: A worked event log example

In a prior authorization queue where 3% of case variants hold 80% of logged events, the log-based cycle-time average understates the real figure by roughly two-thirds, and the automation business case counts savings on volume the bot cannot reach. The numbers below are illustrative, built to show the mechanics; they are not a published benchmark.

Take 10,000 prior authorization requests per quarter at a health plan, with 300 observed variants. Nine variants (3%) cover 7,000 cases and 80% of logged events: the request arrives through the provider portal, the rules engine approves or routes it, a nurse reviewer confirms, and the decision posts, in about 1.5 days.

The other 291 variants cover 3,000 cases and 20% of logged events: the request arrives by fax, an intake coordinator keys it into the utilization management system, and a nurse calls the provider office for missing clinical notes. Status lives in a shared spreadsheet, and only the final two system events are logged, so the recorded span reads about one day while the real elapsed time is nine.

Three outputs shift once the long tail is weighted by observed work rather than by logged events:

• Discovered model: Analysts see the portal path with a handful of exceptions in the log-based model, while they classify the fax path as a rare, short variant. Once they weight the unlogged intake and phone steps, they classify the fax path as a second mainline route with its own stall point at clinical-note retrieval.
• Cycle-time average: Computed from logged events, the blended average is 1.35 days (7,000 cases at 1.5 days and 3,000 at one day). Computed from observed elapsed time, it is 3.75 days, because the 3,000 fax cases take nine days each.
• Automation ROI estimate: A team that scopes a bot on the portal path and costs it across all 10,000 cases claims savings on 100% of volume. The bot can only touch the 7,000 portal cases, so the defensible estimate is 70% of the headline, and the 3,000 fax cases that carry the turnaround-time risk sit outside its reach.

What causes task data to skew

Task data skews when the capture tool observes only part of the desktop or when the team samples capture rather than running it continuously. Log-based tools add another source of skew because they record nothing that is not system-logged. The companion article in this series examines each cause in depth; the summary here covers what the correction steps assume.

• Observation gaps: Task mining recorders capture the applications they can hook, and the rest of the desktop goes unseen. Everest Group’s Digital Interaction Intelligence Playbook states that “Process mining analyzes structured data in processes, but user interactions with business applications go uncaptured” (Everest Group, 2024). A 2026 paper in Business & Information Systems Engineering adds that understanding user actions where structured interface information is unavailable, for example in virtualized environments, “remains unsolved” (BISE, 2026).
• Non-deterministic sampling: HFS Research notes that task mining tools “are often set up to study, for example, two dozen employees over two weeks” and are not typically deployed across an entire workforce (HFS Research, 2024). A 2-week window over two dozen desks is a sample, and a different draw produces a different record. The IEEE Task Force’s Process Mining Manifesto warns that an overfitted model from one sample means “a next sample log of the same process may produce a completely different model” (IEEE Task Force on Process Mining, 2012).
• Integration dependency: Event-log tools need a connector or extractor per source system, so every system without one, and every step that bypasses a system, falls out of the record. The Manifesto’s log maturity scale places logs where “it is possible to bypass the information system” at the second-lowest level, because “events may be missing or not recorded properly”. Deloitte’s 2025 global process mining study reports that legacy systems make that integration complex (Deloitte, 2025).

Four smaller defects accompany those causes:

• Hot keys: Work driven through keyboard shortcuts in a terminal emulator can generate no click for a click-triggered recorder, so the step is undercounted or absent.
• Nulls: Events with a missing case identifier or another required field, such as a resource or timestamp, cannot be joined to a case and get dropped before analysis.
• Dominant variants: The best-instrumented path produces the most events and therefore the most weight in every average.
• Task mining capture bias: A recorder placed on a chosen group of desks for a chosen window reflects that group’s habits, not the process population.

How skewed task data distorts discovery, conformance, and automation ROI

Analysts who use skewed task data can produce a discovered model that omits real paths and a conformance check that flags real work as deviation. They can also produce a cycle-time average that reports only the clean path and an ROI estimate that counts savings on volume the automation cannot reach.

• Discovery and conformance: Analysts train the discovery algorithm on the dominant variant, then use conformance checking to measure the tail against a model that never included it. They may report the stall that costs the most as noncompliance rather than as a path to fix. Timestamp defects compound the error: Suriadi and colleagues show that incorrect timestamps can produce models “substantially different from reality in both event ordering and case duration” (Suriadi et al., 2017).
• Cycle-time baselines: The operations team uses the cycle-time baseline from that model as the number the business case commits to. Deloitte’s 2020 global intelligent automation survey of 441 executives found that only 38% of organizations have mature process definitions and process management standards (Deloitte, 2020). An immature definition plus a skewed record gives the CFO a baseline with no stated coverage and no stated variance.
• ROI estimates: ROI inflation follows the same shape as the skew. Everest Group describes an accounts payable automation that delivered a 60% FTE reduction when scoped on North American data extraction, which fell to 10% when extended across all regions to coding and exception handling (Everest Group, 2019). The narrow scope did not represent the full process.
• Delivery risk: Builds stall when the variants surface during delivery. In a multi-case study of five banking, financial services, and insurance firms, Kaniadakis and Linturn report that one bank’s original process assumptions were wrong because of unappreciated variation between regions and product lines across the system estate, and its automation lead said “The business case pretty much evaporated or became so complex” (IJISPM, 2022).

How do I know if my process intelligence data is reliable?

Process intelligence data is reliable when you can state what share of desktop activity it covers and show that variant concentration holds across employees and capture windows. You must also show that duration variance comes from the work rather than from the recorder.

The following signals, computed on your own record, separate a capture artifact from a process finding:

Reliability signals and flag thresholds for a process record

• Coverage share
  • How to compute: Observed desktop time and applications per process step, against the applications the step is known to touch
  • Flag when: Any step runs on an application with zero observed events
  • Basis: No published coverage benchmark exists; the gap list is your own
• Variant concentration ratio
  • How to compute: Share of cases covered by the top 20% of variants
  • Flag when: Above 80%, and the remaining cases carry most of the delay or rework
  • Basis: The Pareto pattern described above
• Task duration variance
  • How to compute: Coefficient of variation of step duration per variant
  • Flag when: Above 1, with the spread concentrated in unobserved applications
  • Basis: General statistical convention, not validated for process data
• Skewness coefficient
  • How to compute: Fisher-Pearson skewness of step and case durations
  • Flag when: Beyond ±1, the usual textbook band for high skew
  • Basis: Borrowed convention; no process-mining cutoff exists
• Gini coefficient
  • How to compute: Gini of case volume across variants, or of workload across roles
  • Flag when: A rise between capture windows; no validated cutoff
  • Basis: Trend measure only
• Straggler tasks
  • How to compute: Steps running longer than 1.5× the median for that step
  • Flag when: A straggler cluster tied to one application or one team
  • Basis: Engineering convention from Microsoft Research’s Mantri study (Microsoft Research, 2010)
• Partition size spread
  • How to compute: Ratio of largest to median event count per employee or session
  • Flag when: Spread aligned with recorder placement rather than case volume
  • Basis: Capture artifact check

Run the coverage check before the statistical ones. A step with zero observed events invalidates the duration and variant measures computed around it, because those measures describe only the portion of the work the recorder could see.

How to correct skewed task data: A 5-step methodology

Correcting skewed task data means closing the coverage gap and replacing sampled capture with continuous observation. Teams must then validate the rebuilt record against multiple employees and process-level KPIs before any business case uses it. The steps run in order because each depends on the record the previous step produced.

1. Audit your current coverage. List every application each process step touches, from the intake channel to the system of record, and mark which ones your current tool observes. The audit produces a percentage of desktop activity captured, stated for each process, and that percentage becomes the denominator for every later claim. A business case without that percentage has no coverage statement for a finance reviewer to test.
2. Identify legacy and mainframe blind spots in your current tool. Mark the steps that run in terminal emulators, Citrix or VDI sessions, shared spreadsheets, email, and phone calls, since log-based tools and API-hooked recorders see these poorly or not at all. In a KYC review, that is the sanctions screening lookup on the mainframe and the analyst’s spreadsheet of open requests; in claims, it is the policy lookup and the reserving workbook. Each blind spot marks a step whose duration and variants analysts currently estimate rather than observe.
3. Switch from sampling to continuous real-time observation. Replace the 2-week, 24-desk study with desktop-level observation that runs across the full population of the process for as long as the analysis needs. Continuous capture avoids repeated sample draws, so analysts work from one record and can catch variants that occur a few times a quarter. Keep the unit of analysis on the work, with screen content redacted at capture, so privacy and coverage reviewers can evaluate both controls together.
4. Validate process maps with variant analysis across multiple employees. Compare the variant distribution per employee and per team before accepting a map; the PM2 methodology separates verification against the original data from validation with the process experts who do the work (CAiSE, 2015). Analysts should classify a variant that appears on one desk as a habit and one that appears across most desks in one region as a process. No published threshold says how many employees must agree, so set one and record it in the business case.
5. Run a before/after comparison at the process KPI level. Measure cycle time, error rate (rework and resubmission), and application switching frequency per case on the corrected record, then again after the intervention. Compare the corrected baseline with the one the old tool produced; analysts can use the difference to quantify the measurement error the previous business case carried. A CFO can tie those three KPIs to cost per case.

Legacy task mining versus continuous process intelligence

Legacy task mining and continuous process intelligence differ on the four data properties the correction methodology depends on: how much of the desktop they cover, whether two captures agree, whether they see legacy applications, and whether they produce a process-level view.

The comparison below summarizes the two approaches on those properties:

Legacy task mining compared with continuous process intelligence on four data properties

Legacy task mining approach

• Coverage: A sampled group of desks for a fixed window; applications the recorder cannot hook go uncaptured
• Consistency: Each study is a new sample, so two captures of the same process can disagree
• Legacy app support: Terminal and virtualized sessions captured poorly or excluded
• Process-level view: Task-level fragments per user that need manual stitching into a process

Continuous process intelligence

• Coverage: Desktop activity visible across the process population for as long as observation runs
• Consistency: One continuous record reduces differences caused by separate sampling windows
• Legacy app support: Mainframe, VDI, Citrix, legacy desktop, web, spreadsheets, and office tools observed at the screen
• Process-level view: End-to-end flows, variants, and KPIs per case, updated as the work changes

The second approach uses desktop-level observation to close the coverage gaps identified in the five-step methodology:

• Observation and telemetry: Desktop-level observation captures work on the screen, where people do it, so it records steps in mainframe, VDI, Citrix, legacy desktop, web, spreadsheet, and office applications that event logs and API-hooked recorders miss. It produces a continuous stream of observed steps and cross-application handoffs, which the coverage audit in step 1 measures. Skan AI builds its Context Graph of Work from that observation, and its lightweight Skan Virtual Assistant observes process activity across mainframes, VDI, legacy desktop applications, web applications, spreadsheets, and office tools without backend integration for every application or application changes.
• Privacy by design: Pixel-level redaction runs inside the customer environment before anything leaves it. Email content and keystrokes are never captured, and the unit of analysis stays on the work rather than the worker.
• Process analysis: Skan AI Intelligence converts the telemetry into end-to-end process flows, variant analysis, execution metrics, bottleneck detection, and drift detection, which covers steps 4 and 5 of the methodology. The resulting Context Graph of Work functions as a digital twin of operations, a continuously updated model of how cases move across people, applications, decisions, and policies. Teams can update the corrected baseline as the work changes. Initial process insight arrives in weeks, not months.
• Agent execution: Skan AI Agents uses the same Context Graph of Work and agentic operating procedures to define what an agent can do, when it must stop, and when a human must decide. An automation build that starts there starts from the corrected record, including the fax path and the mainframe lookup, rather than from the skewed one.

When skew is the finding, not the defect

Operations teams sometimes find that a lopsided distribution is the answer the analysis set out to find: one rework or exception loop, or one application handoff, holds most of the delay. Smoothing the distribution erases the finding.

Process-mining analysts tend toward smoothing because it improves the discovered model. Conforti and colleagues show that 2% infrequent behaviour can cause a 49% precision drop for one discovery algorithm, which is why filtering rare traces is standard practice (Conforti et al., 2017). Fani Sani and colleagues caution from the other side that “Removing traces potentially distorts the general distribution of common behaviour” (Fani Sani et al., 2018), and the “Process Mining Handbook” records the dispute as unresolved, with some advocating repair while others argue the data should be left alone because it reflects reality (Springer, 2022).

Analysts can use two tests to separate signal from artifact, and each maps to a reliability check from the table above:

• Persistence under coverage and across employees: Analysts should classify skew that shrinks when observation extends to a previously unobserved application as a recorder artifact. If the skew holds once every application in the step is observed and appears across most desks in a team or region, they should treat it as a property of the work. If they find it on one desk, or only on desks where the recorder could not see the terminal session, they should treat it as a habit or a capture gap.
• Alignment with an outcome: Analysts should treat skew that coincides with a measured operational difference, such as cycle time or rework rate, as the finding. They can also compare application switching. If no outcome is attached, they should note the distribution rather than treat it as a problem to fix.

In a KYC review, a loop in which the analyst returns to the sanctions screen three times per case, on most desks, with a longer cycle time each time, is the rework loop the business case should target. The same loop appearing only where the mainframe session went unobserved is a hole in the record, and the process team must fill it before anyone prices a fix.