What Should an AI Readiness Assessment Check Before You Scale an AI Pilot?

An AI readiness assessment scores a pilot on seven dimensions before it scales: data, integration, security, governance, cost, ownership, and adoption. Each dimension carries an explicit go/no-go threshold. The assessment ends in 1 of 3 decisions: scale, remediate, or stop.

What does an AI readiness assessment check before you scale a pilot?

An AI readiness assessment asks whether 1 specific AI use case can run in production, under your controls, at a cost the business accepts, first within a defined scope and then at full volume. It scores the pilot on seven dimensions:

• Data: Production data for the target process meets quality, availability, and lineage criteria, including the exception paths the pilot may have skipped.
• Integration: The pilot connects to the systems of record, legacy applications, and compute it needs at production scale.
• Security: Access controls, permission hygiene, and oversharing risks are resolved before the AI reads or acts on production content.
• Governance: Required sign-offs are complete and the use case maps to the frameworks and regulations that apply to it.
• Cost: Run cost at production volume is modeled and the ROI clears an agreed hurdle.
• Ownership: A named executive sponsor and a named production owner are accountable for the system after go-live.
• Adoption: The people who will use or supervise the system have the skills, incentives, and change support to change how they work.

Each dimension gets a score and a threshold. The combined result is a decision: scale the pilot, remediate named gaps and re-score, or stop. The five steps below cover all seven dimensions: Step 1 sets scope, KPIs, and ownership; Step 2 scores data and integration; Step 3 covers security and governance; Step 4 covers cost; and Step 5 covers adoption.

An AI maturity assessment answers a different question: it profiles enterprise-wide capability rather than the readiness of one pilot. Gartner’s “AI Maturity Assessment,” published 29 April 2025 (Gartner, 2025), rates an organization from Level 1 (“planning/beginning”) to Level 5 (“leadership”) across seven capability areas, including strategy, value, governance, engineering, and data.

MITRE’s “AI Maturity Model” draws the same line by pairing five readiness levels with five separate assessment levels. Alation’s glossary frames readiness as whether the organization can deliver and sustain one specific AI use case now, so a high maturity score can coexist with a pilot that should not scale.

Why do AI pilots fail when they scale without a readiness gate?

At least half of generative AI projects stall between proof of concept and production, and analysts at Gartner, McKinsey, and MIT NANDA cite overlapping causes.

• Gartner (June 25, 2025): Gartner predicts that more than 40% of agentic AI projects will be canceled by the end of 2027 because of escalating costs, unclear business value or inadequate risk controls. Gartner restated the same forecast in October 2026.
• Gartner article (January 26, 2026): At least 50% of generative AI projects were abandoned after proof of concept because of poor data quality, inadequate risk controls, escalating costs or unclear business value. Gartner’s July 2024 release had forecast at least 30% abandonment after proof of concept by the end of 2025.
• McKinsey State of AI (August 25, 2026): 44% of respondents report AI scaling across their enterprise. In a separate measure, 37% attribute any EBIT impact to AI, essentially unchanged from a year earlier.
• MIT NANDA, The GenAI Divide (July 2025): Only 5% of integrated AI pilots extract millions in value, according to the report. The report is preliminary, rests on 52 interviews and 153 survey responses, and defines success as deployment beyond pilot with measurable KPIs six months on. Wharton’s Kevin Werbach and others have challenged how the headline figure was derived, so treat it as directional.

The MIT authors attribute failure to “brittle workflows, lack of contextual learning, and misalignment with day-to-day operations.” HFS Research reports that 67% of enterprises say context is fragmented across systems and 64% say it lives in people’s heads.

McKinsey’s 2026 high performers, the 6% who attribute 5% or more of EBIT to AI and report significant value from it, are nearly three times as likely as others to have redesigned workflows because of their AI use: nearly three-quarters versus one-quarter.

A pilot trained on documented procedures and clean system logs meets production work full of manual handoffs, spreadsheet workarounds, and judgment calls it never saw. A readiness gate exists to find that gap before the scale-up budget is committed. The final section describes how a Context Graph of Work, built from observed work, addresses that gap.

Step 1: Define scope, use case, success KPIs, and an accountable owner

Scoring starts only after four items are written down and signed. Gartner identifies lack of business value as the most fundamental failure mode: without clear prioritization frameworks or success metrics, projects cannot show measurable business value (Gartner, 2026). McKinsey’s 2026 survey finds high performers twice as likely as others to have defined processes to measure AI impact, and twice as likely to say senior leaders demonstrate commitment to AI initiatives (McKinsey, 2026).

Before scoring begins, the use-case charter should contain:

• Scope: The process, the case types in and out of scope, the volume at production scale, and the systems the AI will read from or act in.
• Success KPIs with a measured baseline: Cost per case, cycle time, error or rework rate, and compliance exceptions, measured on current operations before the pilot changes them.
• Executive sponsor: The COO or business-line leader who owns the operating outcome and funds the scale-up.
• Named production owner: One named individual is accountable for performance, incidents, drift, and decommissioning after go-live.

A pilot without a named production owner fails Step 1, regardless of its technical results. Without a pre-AI measurement of the process, no one can show the board whether the scaled system moved cost per case.

Step 2: Score data readiness and integration

Data readiness comes first in the scorecard because analysts flag it repeatedly as a scaling barrier. Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data, and its July 2024 survey of 1,203 data management leaders found 63% either lack or are unsure they have the right data management practices for AI (Gartner, 2025).

Gartner defines AI-ready data as data “representative of the use case, of every pattern, errors, outliers and unexpected emergence that is needed to train or run an AI model for a specific use.”

Clean data does not meet that definition on its own: a pilot often runs on the tidy subset of cases that flow through the system of record. Production includes the pended claim, the manual override, and the case rerouted by email. If the evaluation data does not contain those paths, the pilot score overstates readiness.

Data-quality vendor Soda suggests composite dataset health scores of at least 95 for financial reporting data and at least 90 for operational datasets. Those are vendor benchmarks for whole datasets.

The field-level pass rate below is our recommended threshold, informed by them; your team sets the final figure, higher for regulated processes:

• Quality: Critical fields pass completeness and accuracy rules at 95% or higher on a production-volume sample.
• Representativeness: The test set includes exception paths and variants in roughly their production frequency, and the team can show where that frequency came from.
• Lineage: Every critical data element traces to its source and transformations. Databricks notes that a dataset with high completeness and freshness but undocumented lineage does not qualify as AI-ready, because the team cannot confirm an undetected pipeline error has not contaminated it.
• Freshness: Data arrives within the window the use case needs, defined per use case.
• Monitoring: Drift and data-health checks run in production, with a named person who responds to alerts.

Integration and compute determine whether a working pilot can scale. Gartner’s 2025 agentic AI release warns that integrating agents into legacy systems is technically complex, often disrupts workflows, and requires costly modifications (Gartner, 2025), and McKinsey’s “State of Organizations 2026” names integration with legacy systems as one of the two largest barriers to scaling AI (McKinsey, 2026).

Pass criteria: every system in the production path is connected or has an approved execution route, including mainframe, Citrix, and VDI applications; rate limits and inference capacity are tested at peak production volume; and the fallback when an integration fails is documented.

Where the missing evidence is how work moves across applications that produce no usable logs, a Context Graph of Work (a context graph of people, applications, tasks, decisions, and policies built from observed work) is a way to supply it. The final section covers how.

Step 3: Check security, governance, and compliance readiness

Security and governance gaps surface late and end projects. A Gartner survey of 360 IT leaders in Q2 2025 found only 23% very confident in their ability to manage security and governance when deploying GenAI tools (Gartner, 2026). Gartner also issued a separate forecast on May 26, 2026 that counts enterprises, not projects: by 2027, 40% of enterprises will demote or decommission autonomous AI agents because of governance gaps they identify only after production incidents occur (Gartner, 2026), which means agents scaled back after reaching production, not projects canceled before production.

Oversharing is a documented pre-production risk for assistants and agents. Microsoft states that Copilot operates within existing permissions and access controls, so overshared or poorly governed content can affect its results and increase risk. Microsoft also identifies unresolved Everyone-except-external-users permissions as a top oversharing risk, and designs restricted content discovery as a temporary governance control, not a fix.

Require these sign-offs before a scale decision:

• CISO or security lead: Permission audit and remediation complete, sensitivity labels applied to grounding data, audit logging on, agent permissions scoped to least privilege.
• Chief risk officer or model risk: Risk tier assigned, human-in-the-loop points defined, override and incident response documented.
• Legal and compliance: Applicable regulations identified, data residency confirmed, required documentation in place.
• Data protection officer: Personal data processing assessed, retention and deletion rules set.
• Vendor risk: Certifications verified for the specific product in the pilot, not only the parent company. Certification scope often differs between a vendor’s platform and its individual products.

Map the use case to the frameworks below, using their current status as of October 2026:

Step 4: Validate cost, unit economics, and ROI

Cost is one of the three reasons Gartner gives for canceling agentic AI projects, and its January 2026 analysis is direct: “Projects that appear viable in proof of concept become budget black holes in production,” because organizations cannot see how costs scale (Gartner, 2026). Gartner forecasts that AI inference costs per agentic workflow will increase more than five times through 2028 (Gartner, 2026). An IDC survey of 318 decision makers, reported by CIO, found 92% said their deployed AI agents cost more than expected, with inference the most common cause.

Finance models run cost at full production volume. The full cost stack includes:

• Inference and token consumption per case, including retries, tool calls, and context re-reads in multi-step workflows
• Integration build and maintenance across every system in the production path
• Monitoring, logging, drift detection, and validation infrastructure
• Human review of outputs, exceptions, and escalations
• Licensing changes, including consumption metering
• Security, governance, and audit overhead

Express the result as a unit metric the CFO can evaluate: cost per resolved case or cost per completed workflow. Gartner uses this framing in its forecast that GenAI cost per resolution in customer service will exceed $3 by 2030. Run the model at expected volume and at a high-volume stress case, and show where unit cost crosses the current human or automated cost per case.

The CFO and COO set the ROI hurdle for AI scale decisions before scoring: a minimum net benefit per case against the measured baseline from Step 1, and a payback period the business accepts. A pilot that cannot clear the hurdle at production volume fails this dimension, whatever its accuracy.

Step 5: Assess adoption, skills, and change readiness

Adoption deserves its own score because usage figures overstate readiness. BCG’s “AI at Work 2026” survey of 11,749 workers across 14 markets found 74% of frontline employees are regular AI users, yet only 36% of all respondents feel they have received adequate upskilling, the same share as in 2025 (BCG, 2026). BCG cautions that adoption shows people use AI, not whether it pays off.

Gartner’s 1Q26 Global Labor Market Survey of 12,004 employees and managers across 40 countries found 19% of employees reported no time saved with AI, while employees proficient across multiple use cases were twice as likely to be highly productive (Gartner, 2026).

The constraint usually sits with the organization rather than the individual. CIO’s summary of Microsoft’s “2026 Work Trend Index” (CIO, 2026; Microsoft, 2026) reports that organizational factors such as culture, manager support, and talent practices accounted for more than twice the reported AI impact of individual effort. Gartner research found only 8% of HR leaders believe their managers have the skills to use AI effectively (Gartner, 2025).

For EU operations, Article 4 of the AI Act already requires providers and deployers to take measures supporting staff AI literacy, without mandating a specific level.

Score adoption on signals measured during the pilot:

• Weekly active use among the target roles, broken out by team
• Share of eligible cases handled with the AI
• Output acceptance, override, and correction rates
• Rework and error rates against the pre-AI baseline
• Time saved per case and where that time was redeployed
• Completed role-specific training and named manager accountability for the change

How do you score each dimension and set go/no-go thresholds?

Score each dimension on a five-level band, then apply hard thresholds. Martin Fowler’s guidance on agentic data readiness applies to the whole scorecard: “Start with a hard gate rather than a smooth composite.” A weighted average lets a strong model score hide an unowned system or an unresolved permission problem.

Use these maturity bands for every dimension:

Apply these minimum thresholds per dimension:

The decision rule:

• Scale: Every dimension is at Level 3 or above. Level 3 permits controlled scale within the defined scope. Move to full production volume only when every dimension reaches Level 4.
• Remediate: One or more dimensions sit at Level 2 with a fix that a named person can deliver in a set window. Re-score only the failed dimensions after the fix.
• Stop: Any dimension sits at Level 1, or the cost model cannot clear the ROI hurdle at production volume under any realistic scenario.

Gartner’s October 19, 2026 survey of 2,002 CIOs and technology executives found the organizations it classes as vanguards more likely to set stopping rules at two points: pilots and production (Gartner, 2026). Treat “stop” as a valid result of the gate.

Worked example: Scoring a pilot and remediating failed dimensions

A health insurer whose claims-triage pilot hit its accuracy target can still fail the data dimension, and the correct decision is then remediate rather than scale. In this illustrative example, which does not describe a specific organization, an AI agent triages incoming claims and routes them for adjudication, and the pilot hit its accuracy target on a test set drawn from the claims system. The process excellence lead runs the readiness assessment before the COO approves scale-up.

First scoring:

Data fails the threshold, so the decision is remediate. The failure also puts the cost score at risk, because the cost model assumed exception rates taken from the same incomplete data.

The remediation path has four actions. The team documents how pended claims are handled across applications, including the spreadsheet and email steps, and rebuilds the test set so exception paths appear at their observed frequency. It then re-runs the agent against the new set, recording where it escalates correctly and where it fails, and finally reruns the cost model with the observed exception rate and the added human review volume.

On re-scoring, the process excellence lead raises data to three because the team now tests production variants and documents lineage. The lead keeps cost at three with a lower margin above the hurdle, which the CFO accepts. Every dimension now meets the threshold, so the COO approves scaling, starting with the case types where escalation performance was strongest.

Who runs the assessment, how long does it take, and what does it cost?

The process excellence champion runs the assessment, the COO or business-line owner signs the scale decision, and the CFO governs the investment. Grant Thornton frames the split the same way: the COO owns whether AI translates into operating performance, and the CFO owns the investment case (Grant Thornton, 2026). Forrester predicted that in 2026, CEOs would lean more on CFOs to approve AI investments based on ROI (Forrester, 2026).

The core roles are:

• Process excellence champion: Runs scoring, gathers evidence, and maintains the scorecard.
• COO or executive sponsor: Signs the scale, remediate, or stop decision.
• CFO or finance partner: Approves the cost model and ROI hurdle.
• CIO or enterprise architecture: Owns integration and compute scoring.
• CISO, risk, legal, compliance, and privacy: Provide the Step 3 sign-offs.
• Production owner: Confirms accountability for operating the system after go-live.
• HR or change lead: Owns the adoption plan and training.

Published durations vary with scope: KPMG describes a 3–6 week incubate-to-test approach, a PwC AI in R&D deck shows a readiness assessment of about four weeks, and McKinsey describes a 6- to 12-week value-backed diagnostic for capital projects, while a third-party guide from Cabinco puts enterprise assessments at 4–8 weeks. Cabinco cites $40,000 to $120,000 for enterprise practitioner engagements at $1B+ companies and $150,000 to $500,000 for Big Four work aimed at board-level alignment and a multi-year program. Treat these as third-party ranges rather than published firm fees.

The assessment should produce these deliverables:

• A signed use-case charter with baseline KPIs, sponsor, and production owner
• A scored seven-dimension scorecard with evidence for each score
• A framework and regulatory mapping for the use case
• A production-volume cost model and ROI calculation approved by finance
• A remediation plan with owners and dates for any failed dimension
• A written scale, remediate, or stop decision signed by the COO

How does observed process context close readiness gaps?

Observed process context closes readiness gaps by supplying the evidence most readiness checks lack: exception paths for data, the measured baseline for Step 1 KPIs, and every application a process touches for integration. Skan AI is an agentic AI platform for regulated enterprises that builds a Context Graph of Work (a property graph of people, applications, tasks, decisions, and policies) from captured process activity, so banks, insurers, and healthcare organizations can ground agent decisions and execution in how work runs.

Capture runs through Skan Virtual Assistant (Skan VA), a lightweight desktop sensor that observes screens, clicks, application switches, and workarounds across legacy desktop, mainframe, Citrix, virtual desktop infrastructure (VDI), web, and office applications. It builds end-to-end process evidence with zero integrations, application programming interfaces (APIs), or logs, which covers work in systems that produce no usable event data. The analysis concerns work and processes, not individual workers, in keeping with privacy by design.

Skan PRISM, an on-premises gateway, performs pixel-level redaction of sensitive screen content before transmission. It is designed so that raw screen data stays inside the customer environment and only anonymized, abstracted metadata reaches analytics. That privacy by design approach (building privacy protections into the system from the start) adds application allow-listing (limiting capture to a list of approved applications), employee pseudonymization at collection (replacing names with codes), customer-defined retention, automated deletion, and right to erasure.

The products follow a single sequence on a shared Context Graph of Work:

• Skan AI Blueprint is a lighter opportunity scan that quantifies value and classifies work to agentify, augment, or keep human.
• Skan Intelligence then observes the identified areas in depth, with variant analysis, bottleneck and rework detection, and drift detection.
• Skan AI Agents execute work across UIs and APIs under agentic operating procedures (AOPs), machine-readable process definitions generated from observed behavior that encode steps, decision logic, exceptions, escalation thresholds, and success criteria.

Before production promotion, the team simulates and evaluates agents against human baselines, which gives the readiness gate a direct comparison point for the governance and adoption dimensions.

Skan AI deploys on-premises inside the customer firewall, private cloud, or Skan AI hosted, in a containerized platform (software packaged with everything it needs to run, so it behaves the same in any environment).

Skan AI holds System and Organization Controls (SOC) 2 Type 2 attestation and ISO/IEC 27001 and ISO/IEC 42001 certifications, and is compliant with the Health Insurance Portability and Accountability Act (HIPAA), the General Data Protection Regulation (GDPR), and the California Consumer Privacy Act (CCPA).

Initial actionable insight arrives in weeks, and observation requires no application changes or downtime. Desktop deployment still goes through your security review and endpoint rollout, and observed insight is an input to the readiness gate rather than proof of production readiness.

Ground your next scale decision in how the work runs. Request a Skan AI demo to explore how observed workflows become context for governed AI execution. To discuss requirements first, contact the Skan AI team about your workflow, deployment, and governance requirements.

These answers cover the questions COOs and process leaders raise most when they run a readiness gate.