What Should an AI Center of Excellence Own Once AI Agents Are in Production?

Most AI proofs of concept still do not reach wide-scale deployment, and the operating model of the AI center of excellence (CoE) is where the decisions that change that get made. The model decides where the CoE sits, what it owns, who staffs it, and how it governs, measures, and funds AI work. Once AI agents act on live cases, it also has to name who owns the audit trail, the case-level record an examiner asks to replay, and the authority to narrow or stop an agent’s autonomy.

What is an AI CoE operating model?

An AI CoE operating model is the set of decisions that tells everyone in the organization who does what with AI, from who owns it to who funds it. It is one part of a broader agentic AI operating model. It covers 6 things: the structure and archetype, the functions the CoE owns, the roles that staff it, the governance it enforces, the way it measures results, and how it is funded.

Two failure modes show why these decisions matter: pilot purgatory, where AI pilots stall before they ever reach production, and shadow AI. IDC research reported by CIO.com found that 88% of observed proofs of concept did not make it to widescale deployment (IDC, 2025), a gap IDC’s authors attributed to low organizational readiness in data, processes, and IT infrastructure, plus unclear return on investment and a lack of in-house expertise. S&P Global’s “Voice of the Enterprise” survey found that the average organization scrapped a large share of its proofs of concept before production (S&P Global, 2025).

A Gartner survey of 2,002 CIOs and technology executives, fielded May–July 2026, found that 13% report significant or transformational value from AI (Gartner, 2026). In the same survey, most CIOs lacked guidelines for when a pilot should not move to production and for abandoning low-value AI already in production. Kill criteria are an operating model decision.

The second is shadow AI, meaning employees and business units using AI tools the organization has not sanctioned. A Gartner survey of 302 cybersecurity leaders, published November 19, 2025, found that 69% of organizations suspected or had evidence that employees were using prohibited public generative AI tools (Gartner, 2025), and IBM’s “Cost of a Data Breach Report 2025” found that breaches at organizations with high levels of shadow AI cost more than those at organizations with low or no shadow AI (IBM, 2025). When the sanctioned path is slow, employees and business units build their own.

The six components of the model, and the question each one answers:

• Structure and archetype: Where the CoE reports and how much delivery it keeps centrally versus in business units.
• Functions: Which activities the CoE owns, such as policy, platform standards, intake and portfolio measurement.
• Roles: Who leads the CoE and which specialist roles it staffs directly.
• Governance: How policy ties into enterprise risk management and regulation, including the EU AI Act.
• Measurement: Which KPIs and SLAs hold the CoE and the business units accountable.
• Funding: Whether the CoE runs on a central budget, chargeback, showback or a mix.

What are the 4 AI CoE operating model archetypes?

Analysts use the same four labels inconsistently, so this article separates hub-and-spoke (one central hub with embedded spokes) from hybrid (different capabilities set at different levels of centralization), because the distinction changes who owns what. Gartner’s abstract on AI governance models treats hybrid and federated as one option, while McKinsey defines hybrid as some resources centralized and some distributed across functions. The table sets out the four archetypes on that basis:

The firms disagree on the default: Forrester’s reports on data, analytics, and AI operating models say neither centralized nor decentralized approaches strike the right balance and advocate a federated model, while Gartner’s 2024 “AI Mandates for the Enterprise Survey” found that most organizations at all AI maturity levels prefer a centralized model (Gartner, 2024). Deloitte’s 2026 banking and capital markets outlook says a hub-and-spoke model could be the optimal choice for most banks, anchored by a central unit such as an AI center of excellence (Deloitte, 2026), and BCG’s retail banking perspective recommends starting with a centralized technology CoE and moving to a more federated structure for function-specific initiatives (BCG, 2026). These positions are less contradictory than they look: they describe different points on the same path.

Which archetype fits your size and AI maturity?

The strongest evidence says centralization should vary by capability. McKinsey’s March 2025 “State of AI” report (Exhibit 1, 1,229 respondents whose organizations use AI) shows how far placement differs across five of the six capabilities it tracks (McKinsey, 2025):

• Risk and compliance: 57% fully centralized, 30% hybrid, 13% fully distributed
• Data governance for AI: 46% centralized, 39% hybrid, 15% distributed
• AI strategy: 36% centralized, 48% hybrid, 16% distributed
• Road map for AI products: 35% centralized, 44% hybrid, 21% distributed
• Tech talent: 29% centralized, 49% hybrid, 22% distributed

The same report found that respondents at organizations with less than $500 million in annual revenue are more likely to fully centralize tech talent and adoption.

The decision matrix below is our judgment, built from the evidence above. It uses three triggers: maturity stage (using Gartner’s AI Maturity Model stage names), number of active use cases, and business unit talent density, meaning how many business units employ their own AI and machine learning engineers. The use case thresholds align with a third-party AI Advisory Practice white paper, which lists scaling beyond 15 to 20 active projects as a weakness of the centralized hub:

For banks, insurers, and healthcare organizations at the Operational stage or later, we recommend hub-and-spoke with risk and data governance fully central. That matches Deloitte’s banking guidance and the McKinsey placement data, and it keeps regulatory accountability in one place while delivery moves closer to the work. Deloitte’s bank marketing research adds a size cut: smaller banks may benefit from centralization, while larger banks operating multiple business lines across jurisdictions may opt for hub-and-spoke.

What does the CoE own versus the business units?

The CoE owns the rules, the shared platform standards, and the portfolio view. Business units own the use case, the prompts and agent instructions for their processes, and the outcome. A practitioner in a Gartner Peer Community thread put it in 1 line: the business owns the outcomes while the center provides the platforms, standards, and controls that let AI scale responsibly. The matrix below spreads that split across 4 functions, with 1 accountable (A) party per row:

Three rows tend to get contested.

Prompts and agent instructions belong to the business unit’s workload team. Microsoft’s Cloud Adoption Framework guidance on agent readiness assigns workload teams the end-to-end lifecycle of specific agents, including business requirements, domain data, and conversation flows. The CoE keeps a reusable library and standards; Deloitte describes advanced companies running a hub team of prompt engineers with business units as spokes.

Policies sit with the CoE, but risk approves the classification. Deloitte’s risk guidance warns against relying solely on the AI CoE for risk management, because risk work gets deprioritized when central AI resources are spread across business units. Risk leaders need an independent view.

Outcomes sit with the business unit. The CoE measures outcomes; the business unit answers for them.

How should the CoE govern AI risk and responsible AI?

The CoE should assign policy and regulatory obligations to named owners, then extend oversight to production-agent actions through audit trails, case-level playback, continuous controls monitoring, and human escalation.

The CoE should write AI policy and connect it to enterprise risk management through the three lines of defense: business units manage risk in the first line, risk and compliance provide frameworks and oversight in the second, and internal audit gives independent assurance in the third. AWS describes the same structure in its enterprise risk management guidance for AI.

Two frameworks give the CoE its role definitions. NIST AI RMF 1.0 calls for documented and clear roles, responsibilities, and lines of communication for mapping, measuring, and managing AI risk (GOVERN 2.1), and for executive leadership to take responsibility for decisions about AI risk (GOVERN 2.3). ISO/IEC 42001:2023 sets requirements for an AI management system built on Plan-Do-Check-Act.

Does ISO/IEC 42001 certification give presumption of conformity under the EU AI Act?

ISO/IEC 42001 is an international standard, not a harmonised European standard, so it confers no automatic presumption of conformity under Article 40. The European Commission states that EN 18286, the AI quality management standard mapped to Article 17, is the only AI quality management standard with a presumption of conformity, and that it is designed to layer on top of ISO/IEC 42001 rather than replace it. It was published in the Official Journal as Commission Implementing Decision (EU) 2026/2274, and a CoE that already runs an ISO/IEC 42001 management system should treat EN 18286 as the next layer.

How did the Digital Omnibus changed the EU AI Act timeline?

The EU AI Act timeline changed in 2026, when Regulation (EU) 2026/1744, the Digital Omnibus on AI, was published in the Official Journal on July 24, 2026 and entered into force on July 27, 2026. Most prohibited practices and the AI literacy duty have applied since 2 February 2025, and the new prohibitions on AI systems that generate non-consensual intimate material or child sexual abuse material apply from December 2, 2026. The amended Article 4 requires providers and deployers to take measures that support staff AI literacy, but it does not require them to guarantee any specific level of literacy for any individual, and general-purpose AI model obligations applied from August 2, 2025.

High-risk obligations for Annex III systems, which cover areas including employment decisions, now apply from 2 December 2027, and those for Annex I product-embedded systems from August 2, 2028. A Latham & Watkins deployer guide lists credit checks and credit scores among high-risk uses. That puts lending and HR use cases in scope for banks and insurers.

Each obligation needs a named owner in the operating model:

Two banking details matter: under Article 26(5), financial institutions subject to internal-governance rules under EU financial services law meet the deployer monitoring obligation by complying with those rules, and they keep logs as part of that documentation. In the US, SR 26-2 (April 17, 2026) replaced SR 11-7 and SR 21-8 (Federal Reserve, 2026), and Federal Reserve Vice Chair for Supervision Bowman stated on May 1, 2026 that the revised model risk guidance “does not apply to generative or agentic AI.” (Federal Reserve, 2026) Banks therefore cannot route agent governance through the model risk process by default; the CoE has to define the alternative.

What changes once agents are in production?

Agents shift governance from approving models to supervising actions. Gartner’s May 26, 2026 release predicts that by 2027, 40% of enterprises will demote or decommission autonomous AI agents because of governance failures, most often when organizations fail to distinguish between an agent’s ability to act and the scope of access it is granted (Gartner, 2026). Gartner recommends tiering governance by autonomy level instead of applying uniform controls, and its October 2, 2026 article names the biggest risk as deploying agents with more autonomy than their reliability supports (Gartner, 2026).

The joint guidance from CISA, NSA, Australia’s ACSC, and partner agencies (listed by NSA on April 30, 2026) adds that agents expand the attack surface and bring privilege creep, behavioral misalignment, and hard-to-read event records, and recommends never granting broad or unrestricted access and starting with low-risk tasks.

Four controls move into the CoE’s remit once agents run on live work:

• Audit trail: Every input, output, action and decision needs a record the agent cannot alter. A NIST Cybersecurity Insights post calls for a tamper-evident, append-only chain of evidence for an agent’s identity, delegated intent and actions, with logging independent of the agent. OWASP’s AIVSS v0.5 notes that when agents act across cloud, on-premises and SaaS systems without correlation IDs, the chain of custody breaks.
• Case-level playback: An examiner or internal auditor asks about one claim, one loan or one prior authorization. The CoE needs to reconstruct what the agent saw and did for that case, including the rationale, not just aggregate metrics.
• Continuous controls monitoring: Forrester’s State of Agentic AI in 2026 says governance requires instrumentation that runs while the agent does, rather than quarterly reviews. The FCA’s October 19, 2026 good-practice findings name multidisciplinary governance, backed by continuous monitoring, as a feature of firms that deploy AI safely.
• Governed autonomy with human-in-the-loop: The FCA findings also point to clearly designed human oversight and escalation triggers. CIO.com gives a concrete pattern: agents process refunds up to $500 autonomously and route anything higher to human review.

These controls depend on knowing how the process runs before an agent touches it. Skan AI is an agentic AI platform for regulated enterprises that builds a Context Graph of Work from observed process activity, so banks, insurers, and healthcare organizations can ground agent decisions and execution in how work runs. Work Context is what we understand. The Context Graph of Work is what we build. The Skan Virtual Assistant (Skan VA) is built on privacy by design and observes work across mainframe, Citrix, virtual desktop infrastructure, web, and office applications with zero integrations, APIs, or logs, while Skan PRISM redacts sensitive on-screen content at pixel level on premises before transmission. Skan AI Blueprint, a lighter opportunity scan, identifies where to agentify, augment, or keep work human. Skan Intelligence observes those areas in depth, and Skan AI Agents execute work under agentic operating procedures (AOPs), with audit trails, case-level playback, and continuous controls monitoring. Skan AI deploys on-premises inside the customer firewall, private cloud, or Skan AI hosted, holds SOC 2 Type 2 attestation and ISO/IEC 27001 and ISO/IEC 42001 certifications, and is compliant with the Health Insurance Portability and Accountability Act (HIPAA), the General Data Protection Regulation (GDPR), and the California Consumer Privacy Act (CCPA).

Which roles does the CoE need, and who should it report to?

The CoE needs four named roles before it needs headcount. Each maps to one or more rows of the ownership matrix:

• Head of AI CoE: Owns the AI CoE operating model itself, the intake process, the portfolio and the policy set, and is the single point of contact for AI strategy, as Microsoft’s Cloud Adoption Framework describes the CoE leader.
• AI Governance and Risk Lead: This CoE role prepares each use case’s risk classification for second-line risk and compliance to approve, maintains the EU AI Act owner map, runs fundamental rights impact assessments, and serves as the CoE’s link to second-line risk. BCG reports that a Fortune 100 company created a responsible AI lead role when it set up its AI CoE.
• MLOps / LLMOps engineering lead: Owns deployment pipelines, versioning, rollback, monitoring standards and technical documentation for in-house systems. CIO.com’s 2026 State of the CIO coverage says most organizations now look for engineers who can stand up agents, build testing frameworks and manage the cost-latency-quality trade-off.
• Change and Adoption Manager: Owns training, workforce communication and adoption metrics. Deloitte’s AI infrastructure survey notes that success often lies in reliable adoption in day-to-day workflows rather than deployment, and Gartner’s March 16, 2026 release reports that 78% of 110 CHROs surveyed agree workflows and roles must change to get value from AI.

Team size estimates come mostly from third-party guides: the COMPEL Framework suggests a core team of 5-15, BCG describes a Fortune 100 company that built its AI CoE with more than 30 new full-time specialists, and a separate BCG agentic case has a CoE of about 50 people supporting 100 local deployment teams. Hiring will be the constraint. ManpowerGroup’s “2026 Talent Shortage Survey” of 39,000 employers in 41 countries found 72% report hiring difficulty, with AI model and application development and AI literacy the hardest capabilities to find (ManpowerGroup, 2026).

The Chief AI Officer (CAIO) question has moved fast. IBM’s “2026 CEO Study” (2,000 CEOs, February–April 2026) found 76% of surveyed organizations have a CAIO, up from 26% a year earlier (IBM, 2026). The Data & AI Leadership Exchange’s 2026 benchmark of senior data and AI leaders at Fortune 1000 companies found that 38% had appointed a CAIO or equivalent and no consensus on where the CAIO reports (Data & AI Leadership Exchange, 2026), while Foundry’s “2025 State of the CIO” survey found that among organizations with a CAIO, 40% of CAIOs report to the Chief Executive Officer (CEO) and 24% to the Chief Information Officer (CIO) (Foundry, 2025).

We recommend a reporting line that sits within one step of the executive who owns the value. AWS’s Cloud Adoption Framework for AI says CoE reporting lines should align with the stakeholders who own the AI strategy, with short paths to the C-suite. McKinsey’s analysis found CEO oversight of AI governance is one of the elements most correlated with higher self-reported bottom-line impact.

In banking, insurance, and healthcare operations, where value lands in claims, servicing, and processing costs, that often means the Chief Operating Officer (COO) or Chief Transformation Officer, with a dotted line to the CIO for platform. A CAIO helps when AI spans more functions than any one executive controls; without one, the Head of AI CoE needs a sponsor at that level.

US federal agencies offer a reference point: Office of Management and Budget (OMB) memorandum M-25-21 requires each agency to designate a CAIO and allows the role to go to an existing CIO, chief data officer (CDO), or chief technology officer (CTO) (OMB, 2025).

How should you fund the AI CoE: central budget or chargeback?

Executives should align funding with the archetype and adjust it as delivery responsibilities change. A central budget fits a centralized CoE proving its first use cases; chargeback fits federated delivery, where business units choose and pay for their own work. The mapping below combines FinOps Foundation guidance with general CoE funding practice:

The FinOps Foundation’s maturity table for AI spend moves from no allocation at Crawl to showback to teams at Walk and chargeback with team budgets at Run, which maps onto the centralized-to-federated path. McKinsey’s public-sector analysis argues that central IT cannot carry AI costs alone once AI makes up 30 percent of IT costs, and calls for chargeback or showback across the portfolio.

The CoE must allocate agent costs by use case. Gartner predicts in its August 17, 2026 release that AI inference costs (the cost of running a model to produce each response) per agentic workflow will increase more than fivefold through 2028 (Gartner, 2026). In its October 26, 2026 release, Gartner predicts that through 2027 enterprises deploying IT operations agents on uncapped token-based pricing (billing by the volume of text a model processes, with no spending ceiling) will exceed initial cost expectations by more than 50% compared with organizations using predictable subscription and bundled pricing, and recommends a “no telemetry, no deployment” policy (Gartner, 2026).

The FinOps Foundation adds that an invoice from an AI model provider shows spend by API key (the credential an application uses to call the model) or project, not by business unit or cost center, so the CoE has to build the attribution layer itself, meaning the way each cost is assigned to its owner. Whatever the model, the CoE should own the tagging standard (the labels that tie each cost to a use case) and the token budget per use case, and the business unit should own the bill.

Which KPIs and SLAs should measure the CoE?

The CoE should run against SLAs like any accountable service unit, with value metrics owned jointly with the business units. In its September 1, 2026 release on AI scaling, Gartner’s Tina Nunno warns that organizations without disciplined measurement tied to business outcomes risk wasted resources and unmet expectations (Gartner, 2026). CIO.com’s “2026 State of the CIO” found only 47% of respondents have formal AI metrics (CIO.com, 2026).

The scorecard below combines delivery, quality, security, cost, and value measures. Thresholds marked as heuristics come from vendor documentation and should be tuned to each domain:

CIO.com’s account of high-maturity CoEs groups measures into five dimensions: Productivity, Operations, Quality, People, and Trust. HFS Research found in March 2026 that only 21% of the 101 Fortune 200 C-suite executives surveyed could say with full confidence that AI is creating value (HFS Research, 2026). The value row matters most for the budget conversation: a CoE that reports verified value per use case, with a pre-AI baseline captured at intake, gives the CFO evidence of that value.

How should the CoE roll out over time?

The CoE should start centralized and hand delivery outward as maturity rises. Microsoft’s guidance says most organizations start centralized and evolve toward an advisory role once platform teams can enforce governance automatically. McKinsey describes CoEs that begin with a centralized model, with some development capacity embedded in functions, and move to a federated model as the AI program matures.

The time boxes below are our proposal. A third-party Rework guide places the move to federated or hub-and-spoke at months 12–24. Microsoft’s internal AI CoE describes a 3-year timeline.

1. Centralize and prove (months 0–6): Appoint the Head of AI CoE and the AI Governance and Risk Lead, publish the policy and risk classification process, stand up intake with kill criteria, and take three to five use cases to production with measured baselines.
2. Hub-and-spoke (months 6–18): Embed engineers or product owners in the two or three business units with the most demand, move prompts and agent instructions to those workload teams, introduce showback, and add the audit trail and playback standards before any agent goes live.
3. Federated enablement (months 18–36): Shift business units to chargeback, let them run delivery against their own P&L, and narrow the CoE to policy, platform standards, assurance support and portfolio value tracking, which is the role BCG describes for the hub in its business unit-led phase.

Practitioners can often name four ways CoEs fail along this path:

• Ivory tower: The CoE builds AI disconnected from the work. HFS Research’s consumer goods analysis says neither isolated ivory towers of innovation nor decentralized hubs work, and places success somewhere in the middle.
• Order taker: The CoE builds whatever business units request. The US GSA AI CoE guide calls this “perhaps the most common pitfall”: a central group of practitioners that does AI work for mission areas or program offices on request. Forrester’s advice from customer insights CoEs applies: engage business partners, but do not become the order-taker.
• Gatekeeper: The CoE reviews every request and becomes the bottleneck. Microsoft’s Cloud Adoption Framework recommends replacing a gatekeeper model that blocks work with an advisory group that sets guardrails.
• Sprawl: Business units deploy isolated copilots and agents outside the CoE. CIO.com identifies the rapid proliferation of shadow AI as a major failure pattern that brings duplicate investment and compliance risk.

Map your agent candidates against observed work before you assign who owns them. Request a Skan AI demo to explore how observed workflows become context for governed AI execution.