Skip to main content
We've raised $63M to give enterprise AI the context it's been missing.
Learn more
Blog post22 mins

Deploying Enterprise AI On-Premises for Regulated Operations: A Phased Framework

Regulated enterprises can deploy enterprise AI on-premises in seven phases, from placement and control mapping to drift monitoring and total cost of ownership.
Skan Editorial Staff
deploy-enterprise-ai-on-premises-regulated-operations

A regulated enterprise deploys AI on-premises in seven ordered phases. It decides placement (on-premises, single-tenant VPC, or air-gapped) against HIPAA, GDPR, the EU AI Act, FedRAMP, CMMC, and ITAR, and it maps each requirement to identity, encryption, logging, network boundary, and model registry controls, with the evidence assessors request. It sizes GPU memory from model size, precision, context length, and concurrency; it then builds the serving stack, operates the air gap, governs access and audit, and pilots and monitors drift; finally, it tests the plan against a 3-to-5-year total cost of ownership.

The regimes named above are the Health Insurance Portability and Accountability Act (HIPAA), the General Data Protection Regulation (GDPR), the Federal Risk and Authorization Management Program (FedRAMP), the Cybersecurity Maturity Model Certification (CMMC), and the International Traffic in Arms Regulations (ITAR).

Gartner’s survey of 2,002 (CIOs), published October 19, 2026, found only 13% say their enterprise has realized significant or transformational value from AI (Gartner, 2026). In Skan AI’s view, in banking, insurance, and healthcare, an architecture that cannot produce control evidence widens the gap the survey describes, because the system stalls at the examiner before it delivers any return.

We recommend assigning each phase to one accountable owner. Compliance and legal own placement (Phase 1); security architecture owns control mapping (Phase 2) and the air gap (Phase 5); infrastructure and facilities own hardware sizing (Phase 3); platform engineering owns the serving stack (Phase 4); model risk management and internal audit own governance (Phase 6); and machine learning operations (MLOps) and the business process owner own pilots and monitoring (Phase 7). The executive sponsor must decide:

  • Which workload classes must stay inside the enterprise’s own boundary, because that placement sets the controls and hardware for every later phase.
  • Whether expected utilization justifies owned hardware against reserved cloud or token billing over a 3-to-5-year horizon.
  • Whether to staff GPU and MLOps operations in-house.

The phased framework for on-premises AI in regulated operations

Architects carry constraints from each phase into the next: the placement decision identifies applicable controls, the controls define the hardware boundary, and operators then transfer patches and models across that boundary.

  1. Decide placement: Choose on-premises, single-tenant VPC or air-gapped hosting based on what HIPAA, GDPR, the EU AI Act, FedRAMP, CMMC and ITAR require.
  2. Map controls: Tie each regulatory requirement to identity, encryption, logging, network boundary and model registry components, along with the evidence assessors will request.
  3. Size the hardware: Calculate GPU memory from model size, precision, context length and concurrency, then size networking, storage, rack power and cooling.
  4. Build the serving stack: Run Kubernetes with a serving engine and an open-weight model whose license fits your use.
  5. Operate the air gap: Mirror, scan, sign and transfer artifacts without opening the boundary.
  6. Govern access and audit: Configure RBAC, identity, segmentation, audit trails and model risk management so the same records serve as compliance evidence.
  7. Pilot, promote and monitor: Promote models through evaluation gates and track drift and lineage inside the regulated boundary.

Total cost of ownership and staffing apply across all seven phases.

Phase 1: Decide placement (on-prem, single-tenant VPC or air-gapped)

Few regulations require on-premises hosting outright. The case for on-premises usually comes from a single specific condition: the model must read regulated data in clear text, and the regime does not allow a third party to see that data. We last verified the regulatory dates in this section on October 29, 2026. The deployment spectrum runs as follows:

Deployment modelWho controls infrastructure and keysTypical fit in regulated operations
Multi-tenant SaaS or APIProviderHIPAA workloads under a business associate agreement; FedRAMP-certified services for federal data
Single-tenant dedicated cloud or VPCProvider runs hardware; customer may hold keysData residency and isolation requirements where the provider’s legal exposure is acceptable
Customer-hosted on-premisesCustomerClear-text processing that a regime keeps inside your own boundary, such as ITAR technical data or GDPR transfers with no effective safeguard
Air-gapped on-premisesCustomer, with no network path outDisconnected or classified environments, and sites that require approved cross-domain transfer

Does HIPAA require on-premises hosting for AI?

Many teams assume HIPAA requires on-premises hosting for AI that touches protected health information (PHI). No. The U.S. Department of Health and Human Services (HHS) requires a business associate agreement (a contract that binds the provider to safeguard the data) with any cloud service provider that creates, receives, maintains, or transmits electronic PHI (ePHI). The Office for Civil Rights (OCR) has entered a resolution agreement with a covered entity that stored ePHI for more than 3,000 individuals on a cloud server without one. A provider that has no encryption key to the ePHI is still a business associate. The January 2025 Security Rule notice of proposed rulemaking (NPRM) would make encryption mandatory. According to Clark Hill and HIPAA Journal, the Fall 2026 Unified Agenda moved the rule to long-term actions, with final action targeted for July 2027.

Does GDPR require EU storage or on-premises hosting?

No, GDPR regulates transfers rather than storage location. Transfers outside the European Economic Area (EEA) need a Chapter V mechanism. After Schrems II, an exporter has to assess case by case whether the importing country’s law undermines that mechanism. In the consultation version of Recommendations 01/2020, the European Data Protection Board (EDPB) listed transfers to cloud providers “which require access to data in the clear” (Use Case 6) among the scenarios where it found no effective supplementary measure.

For inference that requires clear-text access, assess whether the provider and transfer fall within this scenario. The General Court upheld the EU-US Data Privacy Framework on September 3, 2025, but the appeal in Case C-703/25 P was still listed as pending in September 2026.

Does the EU AI Act set hosting requirements?

The Act contains no hosting-location provisions. Regulation (EU) 2026/1744, the Digital Omnibus on AI, entered into force on July 27, 2026 and set December 2, 2027 for Annex III high-risk obligations and August 2, 2028 for Annex I. Article 50 transparency duties have applied since August 2, 2026; for generative systems placed on the market before that date, the Omnibus allows until December 2, 2026 to meet the Article 50(2) marking obligation.

When do FedRAMP and CMMC apply to AI hosting?

FedRAMP’s authorization playbook states that cloud services holding federal data must be FedRAMP authorized. A legacy FedRAMP page exempts private clouds built for a single organization entirely within federal facilities; treat that exemption as legacy guidance until FedRAMP restates it under the 2026 Consolidated Rules. FedRAMP will stop accepting new Rev5 certifications on June 11, 2027. Under the CMMC rule, a cloud provider that processes controlled unclassified information (CUI) must meet FedRAMP Moderate or equivalent requirements. The U.S. Department of Defense (DoD) suspended the CMMC Phase 2 transition on July 13, 2026, and Phase 1 self-assessments remain in force.

Can ITAR technical data be processed in the cloud?

ITAR technical data can be stored in the cloud under the encryption conditions below, but a model cannot process it in clear text outside an authorized security boundary. Under 22 CFR 120.54(a)(5), storing unclassified technical data is not an export if the data is end-to-end encrypted with Federal Information Processing Standards (FIPS) 140-2 (or successor) modules, no third party holds the decryption means, and the data is not stored in a §126.1 country or Russia.

The 2019 Directorate of Defense Trade Controls (DDTC) preamble requires the data to stay “continuously encrypted at all times while outside of an authorized security boundary.” (DDTC, 2019) DDTC also rejected a request to allow clear-text processing such as antivirus scanning. Because inference has to decrypt the data, the model must run inside a boundary authorized for that data.

When does a hybrid model with an AI gateway fit better than pure on-premises?

Phase 1 shows that regimes split workloads. HIPAA workloads under a business associate agreement and GDPR transfers with a valid Chapter V mechanism can use a dedicated cloud or an API, while clear-text workloads, such as ITAR technical data or EDPB Use Case 6 transfers, stay inside your boundary. A hybrid design could keep restricted workloads on-premises and send the rest to cloud or API endpoints, with an AI gateway inside the boundary routing each request by data classification.

MLflow, an Apache 2.0 platform hosted by the Linux Foundation, can be self-hosted and includes an AI Gateway among its open-source core capabilities, making it an example of a gateway that runs inside the boundary. Below the 3.7 to 14.9 billion tokens a month break-even range that a 2026 arXiv analysis reports under its own assumptions (arXiv, 2026), API billing tends to cost less than owned hardware. The total cost of ownership section covers the math.

Phase 2: Map regulatory controls to the architecture

The crosswalk below groups the cited requirements by component so architects and assessors work from the same table:

Stack componentHIPAA (45 CFR 164.312)NIST SP 800-53 Rev. 5 (FedRAMP)NIST SP 800-171 (CMMC)ITAR 120.54Evidence assessors expect
IdentityUnique user identification (a)(2)(i); person or entity authentication (d)AC-2 account management; AC-6(9) log use of privileged functions; phishing-resistant access per RFC-000303.05.03 MFA for privileged and non-privileged accounts (Rev. 3); CMMC still assesses against Rev. 2Unencrypted data available only to the originator, U.S. persons in the U.S. or authorized recipientsAccount inventory, access reviews, MFA configuration
EncryptionEncryption and decryption (addressable) under access control and transmission securitySC-8, SC-12, SC-13, SC-28(1)3.13.8 and 3.13.11 FIPS-validated cryptography outside the protected environment; 3.13.16 CUI at restEnd-to-end encryption, FIPS 140-2 or successors or AES-128-comparable strength, keys never with a third partyCMVP validation certificates, key management procedures, FedRAMP cryptographic modules table
LoggingAudit controls (b), with regular review of activity recordsAU-2, AU-6, AU-9(3) cryptographic protection of audit information, AU-1203.03.01 event logging (Rev. 3)Not specified in 120.54Audit logs, review records, event data
Network boundaryTransmission security (e)(1)AC-4 information flow, SC-7 boundary protection3.13.1 monitor and control external and key internal boundariesNo intentional storage in or transfer from §126.1 countries or RussiaAuthorization boundary and data flow diagrams
Model registry and configurationIntegrity (c)(1) for ePHI; no model-specific requirementCM-2 baseline configuration; NIST COSAiS overlays for model weights still in developmentNo model-specific requirementNot addressedIntegrated inventory workbook, ML-BOM, model inventory

FedRAMP packages are built on templates for the system security plan (SSP), security assessment plan (SAP), security assessment report (SAR), and plan of action and milestones (POA&M), along with an integrated inventory workbook and a cryptographic modules table.

FedRAMP’s 2026 assessor guidance states that “Documentation explains what to inspect, but it cannot prove that a measure exists or works.” (FedRAMP, 2026) The same guidance favors configuration data, event logs, and complete populations of automated checks over samples. The CMMC Level 2 guide says that using an approved algorithm is not sufficient because the cryptographic module itself must be validated, and NIST’s CMVP accepts modules validated to either FIPS 140-2 or FIPS 140-3.

Phase 3: Size GPU hardware, networking, storage and facilities

Graphics processing unit (GPU) memory is the binding constraint, and it equals the weight memory (the space the model’s parameters occupy), plus key-value (KV) cache per concurrent request (the stored attention state for each active request), plus 15% to 20% overhead. Weights take 2 bytes per parameter at 16-bit floating point (FP16), 1 byte at 8-bit floating point (FP8) or 8-bit integer (INT8), and 0.5 bytes at 4-bit integer (INT4); the KV cache per token is calculated as 2 × layers × KV heads × head dimension × bytes per value. The table below gives Hugging Face’s published figures for Llama 3.1:

ModelFP16 weightsFP8 weightsINT4 weightsKV cache at 128k tokens, one request
8B16 GB8 GB4 GB15.62 GB
70B140 GB70 GB35 GB39.06 GB
405B810 GB405 GB203 GB123.05 GB

The following worked examples use an 80 GB H100:

  • 8B class: RunPod’s example of Llama 3.1 8B serving 50 concurrent users at 4,096-token sequences in BF16 comes to about 47 GB including overhead. That fits on one H100.
  • 70B class: At FP8 on two H100s, 160 GB minus 70 GB of weights leaves 90 GB for cache. That holds about two 128k-token requests, before overhead. Red Hat reports that vLLM’s FP8 KV cache can roughly double the number of concurrent requests.
  • 405B class: The vLLM project runs the FP8 checkpoint on a single 8×H100 node with tensor parallelism of 8. NVIDIA NIM 1.5.0 lists 16 H100s for the FP16 latency profile. Dell reports that one PowerEdge XE9680 with eight H100s supported 68 simultaneous chatbot users on Llama 3.1 405B, but it does not state context length or precision.

Facilities usually set the rack layout before the GPUs do. An H100 SXM draws up to 700 W. A DGX H100 (NVIDIA’s 8-GPU server) draws 10.2 kW maximum in 8U (8 rack units of height) and weighs up to 130.45 kg. NVIDIA’s SuperPOD guide allows a maximum of 2 DGX H100 systems per rack on single-phase 230 V, 63 A circuits (13.7 kW circuit capacity at 100% derating), with 10.2 kW of peak server demand per circuit, and a maximum of 4 per rack on three-phase 208 V, 60 A circuits (32.8 kW circuit capacity at 80% derating), with 20.4 kW of peak server demand per circuit. The 20.4 kW figure is per circuit, not per rack: 4 systems at 10.2 kW each peak at 40.8 kW.

Uptime Institute puts the point where direct liquid cooling becomes more economical at 20 to 25 kW per rack, and Schneider Electric says air cooling becomes inefficient at 25 to 30 kW. Successor systems move past that threshold: a DGX B300 draws 14.5 kW, and a GB300 NVL72 rack requires up to 142 kW with full liquid cooling. Lower-density sites can use the air-cooled H100 NVL PCIe card at 350 to 400 W.

For networking, NVIDIA positions Quantum-2 NDR InfiniBand at 400 Gb/s for Hopper deployments and Quantum-X800 at 800 Gb/s per port for new large builds, with Spectrum-X Ethernet as the alternative, carrying remote direct memory access (RDMA) over Converged Ethernet (RoCE). NVIDIA’s Dynamo documentation states that aggregated deployments, where prefill and decode run in one worker, need no RDMA, but disaggregated serving across nodes does need it, because without it the KV transfer falls back to standard Transmission Control Protocol (TCP) networking and can dominate time to first token. GPUDirect RDMA requires the GPU and network interface card (NIC) to share a Peripheral Component Interconnect Express (PCIe) root complex, and NVIDIA recommends DMA-BUF over the legacy nvidia-peermem module.

For storage, GPUDirect Storage (GDS) opens a direct memory access (DMA) path, which moves data without routing it through the host processor, from NVMe (fast solid-state storage) or NVMe over Fabrics (NVMe-oF) to GPU memory. NVIDIA’s H100 SuperPOD reference specifies at least 40 GBps of storage bandwidth to each DGX H100.

Phase 4: Build the serving and orchestration stack

Kubernetes (software that schedules and runs containers across servers), Helm (a package manager for Kubernetes), and container images (self-contained software bundles) form the base layer, and serving engines, the software that runs models and answers requests, run on top. The versions below were last verified on October 29, 2026. Recheck each release before you pin it in production:

ComponentCurrent releaseLicense and governanceNote for regulated use
vLLM0.32.0 (October 18, 2026)Apache-2.0; PyTorch Foundation hosted projectVersions before 0.14.1 are deprecated over CVE-2026-22778, a remote code execution flaw
NVIDIA Dynamo-Triton (formerly Triton Inference Server)2.73.0, 26.09 containerBSD-3-ClauseThe 26.09 containers add initial Rubin support
TensorRT-LLMv1.4.0rc1, pre-releaseApache 2.0Pin a stable release for production
NVIDIA NIMRelease notes updated October 6, 2026NVIDIA Software License AgreementNIM Certified Production Branch requires an NVIDIA AI Enterprise subscription; non-certified NIM is available at no charge under applicable license terms
KServev0.21.0Apache-2.0; CNCF incubatingCanary rollout works only in serverless mode

Model licenses determine more than cost:

FamilyRecent open-weight releaseLicenseFlag for regulated buyers
Meta LlamaLlama 4 Scout and Maverick (April 2025)Llama 4 Community LicenseSeparate license above 700 million MAU; “Built with Llama” attribution; multimodal rights not granted to companies with a principal place of business in the EU
Meta MuseMuse Spark Open (October 15, 2026), 816B total, 42B activeMuse Model LicenseModel card recommends deployment within a broader system with additional guardrails
MistralMistral Large 3 and Small 4Apache 2.0Review the license terms for commercial use
MistralMedium 3.5 (April 28, 2026)Modified MITMistral’s help center says companies above $20M monthly revenue need a commercial license
MistralLarge 4 (1.05T total parameters)Mistral Ecosystem Open-Weight licenseModel-as-a-Service operators need a separate license
QwenQwen3 and Qwen3.5Apache 2.0Qwen2.5-72B carries a 100 million MAU clause; each Qwen repository ships its own license file

For retrieval-augmented generation (RAG), which lets a model answer from your own documents, keep the retrieval index, embedding model, and document store inside the same boundary as the generator. Use the NVIDIA GPU Operator to manage GPUDirect RDMA and GDS.

Phase 5: Isolate and operate air-gapped environments

An air gap holds only if every artifact crosses it through one controlled path. Run that path in five steps:

  1. Stage containers on the connected side. A Harbor proxy cache pulls and caches images from upstream registries, and replication pushes them to a target registry. Harbor’s offline installer includes pre-built images, and Harbor permits plain HTTP only in air-gapped test or development environments. Use Zarf’s zarf package create to pull all defined resources into a single package and generate SBOMs. Add --signing-key to sign the package with a Cosign-supported key, then deploy offline with zarf package deploy.
  2. Mirror weights at a pinned revision. Download with snapshot_download at a pinned commit SHA. Inside the boundary, set HF_HUB_OFFLINE=1 so the Hugging Face libraries make no HTTP calls and raise an error if a file is missing from the cache.
  3. Scan before loading. Pickle-format weights can execute arbitrary code on load. Trail of Bits audited safetensors, a format with no executable code path, and found no critical arbitrary-code-execution flaws. Scanners miss threats: ReversingLabs found two malicious models on Hugging Face that its scanning did not flag, and JFrog disclosed PickleScan bypasses including CVE-2025-10155 (CVSS 9.3, fixed in 0.0.31). Accept safetensors by default, and run ModelScan, which reads files without loading them, on anything else.
  4. Sign and verify offline. On the connected side, run cosign initialize and cosign save. Inside the gap, verify with cosign verify-blob --bundle artifact.sigstore.json --trusted-root trusted_root.json. OpenSSF model-signing v1.0 signs models of any format with Sigstore bundles, private PKI or key pairs. Attach SLSA provenance where your build platform produces it.
  5. Transfer through an approved mechanism. NIST SP 800-53 AC-4(20) calls for approved cross-domain solutions in approved configurations, and AC-4(25) calls for sanitizing data between domains. NSA’s NCDSMO maintains the list of approved solutions. CISA describes data diodes as enforcing one-way flow. Sanitize removable media under NIST SP 800-88 Rev. 2, published September 2025.

Patching follows the same path: NIST SP 800-82r3 says to test patches in a sandbox (an isolated test environment), schedule them in maintenance windows, and keep a recovery plan. A blue-green layout (two parallel model volumes, one live and one staged) lets you swap traffic only after health checks pass. Defense Unicorns describes customers that burn artifacts to DVDs, scan them at a secure facility, and load them every week or two, so expect a batched release cadence rather than continuous delivery.

Phase 6: Governance, access control and audit trails

Governance records must hold up as examiner evidence without a second documentation effort. Build them from the following controls:

  • RBAC and identity provider integration: HHS guidance limits access to what each role needs, including administrators. FedRAMP requires phishing-resistant access at Low and Moderate, and NIST SP 800-171 Rev. 3 requires MFA on all accounts.
  • Network segmentation: SP 800-171 Rev. 3 requires separate subnetworks for publicly accessible components and boundary devices for external connections. Place inference endpoints, the vector store and the registry on monitored internal boundaries.
  • Audit logging: HIPAA requires recording activity and regularly reviewing it. AU-9(3) adds cryptographic protection of audit records, such as signed hashes.
  • Model risk management: SR 26-2, issued April 17, 2026 by the Federal Reserve, OCC and FDIC, replaced SR 11-7. It expects a model inventory with “sufficient information to understand model risks,” and it places generative and agentic AI outside its scope. The agencies have said a request for information on bank AI use will follow, so banks currently govern LLMs under their broader risk frameworks.
  • AI management frameworks: NIST’s AI RMF is voluntary and under revision. Its Generative AI Profile (July 2024) covers 12 risks and about 200 actions. ISO/IEC 42001 certification is voluntary and is carried out by bodies that are themselves assessed against ISO/IEC 42006.

KPMG’s Q3 2026 “AI Pulse” found that 86% of organizations reporting established return on investment (ROI) run a formal cross-functional or enterprise AI management layer, compared with 31% of organizations still experimenting (KPMG, 2026). The evidence standard is also moving toward continuous controls monitoring (automated, ongoing checks that controls work). FedRAMP’s RFC-0017 states that assessors must not rely on screenshots or point-in-time dumps except when evaluating the accuracy and reliability of the process producing them (confirm its adoption status before treating it as a requirement), and the 2026 assessor guidance favors complete populations of automated checks where they answer the assessment question and retains representative sampling where full review adds little value or is impractical.

Phase 7: Pilot, promote to production and monitor drift

Most AI projects stall between pilot and production. Gartner’s survey, published in a May 2024 press release, found that on average only 48% of AI projects reach production and that moving from prototype to production takes eight months (Gartner, 2024). S&P Global Market Intelligence’s 2025 “Voice of the Enterprise: AI and Machine Learning” survey found that the share of companies abandoning most of their AI initiatives rose from 17% to 42%, with the average organization scrapping 46% of initiatives (S&P Global Market Intelligence, 2025).

In a 2026 survey of 226 healthcare executives across payers, providers, and pharma, Bessemer Venture Partners and Bain & Company found administrative work advancing fastest, particularly with providers and payers, while clinical applications still need to advance in adoption and trust (Bessemer Venture Partners and Bain & Company, 2026). Bain’s headline figure is 3.5 times ROI on average (Bain & Company, 2026).

The “Stanford AI Index 2026” reports high AI use in financial services risk and compliance functions, which shows adoption, not ROI (Stanford HAI, 2026). Accenture’s “Top Banking Trends 2026” reports that 57% of banking IT executives expect broad or fully embedded AI agent adoption in risk, compliance, and fraud detection within three years, an expectation rather than a measured result (Accenture, 2026).

A first pilot fits administrative or compliance-oriented work where the evidence trail is already defined, such as payer and provider administrative workflows or bank risk, compliance, and fraud-detection work.

Close the gap with gates: Langfuse defines a golden dataset as “a curated collection of inputs with reviewed reference outputs (or explicit pass criteria),” and recommends running every prompt or model change against a pinned dataset version and failing continuous integration (CI), the automated build-and-test pipeline, when scores regress. KServe can then route a fraction of traffic to the new revision (its sample sends 10%) and keep the prior revision available for rollback, while on Day 2, after launch, you track input drift (changes in the data the model receives) and performance even when ground-truth labels arrive late. The table below lists self-hostable tools for this work:

NeedToolLicenseNote
Data and model driftEvidentlyApache 2.0Runs as a library or a self-hosted dashboard
Performance without labelsNannyML OSSApache 2.0Estimates performance with CBPE and DLE
LLM tracing and evaluationLangfuseMIT coreSelf-hosts air-gapped; audit logging requires a commercial license
LLM tracing and evaluationArize PhoenixElastic License 2.0Free to self-host; cannot be offered as a managed service
Model registry and lineageMLflowApache 2.0; Linux FoundationLinks each model version to the run that produced it
Data lineageOpenLineage with MarquezApache 2.0; LF AI & DataRecords lineage events at the run level

What does on-premises AI cost over 3-5 years?

Owned infrastructure beats cloud only when utilization and scale cover fixed staff costs. Over 3-5 years, compare three options:

  • Owned: CapEx (servers, network, storage) + years × (power × PUE × tariff + maintenance + colocation + software licenses + staff) − residual value.
  • Dedicated cloud: hourly instance rate × hours × years, plus storage, egress and the MLOps staff you still employ.
  • Token billing: tokens per year × blended price per million tokens.

For a single 8×H100 server over three years, use CloudZero’s typical HGX quote, ECN Apps’ power figures (10.2 kW-class load at $0.11/kWh and PUE 1.35), Lenovo’s 12% annual maintenance and $1,500 per rack per month colocation, the staffing band from io.net, which sells GPU cloud capacity (1.5 FTE for one to four nodes; treat it as vendor-sourced), and Vantage’s AWS p5.48xlarge rates:

Line itemOwned on-premisesAWS p5.48xlarge
Hardware~$285,000Included
Networking$25,000 to $45,000 per rackIncluded
Power ($13,275 per year)~$39,800Included
Maintenance at 12% per year~$102,600Included
Colocation~$54,000Included
Staff$495,000 to $825,000Not modeled
Computen/a$624,860 at the $23.777/hr three-year reserved rate; $1,446,451 at $55.04/hr on-demand
Total~$1,001,000 to ~$1,351,000$624,860 to $1,446,451, plus staff

For a single server under these assumptions, the enterprise pays less for owned infrastructure than for on-demand compute, but more than for 3-year reserved compute, before adding cloud staff costs. Enterprises can spread staffing costs across additional nodes: io.net, which sells GPU cloud capacity, puts 5-10 nodes at three FTE.

Uptime Intelligence finds dedicated infrastructure cheaper than an established cloud provider at 22% average utilization (the share of time the hardware is busy) or higher, and cheaper than a neocloud (a specialist GPU rental provider) only at 66% or higher. Lenovo’s vendor-published 2026 study puts breakeven for an 8×H200 server at about 3,793 hours against Azure on-demand, and about 10,800 hours against 5-year reserved pricing.

Token billing depends on volume. At the usage assumptions used by Dell, which sells on-premises servers, and Enterprise Strategy Group (ESG), 3,000 tokens per query and 50 queries per user per day, 10,000 users generate a calculated annualized monthly average of about 45.6 billion tokens. That is above the 3.7 to 14.9 billion tokens per month break-even range in a 2026 arXiv analysis, whose own assumptions determine that range.

Applying a blended $1.13 per million tokens, one 2026 paper’s figure for GPT-5 mini (arXiv, 2026), to that calculated volume gives about $619,000 a year. Applying $15 per million gives about $8.2 million a year.

The often-cited 69% to 75% saving comes from ESG research commissioned by Dell and dated April 2024 (ESG, 2024). Business Insider reports that it found Llama 2 70B with RAG on-premises 69% to 75% more cost-effective than AWS.

The companion white paper models 5,000 to 50,000 users, 4-year tables, AWS p5.48xlarge at 3-year reservation discounts, GPT-4o, accessed through an application programming interface (API) billed per token, as the comparator, and power at $0.162/kWh rising 5% a year. It reports 52% to 62% savings against infrastructure as a service (IaaS, rented servers) and 65% to 75% against the API, and it does not state a utilization rate.

API prices have since fallen sharply: a16z estimates a 10x annual decline for equivalent performance. Treat the band as a dated, vendor-funded scenario rather than a forecast.

What in-house expertise does on-premises AI require?

ManpowerGroup’s 2026 survey of 39,000 employers found AI skills are now the most difficult to find globally. Banking Dive reports that 92% of banks cite a skills gap as a hurdle to deploying AI agents, and Uptime Institute’s 2026 survey found that more than half of data center operators struggle to find qualified candidates. Levels.fyi lists a $280,000 median total compensation for machine learning engineers.

Kore1, an AI infrastructure staffing firm, lists data center technicians, platform engineers, network engineers, and MLOps specialists among operations roles. io.net notes that NVIDIA GPU clusters require specialized expertise that general IT teams typically lack, so the staffing band stated earlier depends on hiring it. The team must also cover the control work from Phases 2 and 6: maintaining the SSP and POA&M package, validating FIPS cryptographic modules, and reviewing audit logs.

How do you choose an enterprise AI platform for regulated industries?

Once the model has a compliant place to run, the harder problem is giving agents the operating context of how work moves across the enterprise’s applications. Judge such a platform first by where its data goes, then by what it can prove. Apply these criteria to every vendor, including the one described below:

  • Dual deployment: The platform runs customer-hosted inside your firewall, not only as a vendor cloud service.
  • Data residency: Raw sensitive data stays in your environment, and you can name exactly what crosses the boundary.
  • Auditability: Every input, output, action and decision is recorded in a form an examiner can replay.
  • Explainability: Recommendations trace to explicit rules rather than statistical output alone.
  • Integration footprint: Deployment opens no new connectors into systems of record.

Skan AI is an agentic AI platform for regulated enterprises that builds a connected map of the people, applications, tasks, and decisions in a process, called the Context Graph of Work, from observed process activity, so banks, insurers, and healthcare organizations can ground agent decisions and execution in how work runs. Skan AI is a Leader in the Everest Group PEAK Matrix three years running.

Phases 1 through 7 give models a compliant place to run and a defensible evidence trail, but they do not tell an agent how a claim, loan file, or prior authorization actually moves across a bank’s, insurer’s, or healthcare organization’s applications. In Skan AI’s view, that missing operating context is what sustains the gap Gartner’s survey describes. The ROI gap is not an AI problem, it’s a context problem.

Skan Virtual Assistant (Skan VA) captures, with privacy by design, how work moves across legacy desktop, mainframe, Citrix, virtual desktop infrastructure (VDI), web, and office applications with zero integrations, APIs, or logs.

Skan PRISM, an on-premises gateway, applies pixel-level redaction (obscuring sensitive content on screen) before transmission, so raw screen data stays in the customer environment and only anonymized, abstracted metadata reaches analytics. Application allow-listing, pseudonymization at collection (replacing personal identifiers with placeholders), customer-defined retention, automated deletion, and right to erasure are the controls behind privacy by design, focused on work patterns.

The containerized platform deploys on-premises inside the customer firewall, private cloud, or Skan AI hosted, and each option has its own data boundary.

Skan AI holds SOC 2 Type 2 attestation and ISO/IEC 27001 and ISO/IEC 42001 certifications, and is HIPAA, GDPR, and CCPA compliant.

Neurosymbolic reasoning pairs neural perception of screen activity (machine-learning recognition of what appears on screen) with symbolic rules (explicit, human-readable logic), which produces explainable, auditable process models in the Context Graph of Work, a property graph whose nodes represent people, applications, tasks, decisions, policies, and outcomes. Skan AI Blueprint identifies areas through a lighter opportunity scan, Skan AI Process Intelligence observes those areas in depth (including drift detection, which flags when work starts to depart from its usual pattern), and Skan AI Agents execute under agentic operating procedures (AOPs), the procedures that govern what each agent may do, after each agent is simulated and evaluated against human baselines before promotion. Its governance capabilities include audit trails, case-level playback, continuous controls monitoring, and deviation alerts, which support evidence gathering and control verification.

For pricing based on your users or processes monitored, get a customized Skan AI quote.

Request a Skan AI demo to explore how observed workflows become context for governed AI execution: request your demo.

Frequently Asked Questions

Rarely by explicit rule: the Health Insurance Portability and Accountability Act (HIPAA) allows cloud hosting under a business associate agreement, and the General Data Protection Regulation (GDPR) allows transfers under Chapter V mechanisms. The pressure toward on-premises hosting builds when a model has to read regulated data in clear text, because the International Traffic in Arms Regulations (ITAR) require technical data to stay encrypted outside an authorized security boundary, and the European Data Protection Board (EDPB) found no effective safeguard for cloud providers that need data in the clear. Air-gapped hosting fits disconnected or classified sites that move data only through approved cross-domain solutions.

An 8B model serving 50 users at 4,096 tokens fits on a single 80 GB H100, and a 70B model at FP8 needs about 70 GB for weights and roughly 40 GB of KV cache per 128k-token request, so two H100s hold about two long-context requests. Llama 3.1 405B at FP8 runs on a single 8×H100 node. A DGX H100 draws up to 10.2 kW, and racks above about 20 to 30 kW move toward liquid cooling.

Download weights on the connected side at a pinned revision, prefer safetensors, and scan any pickle files with tools such as ModelScan. Sign them with cosign or OpenSSF model-signing, and move them through an approved cross-domain solution or a data diode. Inside the gap, verify signatures offline against a trusted root, test in a sandbox, and swap traffic with a blue-green deployment after health checks pass.

It depends on utilization and staffing: Uptime Intelligence puts breakeven against an established cloud provider at 22% average utilization. For a single 8×H100 server over three years, staff costs push owned infrastructure above AWS 3-year reserved pricing but below on-demand. The Dell-commissioned ESG figure of 69% to 75% savings rests on 2024 assumptions, including GPT-4o API pricing that has since fallen.

Group requirements by stack component: identity, encryption, logging, network boundary, and model registry. Map each one to HIPAA 164.312, NIST SP 800-53 for FedRAMP, NIST SP 800-171 for CMMC, and ITAR 120.54. FedRAMP packages require the SSP, SAP, SAR, POA&M, inventory workbook, and cryptographic modules table. CMMC assessors expect FIPS validation certificates. FedRAMP’s 2026 guidance favors live configuration data and complete automated checks over static documents.

CONTENTS

Share this article

Subscribe to Skan AI’s blog

Want to stay up to date on all things Skan AI? Subscribe to our monthly newsletter and be the first to know when we release new things!