Enterprise AI cost governance has moved from a back-office accounting concern to a boardroom-level priority in 2025. Gartner's latest CIO Agenda survey ranked AI cost overruns among the top five barriers to scaling AI - the first time cost, rather than talent or technical readiness, topped that list. The problem is structural: inference spend behaves nothing like traditional software licensing, and most enterprises discovered that gap only when the quarterly bills arrived.
McKinsey & Company, State of AI survey series (2023-2025)
| Value (% of organizations) | Citing unclear ROI/cost control as barrier to AI production |
|---|---|
| 2023 | 26 % of organizations |
| 2024 | 34 % of organizations |
| 2025 | 42 % of organizations |
Cheaper tokens, bigger bills - the paradox no one planned for
Token prices from frontier model providers dropped 80 to 90 percent over the twelve months through early 2025, according to pricing histories tracked by The Register and Reuters. OpenAI, Anthropic, and Google all cut per-token rates aggressively as competition intensified. The logical assumption in many AI business cases was that falling unit costs would keep overall spend manageable. That assumption turned out to be wrong.
What happened instead is classic Jevons paradox applied to AI: as inference got cheaper, development teams called the APIs more frequently, chained more models together, and pushed larger context windows through every request. Call volumes and model complexity scaled faster than unit price reductions, so aggregate bills climbed even as per-token rates fell. Bloomberg Technology reported in May 2025 that several large financial-services and retail enterprises had launched emergency reviews of their AI API spend after quarterly invoices landed two to three times over projection.
This is not a niche problem. IDC's AI Spending Tracker, updated in May 2025, projects worldwide AI software expenditure will cross $300 billion by 2026. At that scale, even modest percentage overruns translate into hundreds of millions of dollars in unplanned IT costs across the industry.
42% of organizations cite unclear ROI or cost control as their primary reason for not moving AI pilots to full production - up from 26% in 2023. McKinsey, State of AI 2025.
The FinOps gap: most enterprises are flying blind on inference spend
A spring 2025 Forrester brief found that fewer than one in three enterprises running active GenAI workloads have a formal mechanism to allocate inference costs back to the business units consuming them. That means the majority of organizations currently have no showback, no chargeback, and no per-team visibility into what AI is actually costing them at a granular level.
This is a governance problem as much as a financial one. In a conventional cloud environment, FinOps practices - tagging resources, mapping spend to teams, setting budget alerts - are now fairly mature. But AI inference costs flow through API gateways and managed model endpoints that sit outside the resource tagging conventions most IT shops built for compute and storage. The tooling, the policies, and often the organizational ownership have not caught up.
The knock-on effect is that finance teams cannot build reliable forecasts, and CIOs lose credibility with boards who approved AI investment expecting quantifiable returns. McKinsey's State of AI 2025 survey found that 42 percent of organizations cite unclear ROI or cost control as their primary reason for not moving AI pilots to full production - up sharply from 26 percent in the 2023 edition of the same survey. Cost uncertainty is now the single biggest drag on AI scaling, not technology immaturity.
What a functional AI FinOps framework actually looks like
The enterprises making progress on this problem share a few common practices. First, they treat AI inference as a distinct spend category from cloud compute, with its own tagging taxonomy, budget owners, and alert thresholds. Rather than lumping API calls into a general cloud bill, they route inference traffic through a gateway layer - whether a vendor-managed tool or an internal proxy - that can attribute costs to a specific application, team, or business unit in near real time.
Second, they apply the same showback-before-chargeback discipline that mature cloud FinOps programs use. Start by giving business units visibility into what they are consuming. Accountability tends to follow naturally once teams see their share of a six-figure monthly AI bill. Hard chargebacks come later, after cost allocation logic has been validated and exceptions have been ironed out.
Third, they set model-selection policies. Not every workload requires a frontier model. Routing classification tasks, summarization requests, and other lower-complexity operations to smaller, cheaper, or self-hosted models can cut inference spend significantly without degrading end-user outcomes. This also has a security implication: Forrester and others have noted that cost pressure, without explicit policy, tends to push teams toward consolidating on whichever model is cheapest at a given moment - reducing vendor diversity and concentrating supply-chain risk in ways CISOs should be watching.
What CIOs should do before the next quarterly review
The structural mismatch between consumption-based AI pricing and annual IT budget cycles is not going away. It will get harder to manage as more workloads go to production. The enterprises that build tagging, attribution, and governance frameworks now will have a material advantage when boards press for AI ROI accounting in 2026 budget cycles.
Three near-term actions matter most. First, audit where AI API spend is currently sitting in the general ledger - it is often buried in cloud line items or distributed across departmental budgets in ways that make the true total invisible. Second, designate an AI FinOps owner: this does not require a new headcount, but it does require someone whose explicit job is to track, report, and optimize AI inference costs across the organization. Third, define a tiered model policy that matches workload complexity to model capability and cost, and enforce it through the API gateway layer rather than relying on individual developers to self-police.
The comparison to early cloud governance is instructive. Organizations that ignored cloud cost discipline in 2012 and 2013 spent years untangling shadow IT spend and renegotiating contracts from a position of weakness. The AI equivalent of that problem is arriving faster, because inference costs scale with every user interaction rather than with quarterly infrastructure provisioning cycles. The window to get ahead of it is now.
