CIO confession 2 of 6: Your AI agents look healthy. Are they paying off?
October 15, 2026/5 min read/Julia Berman
An AI agent can run reliably, respond quickly, and pass every technical check while the most important question about it stays open: is it paying off? For CIOs, that question now carries their name. Only a quarter (24%) of CIOs say their organization has mature, highly capable ROI measurement across all or most AI initiatives.
In part 1 of Dataiku’s six-part series on the Global AI Confessions Report: CIO Edition, 2026, a survey of 685 enterprise CIOs conducted with The Harris Poll, we traced how pressure to prove AI ROI travels from the board to the CEO to the CIO. This post turns to the first of the report's four confessions: accountability for AI is rising faster than the ability to measure its value across the enterprise.
We'll look at where the measurement gap sits, how cost uncertainty is already reshaping AI plans, and the four questions that show whether an agent is actually working.
Accountability is outpacing measurement
The board's ROI question assumes CIOs have the numbers ready. Most are still building the capabilities to produce them. Beyond the quarter with mature ROI measurement across most initiatives, less than half (41%) have mature ROI measurement even for select use cases.
That second figure is the telling one. Select use cases tend to be the easiest place to measure, providing a handful of high-visibility projects with clear owners and defined success metrics. When mature measurement is the exception even there, the challenge across a full portfolio of agents in production is considerably larger.
The timing raises the stakes. As we covered in part 1, AI budgets are increasingly tied to hitting performance targets on the CEO's timeline. Measurement has become the capability that protects every other AI investment.
ROI starts with knowing what AI costs
Understanding return starts with understanding spend, and AI spending no longer sits neatly in one place. Models, infrastructure, tools, teams, and individual agents all contribute to AI operating costs across the enterprise.
Full visibility into that spend is rare. Only 21% of CIOs have full, near-real-time visibility into AI agent and advanced AI workload costs with attribution by business unit, team, or use case.
Attribution is the key word. A total AI bill tells CIOs how much they're spending. Attribution tells them which team, use case, or agent generated each line of it, and that's what makes an ROI calculation possible.
CIOs' top cost concerns follow the same thread:
Cost visibility: 38% cite lack of cost visibility by team or use case.
Hidden costs: 38% cite agent proliferation driving hidden costs.
Redundancy: 37% cite duplicate or redundant AI tools.
Each concern compounds as agent populations grow. More agents mean more line items, more overlap between tools, and more spend to trace back to a specific outcome.
Cost uncertainty is already reshaping AI strategy
This uncertainty reaches well beyond the finance review. The cost of tokens and compute required to run AI agents and applications has significantly changed how 39% of CIOs plan, approve, or execute AI strategy. Another 33% say unpredictable AI usage has significantly disrupted their plans.
Cost predictability now shapes which projects move forward, how quickly they scale, and how confidently CIOs can commit to AI targets. Clear, attributed cost data gives CIOs a stronger position in each of those decisions.
Cost is half the equation
A complete view of AI answers half the question. CIOs also need to know whether agents are performing as intended, and whether that performance is producing the results they were built to deliver.
This is where the gap is widest as 72% of CIOs don't consistently measure both operational performance and business outcomes across every AI agent.
Consider an agent that handles customer service requests. Uptime is high, response times are fast, and error rates are low. On a technical dashboard, it looks healthy. The questions the CEO cares about sit elsewhere. Is it resolving issues? Is customer satisfaction rising? Is the cost per resolution coming down? Until those answers are in, a healthy agent and a valuable agent look the same on paper.
Four questions that prove an agent is working
The report points to four questions CIOs need to answer to know whether an agent is actually working:
Cost: What is being consumed, where, and at what cost?
Technical performance: Is the agent running reliably and behaving as expected?
Agent effectiveness: Is the agent successfully doing the job it was created to do?
Business impact: Is the agentic workflow improving the business outcome it was intended to affect?
The first two sit closest to traditional IT monitoring. The last two are where proof lives. Effectiveness shows whether an agent does its job. Business impact shows whether doing that job moves the metrics the board and CEO are tracking.
Part 1 traced a chain of pressure. These four questions build a chain of evidence, running from what an agent costs to what it returns. The value comes from answering them consistently across every agent in production, so CIOs can compare results, prioritize investment, and act on what they find.
From confidence to evidence
Every link in the pressure chain asks for proof, and measurement is how CIOs supply it. CIOs who connect agent cost and technical performance to enterprise outcomes can defend AI budgets to support scaling the agents producing results and answering the board's ROI question in the numbers it asked for.
Measurement tells CIOs whether an agent is delivering value. Our next post will take on a harder case — agents that operate exactly as designed and still miss business intent.
