Measuring AI: Hours, Tokens, Dollars, and Results
Simply by virtue of being around for a while, I’ve been “lucky” enough to have seen booms and bubbles come, go, and occasionally make lasting change. Many of those shifts have turned the impossible into the everyday or made us wildly more productive. For those of us working at the edge of these transitions, it was clear that with hard work and effort, we could build new and wonderful things.
Before we get into the new and wonderful things made possible by AI and how to measure them, let me provide you with the article about AI measurement you probably came here expecting to see. It was entirely written by Anthropic’s Claude while I was making coffee. I don’t know if it is accurate, but it uses some good words:
# Measuring ROI on AI in Corporate Settings
The pressure to prove AI ROI has sharpened. By early 2026, 61% of senior business leaders said they felt more pressure to demonstrate returns than a year prior, and 53% of investors now expect positive ROI within six months. MIT’s *State of AI in Business 2025* found that 95% of enterprise generative AI projects had not shown measurable financial returns within their first six months. The gap between AI spend and AI outcomes is now one of the most scrutinized metrics in enterprise technology.
Teams that consistently extract returns share a few habits.
## They establish baselines before deployment, not after
The most common failure mode is measuring value after the technology is already live. By then the counterfactual is lost. High-performing teams quantify the current cost of the problem first: hours spent, error rates, cycle times, revenue lost to slow response. Conservative projections built on those baselines beat vendor claims every time.
## They tie AI investments to the earnings plan
Mature teams evaluate AI the same way they evaluate any capex or opex: impact on operating expense, margin, and revenue. New York Life’s benefits group, which has reported consistent positive returns, attributes that to treating ROI as a gating question from day one, not a retrofit.
## They measure both hard and soft ROI — separately
Hard ROI is tangible: cost avoidance, throughput gains, EBITDA impact. Soft ROI covers employee morale, decision quality, and strategic optionality. Conflating the two produces dashboards that don’t survive board scrutiny.
## They go beyond adoption metrics
Tracking that 60–70% of employees “use AI tools” is meaningless on its own. The real questions: how much more productive are those users, on what tasks, and is that productivity reinvested in higher output or captured as cost savings?
## They take a portfolio view
Individual pilots will fail. A portfolio approach lets winners fund the losses and surfaces what actually works in a given organization. Teams getting this right treat AI like internal venture investment: staged funding, kill criteria, and explicit graduation gates from pilot to scale.
The throughline: AI ROI doesn’t come from better models. It comes from better measurement discipline, applied before the first dollar is spent.
Now that we have satisfied the instinctual need for listicles, I hope you will stick with me for a more human interpretation of how it feels to try and evaluate the results of a truly significant shift in work.
AI feels uniquely different. Not because it lacks the characteristics of previous shifts like ubiquitous connectivity or touch interfaces but because it adds a dimension that we haven’t seen in many people’s lifetimes. Current-generation AI is not opening a new field for exploration; it’s changing the relationship between effort and results.
Results have usually aligned with three things:
- The quality or creativity of your vision/thinking/decision-making
- How hard you worked
- A sprinkling of luck
The teams that out-executed captured market share. Showing up and working hard was a competitive advantage. Moving fast was great, and moving fast in the right direction was even better.
AI is going to show us how many people are running on effort alone.
The most successful organizations have always delivered results disproportionate to their costs. The reasons are typically unique to the company. In software, it’s not uncommon to have more ideas than time to deliver them. Development costs are typically controllable and predictable. You have however many people you have. Software maintenance costs are less visible, quietly consuming more and more of a fixed team’s capacity as software liabilities accumulate.
In many organizations, increased capacity from team growth ends up being consumed by maintenance, decreasing ROI on human engineering hires. Historically, the solution has been code quality, reliability, automation, and spending the time to make your code small enough and understandable enough that humans could work with it. These techniques let you stay longer on the good parts of the curve.
With AI, things are going to be different. While there is still a large quality gap between AI-written software and what the best engineers produce, this time allocation is changing fast. Modern AI tools can write, change, and wrangle enormous amounts of code with much less understanding than a human engineer. The risky updates and breaking changes that once took days of careful integration are becoming background tasks. It’s not inherently less risky, but it requires less direct effort and attention, and the amount of human knowledge an experienced engineer needs to hold in their head is decreasing.
Teams that start to convert this kind of work into AI-supervised workflows will see this curve start to bend. It will create opportunities for teams to focus on projects, features, and speculative work that we never had the time to do. Not all teams will realize this because it’s not entirely a technical shift; it’s a cultural one. Matching AI to the right work is what creates space for the existing team to produce new, valuable work.
One area where AI will deliver immediate, outsized returns is in table-stakes, effort-based exercises. For example, consider the medium-sized subscriptions that enterprises quietly accumulate, like transcription services or file processing tools. It’s not uncommon to find tens of thousands of dollars a year going to things that are now replaceable with an hour or two of engineering and a few dollars of AI.
Just yesterday, I needed to convert roughly 200 hours of podcast audio into searchable text. AWS quoted about $500, and a more consumer-focused service quoted $3,000. A year ago, I would have spent half a day setting up the AWS service. Yesterday, I spent 20 minutes and $5.37 worth of AI time. Now, I have a command-line tool running high-quality transcription at 8-10x real time, chugging through the back catalog of episodes as I write this.
This is an area where I have some experience. It’s not entirely novel, and the incremental tooling is probably throwaway, but the opportunity is real. I find that most large clients have multiple six-figure contracts for things that are not core to their businesses. Each of those is ripe for replacement, and every dollar recovered could fund more valuable work.
Tokens
The literal coin of the realm in our modern era. Tokens are the chunks of text that an LLM reads and writes, billed as input and output separately. Your AI doesn’t care about time; it cares about tokens.
Token usage is partially within our control, but it’s something of a dark art. There is a constant tradeoff between token use and context.
Context is everything an LLM can “see” at once: your prompts, its replies, files, and tool output. It has a fixed token limit, and once full, older content must be summarized or dropped. Everything in context is measured in tokens, and you pay for those input tokens on every turn. So bigger context means higher cost and slower replies — the two are directly linked.
The tension here is that everything in context ends up being billed over and over again in a conversation, but without that context, the LLM produces wildly lower quality results.
Humans handle memory transparently. We don’t have to consciously decide if we want to remember to check the grocery list before we head to the store. An LLM forgets the grocery list exists if it’s not in the context. Without context, quality collapses. The model goes to the store, buys the first three things it sees, comes home proud, asks if you happen to have a list, then happily returns to buy one item, comes back, and asks if you might have preferred it to get everything in one trip. They try, and they are getting better, quickly. But they bill per token.
Effective use of AI requires new measurements. If I told you that the $341 bill for a day’s work represented a technical refactor saving $20 per day, forever, you would immediately calculate the ROI. However, if that bill is for something less measurable, it’s a lot harder. Even riskier, the magnification of effort can calcify processes that shouldn’t exist or create new work for others at an alarming rate.
If that $341 bill was for filling out my TPS reports or replacing the judgment a skilled professional brings to their work, that ROI is not so clear. Worse: what if it was for creating something that pushed time cost onto others? Let’s say it generated 2,000,000 words of text. How many person-hours did it take to read? What happens when everyone has AI summarize everything they receive? Consider the journey of a brilliant 10,000-word AI-generated strategy document, emailed to 50 people, summarized by AI to 250 words, and responded to via an AI-generated reply based on the summary. Generative is in the name. Magnification is the path of least resistance. It’s on us to encode our real needs into the instructions we give.
Intention still matters. Effort applied to no purpose is still the same ROI it always was: zero. Now, it comes with an extra cost in tokens that will likely increase (at least short term). Token maxxing is an example of the tension between effort and results. If you buy into the idea that all effort is good, then the best outcome is having the AI do as much as possible. That idea will come with a cost measured per million tokens.
Results
Success is often in your definitions. AI pilots are everywhere, and measuring them is incredibly difficult, particularly in the mad dash to claim short-term ROI. Some companies have been confronted with enormous bills or blown through annual budgets in weeks. This author has absolutely never consumed a month’s worth of token budget on the first day.
One thing that may be slightly less visible is that pilots are often constrained in ways that go beyond budget and affect results. Pilots are often shaped by risk, and statements around what we won’t allow can significantly alter the landscape for usage. For example, the enterprise versions of popular coding tools can be configured to allow or disallow certain tools and use different levels of autonomy.
One company concerned about risk might disable access to certain tools, like cURL, which acts like a web browser without a UI. It can have risks, but it’s critical for certain software development workflows and brings context into LLMs. Another company might disallow agentic interactions that allow LLMs to run for a long time with no human intervention.
There isn’t a right, universal answer, but pilots designed around risk and restrictions are likely to return confusing data.
One popular model for AI-supported work is to spend meaningful time upfront writing detailed specifications with the AI, compressing decisions into a detailed plan and handing it off to the tool with a defined workflow for implementation and verification. With proper setup, three or four hours of definition work can return a week’s worth of implementation effort, completed overnight, with a clean summary of what needs attention. It’s not appropriate for every use case, but the results can be exceptional. If your pilot is set up to disable auto mode and workflows, you reduce the value of the setup work, and instead of magnifying your team’s effort, you turn them into babysitters.
Designing an effective pilot
Many enterprise pilots for AI are being designed like previous software rollouts: take a new tool, roll it out to a group, and then ask them if they liked it or measure a productivity or cost change. That worked for single-purpose tools built to solve a specific problem. You had that problem, or you didn’t; the solution fit your business, or it didn’t.
Lately, the subscription models, per user, per month, have led to predictable prices, but the real results usually come much later once you roll out the tool to everyone. With AI, the metric should be different. A pilot that is ‘a little AI for everyone’ will likely result in very incremental results. Learning to use a high-level AI effectively is more like learning how to manage a new team than adopting a new SaaS product.
Pick a real project, something that you didn’t do because it was too expensive or too time consuming. Have someone dedicate a few days to building it. If it delivers a real business outcome and value, do it again with a new project. Look at the costs, look at the results, and talk to the people about their experience. This is an effective pilot.
When we measure completed work, how often do we weigh effort over impact? “Tried really hard to deliver value” will always lose to “delivered value.” The core metric is the same. Does your team have a backlog of vetted work? Things you know you should do, out of reach due to time, effort, or capacity? If so, get AI into the hands of your best thinkers now. The people who represent the quality and judgment that drives long-term success. If instead you only have a list of things you could do, ideas that have never been pressure-tested or initiatives that have lingered for “maybe next year,” AI will let you execute on those too. Results will increase in quantity. Whether that’s the right outcome depends entirely on the business.
Lately, it feels as though we have made it to a new phase of the industrial revolution. The legend of John Henry resonates for a new generation of workers; the costs to compete against a machine may not be as instantly dramatic, but they will magnify. Teaming up with the machine to do things that we could never have done before is within reach.
The determining factor for results will be how well we identify what’s worth doing and in ways that are grounded in what makes this specific business successful. What do customers need? What have they been waiting for that we haven’t been able to provide?
Investing in the team’s capability to understand where effort can translate into meaningful value will be more important than ever. The constraints have changed, but this core capability has always been at the heart of real results. Measurable, sustainable P&L impact won’t come from effort alone, human or AI, but from making the right decisions about where to apply that effort.
Identify where AI creates real value for your business and where it doesn’t.
Get in touchMeet the Author
Mark Thomas is a Catalant consultant and Principal at Future Core, LLC, where he supports leading organizations through technical strategy, software assessment, and interim and fractional executive leadership. With more than twenty-five years of experience building and managing some of the most complex technology products and ecosystems on the planet, Mark has held broad technical and strategic leadership roles across startups and industry leaders like Houghton Mifflin Harcourt, HBO, AMD, and Microsoft. Mark holds a Bachelor of Arts from Hampshire College.
Corporate leaders must quantify the precise operational costs of an existing business problem before deploying AI tools. Evaluating pre-deployment baselines—such as task hours, error rates, and cycle times—prevents companies from losing the counterfactual metrics needed for board scrutiny. Waiting to measure value until an AI system goes live removes the baseline comparison data, making accurate ROI calculations impossible.
AI shifts corporate performance metrics from a focus on human effort to a focus on scalable strategic intent. Traditional software rollouts optimize specific, predictable human workflows, whereas AI scales content generation and data processing at a fixed token cost. Because LLMs decouple human labor time from project output volumes, enterprise success depends on choosing the correct business problems to automate.
Standard software pilot frameworks fail because general technology adoption metrics do not measure true workforce productivity gains. Giving entire departments broad access to AI tools yields only minor operational changes. Successful organizations treat AI integration like managing human teams rather than purchasing subscription software, deploying technology toward specific, high-value backlogs that were previously too expensive to execute.
Strict corporate risk restrictions generate inaccurate pilot data by limiting the operational autonomy of LLMs. Disabling advanced technical capabilities, such as web scraping tools or automated multi-step workflows, converts highly skilled professionals into software babysitters. While risk management is necessary, blocking autonomous interactions minimizes the primary efficiency advantages that AI infrastructure is designed to deliver.
Optimizing for AI output volume creates severe financial liabilities by inflating operational costs and generating unnecessary internal workflows. LLMs bill per token, meaning unmanaged automation exponentially increases cloud computing expenses. Furthermore, mass-producing automated text forces other employees to spend valuable hours summarizing and reading low-value documentation, which destroys net enterprise productivity.
