Boards are asking two unrelenting questions: what is our AI strategy, and when will we see returns? That pressure has teams deploying use case after use case at breakneck speed but rarely with a rigorous ROI model or a realistic sense of what any of it costs to run.

As a result, AI gets deployed one use case at a time, and the bills accumulate quietly until they’re so high that finance starts asking questions.

Understanding the costs of AI is critical, as so many organizations focus on driving efficiency and productivity gains with AI. If the pitch is “AI will save money,” a deployment that quietly burns money on avoidable compute isn’t a minor inefficiency but a failure. 

At this point, leaders hunt for savings via cheaper AI models or reduced usage. Both are reasonable routes, but they miss the bigger opportunity. Real savings requires more than a single substitution. It requires treating cost as an architecture decision instead of a line item to manage after the fact.

Model routing is where a lot of that architecture work happens. Done well, it can meaningfully cut what a company spends on AI while holding onto most of the output quality, though exact savings vary depending on the workflow, and they aren’t automatic.

The companies that realize those savings are building deliberate cost optimization strategies around how they deploy AI. To start, investigate where the money is leaking.

Where AI Costs Hide

Walk into a typical company running AI at any scale, and you’ll find a default: one capable, general-purpose model handling everything. This is rarely a strategic decision. It’s something teams pick because they need a reliable default to produce “good enough” results on everything in order to move quickly.

In that situation, the model isn’t the problem, but most of what it’s tasked with doesn’t require its full power. Route minor tasks to the same expensive model that’s handling your hardest analytical work, and the costs add up. The framing of selecting a “good enough” model isn’t working because it’s being applied to all use cases at once.

Leaders looking to cut AI costs may be tempted to ask if there’s a cheaper model that would work across the board. Instead, they need to map the complexity of the tasks within their workflows and reserve high-cost models for the hardest ones. Matching the model to the task is the foundation of model routing.

Getting Started with Model Routing

There’s no universal first move. The right place to start depends on what your company is doing with AI.

Companies running mostly off-the-shelf tools (e.g., enterprise Claude or ChatGPT) don’t need to worry about routing. They need to focus on access and policy: which model is the default, who has access to more expensive tiers for specialized work, and what is enabled for each team. 

The starting point for those organizations is to audit where costs are being incurred, how employees are using the tools, and what workloads are driving spend. From there, the conversation can shift to whether these are the right use cases for AI and if a more efficient workflow can achieve similar value at a lower cost.

In contrast, companies building custom AI systems face engineering decisions. The audit looks similar, but the fix looks different. Look at what models are the default, and ask why. Then evaluate each use case, simplify to lower-cost models where possible, and implement routing while building test suites to measure the accuracy and quality tradeoffs.

A thorough evaluation requires multiple perspectives in the room. Engineering needs to explain the technical tradeoffs. Business decision-makers need to own the budget conversation. And subject matter experts — the people who know what a good answer looks like in a given domain — need to evaluate the quality of the output. 

The most common mistake for many companies is overexcitement. In a moment of enthusiasm for a new model, architecture, or use case, it’s easy to build a massive multi-agent system for a problem that one or two agents could solve. Resist the temptation to overbuild or overspend, and focus on the efficient path to your desired result.

Model Routing, Not Model Substitution

Once you see your workflows as a set of distinct tasks, routing stops looking like a shopping decision (which model to pick) and starts looking like an architecture decision (how to direct the right tasks to the right models). 

Model routing, done well, means deconstructing workflows into component parts and assigning each part the level of capability it requires. Some of that assignment may happen while building, with a developer pre-selecting a model for each step based on expert feedback about different models’ responses. Some of it may happen in real time, with an AI-based orchestration layer deciding how difficult each task is and routing accordingly.

The mechanics of deciding what’s easy versus hard don’t need to be overly complex. Teams can start with lightweight orchestration prompts that define task tiers — easy, medium, hard — and refine them through testing and monitoring. Companies that want to go further can fine-tune a dedicated classifier model whose only job is figuring out where tasks belong, but most don’t need to go that far to see meaningful savings.

Optimization Opportunities Beyond Routing

Routing is the highest-profile piece of cost optimization, but it’s not the only one. Other techniques can do a lot of quiet cost optimization work as well.

For example, instead of simply routing straightforward tasks to smaller external AI models, some companies are leaning on internally owned and trained models. Locally run small models can be trained on a narrow domain, such as a specific product’s support questions, and get very good at related tasks while doing less work per query. In some cases, these small models can outperform large general-purpose models, in addition to being cheaper to run.

Prompt caching and memory compression solve a more mundane problem: sending the same context over and over. Caching a prompt that doesn’t change between calls or summarizing a chat history instead of resending the full transcript cuts token volume without touching routing logic at all. Some model APIs now handle a version of this automatically, but it’s worth confirming rather than assuming the defaults are doing the job.

There’s a related benefit that doesn’t show up on any cost report but belongs in the same conversation, and that’s speed. Smaller models are faster models. That’s not a savings mechanism in the same way routing or caching are, but it’s a direct consequence of the same architecture choices. For many applications, response time is the difference between a tool that lets the user perform at peak efficiency and one that delays their actions. A company evaluating routing purely on the cost line is measuring only part of the benefit. 

The Judgment Calls in Cost Optimization

Evaluating cheaper models forces a few classic business judgment calls.

There is no universal line between an acceptable error rate and an unacceptable one. It depends on what a bad output costs you. A wrong ingredient when ChatGPT suggests a recipe likely isn’t a big deal. A security vulnerability introduced by an AI coding agent is a different category of problem. Instead of asking how good the output needs to be, ask what failure will cost. Model selection should mirror the financial and operational risk of that failure.

Open-weight and locally run models carry a related wrinkle, since they’re central to a lot of these savings conversations and get pitched as both a cost and a security tradeoff. Building and securing the models incur upfront labor and infrastructure costs, and they place the burden of security squarely on your team. 

In the long run, the costs are often much lower, and keeping the model and data on your own infrastructure is often more secure than handing them to a third party. The upside here is conditional: an open-weight or locally run model can be the right choice, but only if you have the expertise to set up the infrastructure correctly and are willing to make the short- and long-term investments in maintaining it.

AI Cost Optimization as an Ongoing Discipline

It’s tempting to treat model routing as a temporary fix until the underlying models get cheaper and more capable. But the underlying discipline of matching the capability of the tool to the difficulty of the task won’t become less relevant as models are deprecated and replaced. It will become more relevant because the volume of AI usage inside most organizations is only going up. 

Also, remember that current API pricing is not necessarily the floor. A lot of today’s model pricing is subsidized by venture capital chasing market share, which means the true cost of running these systems may be higher than what companies are paying right now. Not only is the cost pressure here to stay, but it’s likely currently understated.

Leaders who have an eye on the future are taking this opportunity to ensure cost architecture is part of their AI builds from the start. Crafting that architecture starts with an internal audit, but it takes more, including expertise from people who have built these systems and understand what moves the costs.