The Cost per Token is projected to tumble over the next few years, yet overall token spend will increase. How is this possible, and where does the concept of token value routing come into play?
As we see the continued adoption of Agentic AI, coupled with the increased capability of AI models, the quantity of tokens consumed will rise sharply. This means that while the cost of each token might go down, our usage will increase at a higher rate, causing an overall net increase in spend. Managing this increasing spend will need new way of thinkings and architecting solutions and one core concept will be Token Value Routing.
What is Token Value Routing?
Let's define it before the term gets picked up and warped into many different meanings, as many tech terms tend to be. Many AI applications today are built by selecting the most advanced model available and directing all requests to it. While this approach works well in demonstrations, it often leads to high costs once deployed at scale.
Token Value Routing is the practice of sending each request to the model that delivers the best value for that specific task, rather than defaulting every request to the biggest, most expensive model in the room.
The focus should be on value, not just cost; selecting only the cheapest model can compromise quality. Instead, requests should be matched to the appropriate model based on three key factors:
-
Cost: what you pay per token in and out.
-
Quality: Does the model actually answer the question well enough for this task?
-
Latency: Does the user need an answer in 200 milliseconds, or can a background job wait?
Token Value Routing applies the principle of using the right tool for the right job to the model layer, making routing a dynamic decision rather than a fixed one. Simple tasks such as password resets and complex reasoning tasks do not require the same model, so costs should reflect this distinction.
Why does it matter now?
This is increasingly important because token costs are now a significant and growing part of operational expenses. Industry estimates indicate that enterprise spending on LLM APIs exceeded $8.4 billion in 2025 and will continue to grow in 2026. I suspect a significant portion of this expenditure is wasted on routine requests processed by high-cost models because there is no decision layer.
Consider the following example:
For a workload of one million requests per day at 2,500 tokens each (2.5 billion tokens daily), sending all requests to a frontier model at $15 per million tokens results in a daily cost of $37,500. By routing 80% of simple requests to a smaller model at $1.50 per million tokens and reserving the frontier model for the remaining 20%, costs can be significantly reduced.
In the 80/20 scenario, daily costs decrease from $37,500 to $10,500, resulting in a savings of $27,000 per day, or approximately $9.86 million annually for this illustrative workload.
Determining what portion of traffic is genuinely 'simple' becomes a key cost consideration and technical design requirement. These savings are being proven in the market; for example, the RouteLLM team demonstrated that trained routers can reduce costs by up to 85% while maintaining approximately 95% of GPT-4's performance on standard benchmarks.
More broadly, routing strategies have been shown to cut inference costs by 40–70% without significant quality loss. The initial enthusiasm for new technology often leads to the use of the most advanced model for all tasks, which now significantly impacts budgets. The goal is not to reduce AI usage, but to allocate resources where they provide the most value.
How do you actually do it?
Token Value Routing is implemented through a gateway that sits between your application and the models, making routing decisions for each request. Your applications interact with a consistent interface, while the gateway determines the destination, logs costs, and manages failover if a provider experiences issues. A few routing strategies, which are usually combined rather than picked one-and-done:
-
Complexity-based routing: classify the request, then send the easy ones to a cheap or local model and the hard ones to the frontier model.
-
Cascading: try the cheap model first, then only escalate to a stronger model if the first answer fails a quality check.
-
Cost-based routing: pick the cheapest model that clears the quality bar for that task type, within a defined budget.
-
Latency-based routing: use fast models for interactive, user-facing requests; use cheaper, slower models for batch and background work.
-
Domain routing: send code to a coding-specialised model, vision to a multimodal one, and so on.
Tooling in this area is maturing rapidly; a range of gateway solutions are now available, including open-source proxies, managed edge services, and platform-level routers from cloud providers. Developing a custom solution is no longer necessary. As this is a relatively new area, it is important to note that routing without proper measurement may save money but can compromise accuracy or confidence. Aggressive routing is effective only if the less expensive model can reliably handle the assigned requests. This requires pairing routing with a comprehensive evaluation suite to ensure quality is maintained.
A recommended approach is to begin with observability by implementing a gateway to monitor spending, then use actual traffic data to inform routing and caching decisions. Think about it as three layers:
-
A decision layer: the router and the logic that classifies each request.
-
A model layer: a mix of frontier, small, local and specialised models to route across.
-
A governance layer: budgets, rate limits, cost attribution per team and per feature, and the evals that keep everyone honest.
The When to act is a critical decision point.
In most cases, the optimal time to implement Token Value Routing is during the transition from pilot to production, and certainly before scaling, to avoid unexpected costs. If you are still in the experimental phase with a small user base, using a single hardcoded model is acceptable. However, as AI features begin to handle higher volumes, costs can quickly become unpredictable. With the shift from assistants to agentic systems, both request volumes and token spend are expected to increase significantly.
Organisations that implement a routing layer now will be better positioned to manage costs as they scale, while those that do not may face urgent cost-optimisation efforts in the future. It is best to implement the decision layer proactively, learning from past cloud billing surprises to avoid similar issues.
