Emerging Economics: Token Pricing and the Case for Local Inference

Part 4 in the Demystifying Recent AI Advances Series

The previous three posts in this series covered what the models are, how much autonomy to grant them, and how to ground them in your data. This one covers the constraint that quietly governs all three decisions: cost.

For the first few years of the generative AI era, cost was mostly an afterthought. Providers competed for adoption with flat-rate subscriptions and steadily falling per-token prices. Most organizations assumed inference would keep getting cheaper. Pilot projects were budgeted accordingly. In other words, they were often barely budgeted at all. A proof of concept that burned through a few dollars of tokens a day did not warrant a conversation with finance.

That assumption is now being tested. Low-end prices have indeed collapsed, but high-end capacity has not kept pace with demand. Providers are beginning to restructure how they charge. For organizations moving from experimentation into daily production workloads, the difference between those two trends is the difference between a rounding error and a line item that shows up in the quarterly review.

What follows is a look at where pricing actually stands today, what the recent shifts signal about where it is heading, and why a growing number of teams are taking a serious look at running models on their own hardware.

One factor accelerating the adoption of agentic AI, and simultaneously reshaping how organizations think about deployment, is the shifting landscape when it comes to inference costs.

Where Pricing Stands Today

When GPT-4 launched in March 2023, API pricing was $30 per million input tokens and $60 per million output tokens. Fast-forward to 2026, and the pricing structure looks radically different. Budget-tier models like GPT-5.4 Nano run at $0.20 per million input tokens. Google’s Gemini 3.1 Flash-Lite costs $0.25. The pricing floor has dropped roughly 300 times since GPT-4’s launch. Even flagship models have become dramatically more affordable. See Table 1 for a summary of some of these prices.

Table 1: LLM-as-a-Service pricing

The End of the Subsidized Era

But here is where the story gets more nuanced. While per-token prices for legacy and mid-tier models have fallen dramatically, demand for compute using the latest and most capable models continues to outpace supply. The AI labs and hyperscalers are beginning to revisit their pricing strategies in ways that signal a shift away from the generously subsidized flat-rate offerings that fueled early adoption.

The most visible example of this is GitHub Copilot’s recent shift from flat-rate subscriptions to usage-based billing tied to actual token consumption. Under the old model, a quick chat question and a multi-hour autonomous coding session cost the same flat fee. Under the new model, each interaction is metered by tokens consumed, with per-model multipliers that determine how fast credits drain. The impact was stark. Claude Opus 4.7, one of the most capable models available in Copilot, went from a 7.5x multiplier to 27x. This means a user’s monthly credit allocation could be exhausted in as few as three premium requests.

As a result, some companies are suddenly blowing past AI inference budgets by orders of magnitude. In engineering departments, inference costs are approaching 10% of total headcount costs. The developer community reacted sharply, noting that the new system forces conscious model-selection trade-offs that did not previously exist.

Anthropic has experienced similar growing pains on the supply side. Heavy demand for Claude’s most capable models has led to periodic compute constraints, with users encountering rate limits and throttling during peak usage periods. When millions of developers and enterprise customers compete for the same GPU clusters, even well-provisioned infrastructure hits ceilings. These constraints remind us that raw cost per token is only part of the equation. The availability and reliability of that compute matter just as much.

As AI usage matures from experimentation into daily production workloads, providers will increasingly push consumers to bear the full cost of usage on a per-token basis. They are likely to roll back the generous flat-rate pricing that characterized the land-grab phase of AI adoption. The early “all you can eat” era was likely a customer acquisition strategy, not a sustainable business model. Organizations that plan their AI infrastructure around the assumption of perpetually cheap, unlimited cloud inference may suddenly find themselves caught off guard and facing ballooning IT costs.

Spending Tokens Where They Count

One solution is to encourage developers to be judicious about which models they use for certain tasks. We need to emphasize that not every request requires top-tier frontier model intelligence. This is by no means meant to be an authoritative guide. However, recent research points to a more cost-effective approach. Top-tier models can handle initial planning and help create detailed implementation specifications. Mid-tier models can then execute against those specifications, often producing high-quality results at a lower cost than relying on a single model offering. Table 2 summarizes a typical approach to managing this hybrid approach.

Table 2: Matching model tier to task type

The Case for Local Inference

The trend in rising per-token costs is also driving a growing interest in local and on-premises inference, powered by four converging forces:

• Cost predictability: Local inference decouples your AI costs from external providers’ pricing decisions. Once the hardware is in place, your per-token cost is fixed and under your control, regardless of upstream multiplier changes or rate-limit adjustments.
• Data sovereignty: Local processing eliminates cross-border data transfers, automatically satisfying GDPR data residency and HIPAA constraints. Data never leaves organizational infrastructure.
• Latency and availability: On-premises inference eliminates network round-trips and, crucially, eliminates your dependence on a provider’s capacity constraints. Your models are available when you need them, not when the provider’s GPU cluster has headroom.
• Regulatory pressure: The EU AI Act and sector-specific standards increasingly favor architectures where organizations maintain full control over their AI processing pipeline.

The local inference ecosystem has matured rapidly to support this shift. This shift is driven less by any single runtime and more by a wave of open-source techniques that shrink the compute and memory footprint required to host a model locally. Three are worth understanding in particular, because they attack different bottlenecks and compose well with one another.

Local Inference Optimizations

Model quantization is one of the most widely used techniques for making AI models smaller, faster, and less expensive to run. AI models normally store billions of numerical values (called weights) using 16 or 32 bits of precision. Quantization reduces this precision to 8 bits, 4 bits, or even fewer. This significantly reduces memory requirements while preserving most of the model’s capabilities. Think of it like compressing a high-resolution photograph. By reducing subtle color variations that are barely noticeable, we can create a much smaller file without significantly affecting image quality.

Quantization applies a similar principle to AI models. It simplifies how numerical values are stored while preserving the most important information. Modern techniques carefully identify which values require greater precision or allow the model to adapt during training, helping minimize accuracy loss. The benefits can be substantial. A 7-billion-parameter model requiring approximately 14 GB of memory at 16-bit precision might need only around 4 GB when quantized to 4 bits. This can make the difference between requiring an expensive data center GPU and running the model on a consumer GPU or capable laptop. Quantization can also improve processing speed by reducing the amount of data that must move through memory when generating responses. However, actual performance gains depend on the hardware and software.

TurboQuant and similar newer quantization techniques take compression a step further by finding smarter ways to represent numerical values using even less memory. Traditional quantization reduces precision using relatively simple rules. These newer approaches adapt to the data’s patterns and distribution to preserve more information. Think of it like packing a suitcase. Instead of forcing everything into identical-sized compartments, you arrange items by shape and size, giving delicate or unusually large items special treatment while packing everything else more tightly. Similarly, TurboQuant mathematically reorganizes numerical data before compressing it. This helps preserve important information while reducing storage requirements.

One particularly valuable application is compressing the key-value cache (KV cache). This acts as the model’s short-term memory during conversations and grows as it processes more text. By shrinking this cache, AI systems can handle longer conversations or more simultaneous requests without requiring additional GPU memory. These techniques make extremely compact 3–4-bit representations increasingly practical. However, pushing compression too far can come at a cost. Below about 4 bits, some models may experience noticeable reductions in accuracy or processing speed. This is especially true for handling lengthy documents or complex reasoning tasks.

Multi-Token Prediction (MTP) attacks a different bottleneck entirely. Think of it like meta-prediction or predicting the prediction. Standard autoregressive decoding emits exactly one token per forward pass. So generating a 500-token answer means 500 sequential trips through the entire model. That is inherently serial, leaving much of the available compute idle while the hardware waits on memory.

MTP trains the model with additional prediction heads so that a single pass proposes several future tokens at once. It then verifies those extra tokens in one batched pass and accepts them when they match what ordinary decoding would have produced. This cuts the number of sequential steps without changing the output. Fewer passes mean lower latency and lower compute cost for the same volume of generated text. MTP is a training objective rather than a post-processing step, so it can also improve model quality as a side effect. By predicting several tokens ahead, the model is forced to plan beyond the very next word.

Together, these open-source advances mean a well-quantized model using MTP-style decoding can now deliver latency and throughput on commodity hardware that would have required a full data center GPU just a couple of years ago. And the SLM thesis from the earlier series makes local deployment particularly attractive. A 3 billion parameter model running on a laptop is a very different proposition than self-hosting a 70 billion parameter behemoth. For many of the specialized tasks within an agentic workflow (extraction, classification, guardrail checks, structured output generation), a well-tuned local SLM can match or exceed the performance of a cloud-hosted LLM at a fraction of the cost and with none of the supply constraints.

None of this means cloud APIs are going away. For frontier reasoning tasks, agentic coding harnesses, and organizations early in their AI journey, cloud inference often remains the most practical starting point. But the writing is on the wall. Organizations that want to run AI at scale, reliably and affordably, may need a hybrid strategy that pairs cloud APIs for tasks that demand frontier-scale models with local inference for the high-volume, specialized work that does not. These core concepts are summarized in Table 3.

Table 3: Local inference and hybrid strategy

For most organizations, the practical next step is not a wholesale migration but an inventory. Which of your current AI workloads are high-volume, narrow, and well-specified? And which genuinely require frontier-scale reasoning? That question is answerable today, and the answer often surprises people. A large share of production token spend goes to classification, extraction, routing, and formatting work. This is exactly the category where a well-tuned local SLM performs as well as a frontier model at a small fraction of the cost. Quantifying that share is what turns a hybrid strategy from an architectural aspiration into a budget line you can actually defend. It is worth doing before the next round of pricing changes, not after.

Coming Up Next

That covers the concepts: models, autonomy, plumbing, retrieval, and economics. The final post in this series shifts from theory to practice and shows how SAS is putting these ideas to work through Viya Copilot, the SAS Viya MCP Server, Retrieval Agent Manager, the Agentic AI Accelerator, and AI Navigator, with governance built in rather than bolted on.

Additional Resources

Blog Series: