The Cost of Intelligence

When OpenAI launched the GPT-3 API in March 2020, the per-token pricing meant only well-funded startups could afford to experiment. Five years later, generating a million tokens costs roughly a thousandth of what it did then. The trend isn't slowing, and if you're building anything on top of these models, the curve matters more than the headline benchmarks.

A brief price history

Inference pricing has dropped pretty consistently:

Model	Date	Input (per 1M tokens)	Output (per 1M tokens)
GPT-3 (davinci)	Jun 2020	$60.00	$60.00
GPT-3.5 Turbo	Mar 2023	$2.00	$2.00
GPT-4	Mar 2023	$30.00	$60.00
GPT-4 Turbo	Nov 2023	$10.00	$30.00
GPT-4o	May 2024	$5.00	$15.00
GPT-4o mini	Jul 2024	$0.15	$0.60
Gemini 1.5 Flash	May 2024	$0.075	$0.30
Gemini 2.0 Flash	Feb 2025	$0.10	$0.40
DeepSeek V3	Dec 2024	$0.27	$1.10
Llama 3.1 405B (hosted)	Jul 2024	$0.80	$0.80

Per-million-token pricing across major models, 2020–2025. Y-axis is logarithmic — three orders of magnitude separate GPT-3 (2020) from Gemini 1.5 Flash (2024). Hover any point for the model and exact price.

The pattern is the same every cycle. A new frontier model launches at a premium, then within a few months a cheaper variant lands at 10x to 100x lower cost. The "good enough" tier keeps getting both cheaper and better at the same time, which is the part most cost projections still get wrong.

What's actually driving the drop

It isn't one thing. The price collapse is the compounding effect of improvements stacked across the inference pipeline, and you really have to look at each layer separately to understand why the curve looks the way it does.

Hardware

Each GPU generation delivers roughly 2-3x the inference throughput per dollar. The H100 does about 3x the transformer inference of the A100 at similar power draw, and the B200 pushes that further. Custom silicon is doing even better in narrow regimes — Google's TPU v5e and Amazon's Trainium2 are designed for high-throughput inference rather than the general training-and-inference role GPUs play, and for specific architectures they often beat NVIDIA on cost per token.

Quantization

Full-precision FP32 inference is essentially gone in production. The progression most teams have followed:

FP32 → FP16 → BF16 → INT8 → INT4 → GPTQ/AWQ/GGUF

Each step roughly halves memory and often improves throughput. AWQ and GPTQ in particular hold up surprisingly well — INT4 quantization of LLaMA 70B typically shows less than 1% degradation on standard benchmarks versus FP16, and a 4-bit quantized 70B model fits on a single 80GB GPU instead of needing two. For most production workloads the quality hit is invisible.

Distillation

Training a smaller model to mimic a larger one has become standard practice. GPT-4o mini isn't just a smaller architecture; it's a distilled model that captures most of GPT-4's behavior at a fraction of the parameter count. This is probably the single biggest reason quality-per-dollar has moved as fast as it has.

The recipe is straightforward. Generate millions of input-output pairs from a large teacher, then fine-tune a smaller student on that data. The student picks up the teacher's output distribution without needing the teacher's parameter count.

Batching and serving infrastructure

Continuous batching, speculative decoding, PagedAttention as implemented in vLLM, KV-cache optimization — these aren't research curiosities anymore, they're how production inference works. Early LLM serving was embarrassingly inefficient. GPUs sat idle during the autoregressive decode phase, and you could watch utilization graphs flatline. Modern serving keeps utilization above 80% by scheduling requests intelligently.

Self-hosting vs API pricing

The build-vs-buy decision depends on volume, latency, and which model you actually need.

API pricing wins when your volume is below roughly 50-100M tokens per day, you need a frontier model like GPT-4 or Claude Opus, you don't want to manage infrastructure, or your traffic is spiky rather than sustained.

Self-hosting wins when you're processing hundreds of millions of tokens daily at steady volume, a smaller open model in the 7B-70B range meets your quality bar, you need sub-50ms latency or custom modifications to the model, or data privacy rules out sending data to a third party.

The break-even point keeps shifting toward higher volumes as API prices drop. In 2023, self-hosting made sense above roughly 10M tokens/day. By late 2025 that threshold is closer to 100M tokens/day for most cases, because API providers capture infrastructure improvements faster than individual teams can replicate them.

What cheap inference actually changes

The compounding cost reduction has concrete consequences for who can build what.

At $60 per million tokens in 2020, only well-funded companies could experiment. A chatbot handling 1,000 conversations per day cost $2,000-5,000/month in API fees alone, before you paid for anything else.

At $0.15 per million tokens in 2024, the same chatbot costs $5-10/month. A solo developer can build and run an AI-powered product for the cost of a coffee per day.

On the current trajectory, by 2026 or 2027 the cost of a typical LLM call will be roughly the cost of a database query. That changes the calculus for every software product. Adding AI features becomes a UI decision, not a cost decision.

You can already see this in the market. Products that would have been "AI startups" in 2023 are now features inside existing products. Email clients summarize threads. Code editors autocomplete functions. CRMs draft follow-ups. When intelligence is cheap, it becomes infrastructure, and the differentiation moves elsewhere.

The "$0 inference" horizon

Where does this end? There are real physical limits. Matrix multiplications need energy and energy costs money. But several trends point toward inference becoming a rounding error for most applications.

On-device inference cuts out the network round trip and the API margin. Apple, Google, and Qualcomm are shipping NPUs capable of running 3-7B parameter models locally at zero marginal cost per query.

Caching and pre-computation eliminate redundant work. A lot of LLM queries are variations of the same intent, and semantic caching can serve repeated intents without a full forward pass.

Amortization at scale matters too. As AI becomes ubiquitous, per-query cost gets absorbed into platform cost the way CPU cycles and bandwidth already are.

My read is we're 2-3 years from the point where inference cost stops being a meaningful line item for most software products. If that's right, the value shifts entirely to data, distribution, and product experience. The moat isn't the model. It's everything around it.