The Cost of Intelligence

Imagine adding a draft reply to every incoming support ticket. For a worked example, give the model 2,000 input tokens containing the ticket and relevant policy, then allow 500 output tokens for its reply. At GPT-4's historical prices, 1,000 drafts a day would cost $2,700 over a 30-day month. At GPT-4o mini's launch prices, the same token budget would cost $18.

That difference can change whether a small product gets the feature at all. But there is a condition hiding inside "the same": the cheaper model must produce replies you can use. A beautifully priced draft that invents a refund policy creates work for the person who catches it, or trouble for the customer who doesn't.

Price the task before pricing the model

This is an illustrative calculation, not a benchmark of either model. Assume one request per ticket, no retries, no cached input, and exactly the token counts above. Tokens are the pieces of text the service processes; input includes the instructions and supporting material, while output is what the model generates.

The historical rates are $30 per million input tokens and $60 per million output tokens for GPT-4's 8K version, versus $0.15 and $0.60 for GPT-4o mini. Those rates come from OpenAI's GPT-4 pricing comparison and July 2024 mini announcement.

The arithmetic gives a 150-fold reduction in model charges for this token budget. It says nothing about whether mini passes the support team's requirements. Different tokenizers can also produce different counts for the same text, so an actual comparison must use each service's billed usage.

I would define a usable draft before comparing prices: it answers the customer's question, stays within the supplied policy, preserves relevant dates and amounts, and hands uncertainty back to a person. Run both models against representative tickets, including exceptions and ambiguous requests. Count the drafts that meet those conditions. Then divide the total cost by that count.

A retry adds another billable request. Escalating a difficult ticket to a larger model adds that model's charges too. Human review, retrieval, logging, and the application server belong in the operating budget. The $18 figure is an inference subtotal, and that distinction matters much more than its impressive ratio.

What the historical curve tells us

OpenAI introduced its API on June 11, 2020. By 2024, developers had access to models charging cents per million input tokens. The table below preserves the article's historical price series while identifying which dates and conditions those numbers describe. These are US dollars per million text tokens, not current purchase recommendations or measurements of equivalent capability.

The GPT-3.5 announcement records its launch rate. Google's announcements document the 1.5 Flash price reduction and 2.0 Flash release. DeepSeek's V3 announcement separates its promotional period from the later rates and lists a different price for cached input.

Historical price series, USD per million tokens on a logarithmic scale. The archived dates mix release months and pricing periods: Gemini 1.5 Flash's plotted rate began in August 2024, and DeepSeek V3's in February 2025. GPT-3 and hosted Llama entries lack verified rate provenance. Read the qualifications in the table; these points do not measure equal-quality work.

The spread is substantial, but a line through these points would mix frontier models, small models, different providers, and price changes. Even the dated rates do not fall at every step: Gemini 2.0 Flash's listed text prices exceed the reduced 1.5 Flash rates. A newer model may justify that difference by doing more of your work correctly. The useful trend is the expanding choice of inexpensive models to test, not a guarantee that every new release lowers every bill.

How inference gets cheaper

Several engineering improvements help explain why providers can offer lower prices. They reduce different kinds of waste, and their benefits depend on the workload. A retail price also reflects a provider's commercial decisions; it does not reveal the provider's cost of running the model.

Hardware and memory

Accelerators can process more work when their compute, memory bandwidth, and software match the model. The A100, H100, and B200 belong to different generations, but a single multiplier such as "3x faster" tells little without the model, precision, batch size, and latency target. Buying more compute does not help much if each decoding step is waiting for data from memory.

Custom accelerators offer another route. They also need workload-specific comparisons. For example, AWS describes Trainium2 instances as supporting both training and deployment of large models.

Quantization

A model's weights are numbers. Storing each number with fewer bits reduces the memory needed to hold and move them. The familiar shorthand is:

FP32 → FP16 → BF16 → INT8 → INT4 → GPTQ/AWQ/GGUF

Read that as a list of related terms, not a sequence of equal improvements. FP16 and BF16 both use 16 bits. INT8 and INT4 describe bit widths; GPTQ and AWQ are quantization methods, while GGUF is a model file format. They cannot all be successive halvings of memory.

For 70 billion weights, 16 bits per weight is about 140GB before runtime overhead; four bits is about 35GB before quantization metadata and runtime overhead. That can make a model practical on one 80GB GPU, but the context cache and concurrent requests still need room. Weight size alone is not a capacity plan.

The GPTQ and AWQ papers show why low-bit weights can preserve useful accuracy. They do not establish that every 70B model loses less than 1% on every task. For our support example, the relevant test is whether the compressed model still handles the policy exceptions correctly.

Distillation

Distillation trains a student model using a teacher's outputs or predictions. It can concentrate useful behavior into a model that costs less to run, as the original distillation paper demonstrates in its own experiments.

For language models, one approach is to generate training examples with a capable teacher and fine-tune a smaller student. The saving pays off when the student handles the intended requests well enough. It may still need help on unfamiliar cases.

Serving more requests with the same hardware

Continuous batching lets a server admit new requests as others finish, instead of waiting for a whole group to complete. PagedAttention manages the memory used to retain earlier attention information, usually called the KV cache. Both can help a server fit more useful work onto its GPUs. Speculative decoding takes a different approach: a smaller model proposes tokens for a larger model to verify.

The vLLM launch benchmarks reported up to 24 times the throughput of HuggingFace Transformers and up to 3.5 times that of TGI under their test conditions. They do not validate this article's separate illustrative comparison:

Those three figures have no accompanying benchmark record, request lengths, precision, or concurrency settings. They should not be used to size a server or estimate savings. A meaningful serving test needs both throughput and response time: a system that maximizes tokens per second by making customers wait may be optimizing the wrong thing.

When owning the server pays

The support example also explains why there is no universal token volume at which self-hosting wins. At $18 a month in model charges, there is little room to save money by acquiring and operating a dedicated server. At $2,700, there is much more room, provided a model you can host meets the same acceptance bar.

Compare monthly API charges for the actual input/output mix with hardware rental or depreciation, electricity, operational work, and enough spare capacity to meet peak demand. For a simple break-even calculation, divide the server's fixed monthly cost by the saving per accepted task. If serving locally has no saving per accepted task, greater volume will not create a cost crossover.

Traffic matters because a dedicated GPU costs money while idle. Privacy requirements and control over model versions may justify hosting even when it costs more. Latency must be measured too: eliminating a network trip does not guarantee a complete answer in under 50 milliseconds.

The self-hosting article explores that operational choice. A useful break-even estimate needs a model, measured throughput, traffic pattern, and hardware cost.

Cheap enough for which feature?

Caching can reduce repeated work, but two questions that sound similar may concern different customers or policy versions. Reusing an answer safely requires checking what makes it valid. On-device inference can avoid an API charge and a network trip, while still consuming hardware capacity, energy, and engineering effort. Neither makes inference physically free.

Return to the support inbox. Suppose the product can allocate $50 a month to model calls for 30,000 draft replies. Under our fixed assumptions, the $18 calculation makes the feature worth evaluating; the $2,700 calculation rules it out at that budget. The remaining $32 provides room for some retries or escalations, whose actual frequency must be measured. Human review and the rest of the application still require their own budget.

That is a concrete consequence of falling inference prices: a feature can move from unaffordable to testable without needing a prediction about the price of intelligence in 2027. I would put the savings into a broader evaluation before expanding access. If the inexpensive model preserves policy accuracy, produces useful drafts, and stays within the budget once failures are counted, the lower price has bought something real: useful assistance on tickets that previously could not justify the expense.