You are about to commit budget to a feature that calls a model thousands of times a day, and the only number in front of you is a price per million tokens. That number is real, but it does not tell you what the feature will cost. The cost of AI inference in production is controlled by five things: token volume, model choice, context length, calls per task and hardware utilisation. A per-token rate tells you what one unit costs inside a narrow boundary, not what your workload will cost.
Inference cost is the cost of running data through a trained model to produce an output. [1] It is charged every time a model generates output, and it runs every time a user sends a request. [2] [1] Five inputs set the number: token volume, model choice, context length, calls per task and hardware utilisation. [2] Everything below builds the boundary those five inputs sit inside.
Key takeaways
- Inference cost is what you pay each time a trained model runs data to produce an output, not what you paid to train it. [1] [2]
- The unit changes with the modality: dollars per million tokens for LLMs, dollars per image for image generation, and per second for audio and video. [3]
- Five inputs set the number: token volume, model choice, context length, calls per task and hardware utilisation. [2]
- Token-only figures exclude caching, retries, tools, storage, tax, hosting and human review. They do not measure cost per accepted task.
- Per-token prices have fallen steeply, but the fastest drops are the most recent and the least likely to persist, so treat decline as an assumption you state, not a plan you fund. [1] [4]
What is AI inference cost, and what does the figure measure?
Inference cost measures the working stage of a model, not the building stage. It is the cost of running data through a trained model to produce an output: a prediction, a generated response or a classification. [1] Training is a sunk event. Inference runs every time a user sends a request. [1]
The system boundary includes more than the request itself. Inference cost is the total compute, memory and infrastructure expense of running that working stage. [5] It covers accelerator time spent generating each output, the memory that holds the model in place, the networking that moves requests in and responses out, and the platform overhead that keeps endpoints available. [5] A headline price per million tokens compresses all of that into one number and hides which part of it moves when your traffic changes.
Which unit is the figure measured in, and does it match your workload?
The unit depends on what the model produces. Inference cost is measured in dollars per million tokens for LLMs, dollars per image for image generation, and per second for audio and video. [3] If your product generates images and you budget in tokens, the unit and the workload do not match, and the forecast fails before the first request.
For text workloads, the driver underneath the unit is tokens, the units of data models process. [1] Tokens are the meter. Your job is to know how many of them one completed task consumes, not how many one call consumes.
Prices and product terms move. We date every mutable price we publish, and we expect the same discipline from you: check the rate in the current provider page or quote before it enters a plan. Our methodology sets out how we handle checked dates on figures that change.
What drives the production inference bill?
Five things set the number: token volume, model choice, context length, calls per task and hardware utilisation. [2] Four of those five are yours to control. Only the posted rate belongs to the vendor.
A wider view names the same mechanics. Cost drivers include model size, token volume and context length, hardware choice, runtime efficiency and scaling behaviour, and the optimisation levers span the model, runtime, infrastructure and platform levels. [1]
The constraint is usually calls per task and context length, not price per token. A retrieval step that stuffs 40,000 tokens of context into every call, or an agent loop that makes six calls where one would do, multiplies the meter regardless of which vendor you picked. Constraints reveal more than announcements. Read your own call graph before you read the next price cut.
What does a per-token rate exclude, and where does the rest of the cost sit?
It excludes most of the operating cost. The calculation divides the sum of billed input tokens times input rate and billed output tokens times output rate by 1,000,000. These token-only figures exclude caching, retries, tools, storage, tax, hosting and human review. They do not measure cost per accepted task.
The cost moves to the parts nobody prices on the pricing page. Retries on failed outputs. Tool calls and the services they hit. Vector storage. The person who checks the output before it reaches a customer. Total cost of ownership formulas handle this by incorporating hardware depreciation, hosting fees and software licensing to produce a yearly server cost, a cost per 1,000 prompts, and a cost per 1 million input and output tokens. [6] That structure is the one to copy, because it forces the boundary to be stated.
How do you compare model prices without misreading the comparison?
Hold the workload constant and let only the rate vary. The scenario below is an illustrative billing scenario, not a benchmark: 10,000 billed input tokens and 2,000 billed output tokens per call, with 1,000 calls per day. Billed output includes reasoning where charged. Equal tokens do not imply equal text, quality or completion rates. Standard paid direct API rates checked 9 September 2026; short context for OpenAI.
| Model | Input USD per million | Output USD per million | USD per call | USD for 1,000 calls per day |
|---|---|---|---|---|
| GPT-5.6 Terra | 2 | 12 | 0.044 | 44.00 |
| Claude Sonnet 5 | 2 | 10 | 0.040 | 40.00 |
| Gemini 3.8 Flash | 0.75 | 3.75 | 0.015 | 15.00 |
The posted rates behind those rows carry their own conditions. GPT-5.6 Terra on the direct API standard short context rate lists input at $2.00, cached input at $0.20, cache write at $2.50 and output at $12.00 per million tokens, checked 9 September 2026. [7] Claude Sonnet 5 on the direct API standard rate lists input at $2.00, a five-minute cache write at $2.50, a one-hour cache write at $4.00, a cache hit at $0.20 and output at $10.00 per million tokens. [8] Gemini 3.8 Flash on the direct API standard paid tier lists input at $0.75, output including thinking at $3.75, cached input at $0.075 and cache storage at $0.50 per million tokens per hour, on promotional rates through 31 December 2026. [9]
The decision changes when completion rates differ. A cheaper model that needs two attempts on a third of tasks is not cheaper. Compare on cost per accepted task, then read the table.
Should you rent inference or self-host it?
Rent while your load fluctuates, and model self-hosting once it does not. Companies weigh building a self-hosted GPU cluster against renting from the cloud, and cloud options offer more flexibility for small to medium-scale or fluctuating workloads, while self-hosted clusters give better marginal cost advantages under sustained high load. [10]
The limiting condition is duty cycle. Owned hardware bills you for the hours it sits idle; an API does not. Estimating the infrastructure required for inference applications and the total cost of ownership is a problem enterprises face before deploying generative AI systems at scale. [6] Vendor claims on either side establish what the vendor says, not what your utilisation curve will do. Build the comparison on your own traffic profile and compare it against the same category work in our research library.
Are inference costs falling, and can you plan on that?
Yes to the first, with caution on the second. The Stanford HAI 2025 AI Index Report states that the inference cost for a system at GPT-3.5 level dropped over 280-fold between November 2022 and October 2024. [1] That is a steep decline over two years.
The evidence remains incomplete where it matters most for planning. The fastest price drops in that range occurred in the past year, so it is less clear that they will persist. [4] A plan that funds growth out of expected price cuts is a plan with an unstated assumption inside it. State the assumption, hold the current rate as the base case, and treat any decline as upside you did not spend in advance.
How do you build an inference cost figure you can commit to?
Start from the load case, not the price list. Infrastructure sizing uses latency constraints and peak request volumes to determine the minimum number of model instances and servers required for a deployment. [6] Peak volume and the latency you promise set the floor on capacity, and capacity sets cost.
Then pick the operating point. Latency-throughput trade-off curves and Pareto fronts identify optimal deployment configurations where throughput is maximised while latency stays minimal, across precision formats such as FP4, FP8 and BF16. [6] Where you sit on that curve is a commercial decision wearing an engineering label.
The estimate assumes a token profile per task, a completion rate, a retry rate and a fixed context length. Write those four assumptions down next to the number. Then run one real workflow end to end at production volume and check the figure against the invoice. The economics only close when you can say who pays, which unit governs the charge and what happens to that unit when volume triples. See how we frame the same question across other categories, or subscribe for new cost analyses.
Frequently asked questions
What is inference in AI?
Inference is the stage where a trained model produces an output. Inference cost is the cost of running data through a trained model to produce that output: a prediction, a generated response or a classification. [1] It runs every time a user sends a request. [1]
Why does running a model cost money on every request?
Because every request consumes physical resources. Inference cost covers the GPU or accelerator time spent generating each output, the memory needed to hold the model in place, the networking that moves requests in and responses out, and the platform overhead that keeps endpoints available. [5] It is the total compute, memory and infrastructure expense of running that working stage. [5]
How do you calculate cost per million tokens?
Multiply billed input tokens by the input rate, multiply billed output tokens by the output rate, add the two, then divide by 1,000,000. These token-only figures exclude caching, retries, tools, storage, tax, hosting and human review. They do not measure cost per accepted task. Test the formula against a stated scenario: 10,000 billed input tokens and 2,000 billed output tokens per call, with 1,000 calls per day, where billed output includes reasoning where charged. Equal tokens do not imply equal text, quality or completion rates.
What do LLM providers charge per million tokens right now?
GPT-5.6 Terra lists $2.00 input and $12.00 output per million tokens on the direct API standard short context rate, with cached input at $0.20 and cache write at $2.50, checked 9 September 2026. [7] Claude Sonnet 5 lists $2.00 input and $10.00 output, with a five-minute cache write at $2.50, a one-hour cache write at $4.00 and a cache hit at $0.20, on the direct API standard rate. [8] Gemini 3.8 Flash lists $0.75 input and $3.75 output including thinking, with cached input at $0.075 and cache storage at $0.50 per million tokens per hour, on the direct API standard paid tier at promotional rates through 31 December 2026. [9]
Is the cost of inference going down?
Yes. The Stanford HAI 2025 AI Index Report states that the inference cost for a system at GPT-3.5 level dropped over 280-fold between November 2022 and October 2024. [1] The fastest drops in that range occurred in the past year, so it is less clear that they will persist. [4]
Why can your inference bill rise while the per-token price falls?
Because price is one of five inputs. Token volume, model choice, context length, calls per task and hardware utilisation together set the number. [2] The cost is driven by tokens, the units of data models process, so longer context and more calls per task raise the meter even as the rate per token drops. [1]
Where are the levers to reduce inference cost?
Work the drivers, in order of size. Cost drivers include model size, token volume and context length, hardware choice, runtime efficiency and scaling behaviour, and the optimisation levers span the model, runtime, infrastructure and platform levels. [1] Cut context and calls per task first, because they multiply every other charge.
Is self-hosting cheaper than paying an API per token?
It depends on load. Cloud options offer more flexibility for small to medium-scale or fluctuating workloads, while self-hosted clusters give better marginal cost advantages for sustained high-load scenarios. [10] Compare them on the same basis: total cost of ownership formulas incorporate hardware depreciation, hosting fees and software licensing to produce a yearly server cost, a cost per 1,000 prompts and a cost per 1 million input and output tokens. [6]
How do you size infrastructure for an inference workload?
Start with latency constraints and peak request volumes, which determine the minimum number of model instances and servers a deployment requires. [6] Then select the operating point using latency-throughput trade-off curves and Pareto fronts, which identify configurations where throughput is maximised while latency stays minimal for precision formats such as FP4, FP8 and BF16. [6]
How do you tie inference cost to a product or feature?
Fix one workload, price it across candidate models, then attach the result to the feature that generates the calls.
| Model | Input USD per million | Output USD per million | USD per call | USD for 1,000 calls per day |
|---|---|---|---|---|
| GPT-5.6 Terra | 2 | 12 | 0.044 | 44.00 |
| Claude Sonnet 5 | 2 | 10 | 0.040 | 40.00 |
| Gemini 3.8 Flash | 0.75 | 3.75 | 0.015 | 15.00 |
Source notes
Sources checked
Vendor-owned pages establish current product and pricing claims. They do not prove comparative superiority. Glamdring states the source class so readers can judge each claim.