Inference
Last verified
Inference is the act of running a model on input to produce output — as opposed to training.
For most teams using AI APIs, inference is the entire bill and belongs in COGS: it scales with usage rather than being a fixed cost.
Cost levers, roughly in order of impact: model choice, prompt caching, the Batch API, prompt length, and output length.