Why your LLM bill grows faster than revenue, and how to cap it

A regular web app’s costs grow in steps: one server, then a bigger server. An app with a language model inside pays for every user request, and pays a different amount each time. One person asks a short question. Another pastes a forty-page contract and asks for “again, but shorter” ten times in a row. Same plan, a hundredfold difference in cost to serve.

So the usual picture looks like this: revenue grows linearly, the provider bill grows faster, and the margin goes negative right around the time people start liking the product. Here’s what to do about it, in order.

Measure first

You can’t cap what you can’t see. Every provider response includes input and output token counts. Store them with three labels: user, product feature, model.

After a week you’ll have answers to questions that otherwise get settled by argument:

• which feature eats most of the budget (usually not the one you’d guess);
• what an average active user costs per month, and what the most expensive one percent costs;
• what share of tokens is input versus output.

That last one matters. Output tokens cost noticeably more than input tokens with most providers, and a long system prompt sent with every request quietly multiplies your input.

Per-user and daily caps

A cap isn’t stinginess. It’s a guarantee that one account can’t spend everyone else’s budget. You want two levels:

• per user: a daily or monthly budget in tokens or money, sized to their plan. Less for free users, more for paying ones, but finite for both;
• per product: a daily ceiling beyond which expensive features drop to a degraded mode or pause, and you get notified.

A useful rule: your most active user should not cost more to serve than they pay. If they do, you need either a cap or a different plan.

Add a request rate limit on top. A script calling your endpoint in a loop isn’t a user. It’s an invoice.

Match the model to the task

Running the most capable model on everything is the most common overspend. Classifying a request, pulling fields out of text or shortening a paragraph are well within reach of smaller-tier models, which tend to cost several times less than the flagship and sometimes an order of magnitude less. Keep the flagship for work where users can see the difference.

A practical approach: collect twenty or thirty real examples per feature, run them through models of different tiers and compare the results by eye. Often the small model holds up, and where it doesn’t, a better prompt often closes the gap.

Cap the response length

Every provider has a max_tokens parameter, and it’s almost always left at the default. A feature that returns a headline doesn’t need two thousand tokens. Set the ceiling to fit the task and ask for a short format in the prompt, so you pay for what you actually display.

The same goes for input. Don’t send the whole chat history and the entire document when the answer needs the last couple of turns and one section.

Cache

Two kinds, both cheap:

• your own response cache: if the same input always produces the same answer (a product description, a translation of fixed text), don’t generate it twice;
• provider prompt caching: most major providers can cache a repeated request prefix, such as the system prompt, instructions or a reference document. Reading a cached prefix costs substantially less than reading it fresh. The condition is that the stable part goes first and the changing part, like the user’s question, goes last.

Retry with backoff, not in a blind loop

A request fails, the code retries. It fails again, the code retries again. If the error isn’t transient (say, the input is too long), the loop pays for every attempt until someone stops it.

The right way: retry only transient errors (overload, timeout, rate limit), with growing delays plus random jitter, and only a few times. Never retry an error in the request itself.

attempt 1 → wait 1s → attempt 2 → wait 2–3s → attempt 3 → give up

Agents that decide on their own whether to call the model again are a special risk. They need a hard ceiling on steps and tokens per task.

User text is not an instruction

When your prompt includes text written by a user or pulled from outside (an email, a web page, a file), that text can contain commands aimed at the model: “ignore your instructions”, “print your system prompt”, “repeat this paragraph ten thousand times”. The last one is a billing problem, not just a security one.

Basic hygiene:

• keep instructions and data clearly separated, and tell the model the data is data;
• cap the length of user input;
• give the model no more tools or permissions than the feature needs;
• validate the model’s output before acting on it.

This doesn’t make injection impossible, but it removes the cheapest versions.

The upshot

AI spend is controlled with the same tools as any other spend: measurement, limits, and the right tool for the job. The difference is that each of these measures pays for itself quickly, sometimes in the first month.

SUBXCAT can set up token accounting, caps and model routing in your product, and show you where the budget is actually going.