At Prepairo we were generating 10,000+ questions a day with LLMs. The bill grew faster than the product did. We cut AI infra cost by about half, and quality didn't drop. None of it was clever. All of it was boring and specific.
There are two different worlds here: pipelines (you control every call) and agents (the model decides what happens next). The tricks are different.
Pipelines: batching, caching, and both at once
Batch everything that isn't user-facing. If nobody's waiting on the response, it doesn't need to be real-time. Most providers have a batch API at roughly half the price, with results in minutes to hours. Daily content generation, backfills, evals, re-scoring old data: all batch.
Design prompts for the cache. Prompt caching gives you a big discount on tokens the provider has seen recently, but only for an identical prefix. So the order of your prompt matters:
- Instructions and rubric (never change)
- Few-shot examples (rarely change)
- Shared context, like the source article (same across many calls)
- The specific task (changes every call)
Put a timestamp or a request ID at the top and you've just turned your cache off. I've seen that exact mistake cost more than the actual feature.
Batch + cache together. Group calls that share context and send them close together. Ten questions from the same article means the article is paid in full once and cached nine times. Inside a batch, that stacks on top of the batch discount.
Route by difficulty. Not every call needs the smartest model. Classification, extraction and formatting go to a small model. Generation that needs judgment goes to a bigger one. A cheap check decides when to escalate.
Cut tokens you don't need. Trim inputs to what the task needs. Ask for structured output instead of prose you'll parse anyway. Don't ask for reasoning you'll throw away.
Agents: the tool system is the bill
Agents are trickier because every turn re-sends the whole conversation. Cost scales with context, and context scales with your design choices.
Don't inject everything into the system prompt. If you have 40 skills and 12 MCP servers, dumping every tool schema and skill doc into the prompt costs you on every single turn, and makes the model worse at picking. Give the agent a search instead: a small tool that finds the right skill or tool when it needs one, and loads only that.
Cap tool results. A grep that returns 4,000 lines or a file read that dumps 2MB will sit in context for the rest of the session, getting re-billed every turn. Truncate, and tell the model how to get more: "showing 200 of 4,000 matches, refine the pattern or use offset." The model handles that fine.
Protect the cache hit rate. Keep the system prompt and tool list stable for the whole session. History should only ever be appended to, never rewritten in the middle. Anything dynamic (time, environment info) goes at the end, not the top. A session with a high cache hit rate can be several times cheaper than the same session without it.
Design fewer, better tools. Each tool costs tokens in every request. Five composable tools beat twenty narrow ones, for cost and for accuracy. Tool outputs should be compact and structured, written for a model to read, not a human.
The mindset
Treat tokens like you'd treat database queries. You wouldn't SELECT * a million rows on every page load. Same thing here: know what's in the context, why it's there, and whether you're paying for it twice.
The best part is most of these make the system better, not just cheaper. Smaller contexts mean fewer distractions for the model. Fewer tools mean better tool choice. Cheap and good aren't opposites here. Sloppy is just expensive.