Cutting token usage without breaking the answer

Token usage in a product like ours doesn't accumulate evenly. Almost none of it is the user's question: that's a sentence. Nearly all of it is what gets built around the question before it reaches the model: the schema, the conversation so far, and the data coming back. Cut the wrong one of those and you save money by making the answer worse, which is a trade nobody actually wants, even if the bill goes down.

So the useful version of "reduce token usage" isn't a single trick. It's knowing which of these three is padding and which is load-bearing, because two of them look interchangeable and aren't.

Don't send the schema. Send eight tables.

The naive approach to "let the model write SQL against this database" is to paste the whole schema into the prompt. It works on a demo with twelve tables. It falls over, in both cost and accuracy, on a production database with four hundred, most of which have nothing to do with the question being asked.

The fix isn't a smaller schema dump. It's not sending the schema at all, and retrieving the handful of tables that are actually relevant instead: embed the question, look up the nearest table descriptions in a vector index, hand the model eight tables instead of four hundred. This is the same shape as retrieval-augmented generation everywhere else. The model doesn't need to know the whole world; it needs the eight facts that bear on this one question.

The side benefit matters as much as the token count. A smaller, relevant schema is also a more accurate one to generate SQL against. Four hundred tables full of near-duplicate column names is where a model picks the wrong status column, not just where it gets expensive.

Cap the conversation, but cap it as two different things

A multi-turn conversation is the other place tokens quietly compound. Turn ten of a chat is paying for turns one through nine, every time, unless something bounds it.

The obvious fix is a sliding window: keep the last N messages, drop the rest. The detail that's easy to miss is that a conversation isn't one history, it's two, and they don't want the same window.

The prose history, what the user asked and what they were told, needs enough turns for a follow-up like "what about last quarter instead" to resolve, but doesn't need to go back twenty messages to do it. The SQL history, the sequence of questions paired with the SQL that answered them, needs even less. A model correcting its own SQL from feedback rarely needs more than a handful of prior turns to see the pattern, and every additional turn is another full query's worth of tokens paid on every subsequent call. Windowing them separately, at different lengths, costs less than one shared window sized for the least forgiving case.

The truncation trap

This is the one that actually breaks something if you get it wrong, and it's the reason this post isn't just a list of wins.

A query can return ten rows or ten million. Sending ten million rows to a model for summarization is absurd on cost grounds alone, so the obvious move is to cap it: show the model the first few hundred rows and let it summarize those.

Do that naively and the model will confidently tell you the wrong total. Language models are unreliable at counting rows out of a table they're reading, and a display capped at 500 rows with no further signal reads to the model exactly like a complete result of 500 rows. It doesn't know it's looking at a fraction, so it answers as if it isn't. "The model quietly made up a number" is a worse failure than a slow response, because nothing about the output looks wrong.

The fix is not "send more rows." It's sending fewer rows plus one piece of information a model can't get by counting: the real total, computed by the database directly, stated in the prompt in words a small model won't misread. Something closer to "showing the first 500 of 48,213 rows; state the total as 48,213, computed exactly, never as 500." Same principle as verifying outside the model: the number the user sees should come from a COUNT(*), not from a language model's impression of a truncated table. Cutting tokens here is free. Cutting tokens and the exact-total guardrail together is not; that's the pairing that actually costs you.

Cache the answer, not just the tokens

The cheapest tokens are the ones you never spend twice. If the same question gets asked again, a page refresh, a second person on the same team, a dashboard tile re-rendering, re-running SQL generation, execution, and summarization from scratch spends the full cost again for an answer that's already sitting there.

Caching the finished answer for a short window, keyed on the question and its context, means a repeat question is close to instant and free rather than a second full pass through the pipeline. It's a small window on purpose: long enough to absorb the refresh-and-retry pattern, short enough that nobody is looking at data that's gone stale without knowing it.

What we haven't fixed yet

The honest gap: every one of these calls still pays full price to re-send the system prompt and instructions on every turn. Some model providers support marking a prefix of the prompt as cacheable, so a long, mostly-static instruction block costs full price once and a fraction of that on every subsequent call in the same session. We don't have that wired up yet.

It's the most mechanical win on this list, no risk to correctness, unlike the truncation trap above, and it's also the one that requires the least judgment to get right, which is usually a sign it should have been done already. Worth naming rather than leaving implicit: a token-usage post that only lists what's shipped reads as a highlight reel, and the gap is a more useful thing to know than another confirmed win.

Where to start

If you're looking at your own token bill, check these in order. Are you sending a model context it didn't ask for (whole schema instead of retrieval)? Are you sending it context twice (unbounded history instead of a window)? Are you sending it less information without telling it so (a truncated result with no signal that it's truncated)? The first two are pure savings. The third is the one worth double-checking before you ship it: cheaper and wrong is not actually cheaper.