One LLM call per meal: you type what you ate, the model returns per-organ numbers. Every LLM call is a line on an invoice, and people eat repetitively. A hundred users typing the same dish meant a hundred calls.
Adding a cache seemed obvious. Justifying it correctly took longer than writing it.
temperature: 0 does not make an LLM call cacheable
My first reason for treating the LLM response as storable was temperature: 0. Deterministic output, same input, same result, therefore safe to cache.
That reasoning doesn’t survive contact with the actual request shape. The prompt takes a list: ["kimchi stew", "rice"]. Nothing guarantees the result for that list matches the result for ["kimchi stew"] alone. Determinism says identical inputs produce identical outputs. It says nothing about whether a subset produces a subset of the output.
My cache key was one food. My call unit was a list. Those are different things, and temperature has no opinion about the gap.
The answer was in the output schema, not the sampling parameters:
items: [{ name, category, portion, scores, reason }]
totalScores: <sum of items>Each item closes over a single name. The total is arithmetic on top. The prompt accepts a list but scores each entry independently, so keying on one name changes nothing.
Two conditions, then, not one:
- Output is decomposable into per-input units
- That per-unit judgement is deterministic
temperature: 0 is part of condition 2. Without condition 1 it buys you nothing, and condition 1 is a property of your prompt design, not of the API.
Partial cache hits corrupt aggregates silently
The common case is a mix: two foods in the dictionary, one not. Send the miss to the model, merge the response with the hits.
I got this wrong on the first pass. The model’s response includes totalScores, and that total covers only what the model was given. Ship it as-is and every organ score is short by exactly the cached portion.
Nothing errors. Schema validation passes, because the shape is fine and the numbers are plausible. Worse, the error grows with cache effectiveness: the better your hit rate, the more wrong the totals.
I moved the summation server-side, after the merge.
The general rule: when you introduce partial caching, distrust every aggregate that arrives inside the response. Those values were computed in a world where the cache didn’t exist.
Guessing which resource is expensive puts the wall in the wrong place
The same change needed free and paid tiers. I reached for storage first, since videos and photos accumulate forever.
Measurement said otherwise.
| Unit cost | Typical use | |
|---|---|---|
| Video storage | ~$0.00005 / clip / month | Negligible |
| Model call | ~$0.0075 / call | 3 meals a day = $0.68 / month |
A ten-second clip is 3.33MB and object storage runs $0.015 per GB-month. You need roughly thirty thousand stored clips before storage matches one subscription. A user logging three meals a day burns a third of that subscription on inference alone.
Before I ran those numbers I had written a cap of 300 videos per month. That number came from nowhere. It came from the feeling that a limit belonged there, and it put a wall in front of something costing under one percent of what I was actually paying for.
The thing you must not call unlimited was inference, not storage.
Cache hits should not consume LLM quota
I placed the quota check before the model call, which meant dictionary-served requests still counted against the user’s monthly allowance.
That penalises exactly the wrong person. Someone who eats the same lunch every day generates zero marginal cost and watches their remaining calls tick down anyway. A quota exists to bound spend, not to count HTTP requests.
The check moved to immediately before the actual inference. A fully-cached request now completes without touching it.
Counting from records beat maintaining a counter
For the monthly video cap, the obvious implementation is a counter on the user document.
Two days earlier, in the same codebase, that pattern had already failed. Increment lived in one place, decrement in another, and they drifted. One account had seventeen real records and a counter reading zero. Another feature read that counter, so that user’s screen simply stopped responding to their own activity.
So the video count runs as an aggregation query over the records themselves. Count queries return a number without reading documents, so they’re cheap, and more importantly the number is derived from the data rather than maintained alongside it. It cannot drift.
Only the inference counter stayed a counter, because there is no collection to count. The mitigation is scope: that value is read by the server and nothing else, so if it drifts, nothing downstream drifts with it. If you must keep a counter, at least bound the blast radius. (The same “one value, two places” failure shows up all over store billing too, which I wrote about in most of the work in shipping in-app subscriptions was not code.)
What I still don’t know
The $0.0075 per call is a token-count estimate, not a figure from an invoice. The 3.33MB clip size is a sample of one, because the field recording file sizes is only days old. I trust the orders of magnitude and not the leading digits.
I also don’t know the hit rate yet. People spell food differently, and free-text keys may match far less often than the design assumes. That takes days of real traffic to measure. What I can say today is only that the cache doesn’t change the answer.