Engineers are now adding a response‑caching layer in front of large language model (LLM) calls to avoid re‑executing identical or near‑identical queries. By fingerprinting the request, its context, model settings, and any source data, a cache can return a previously generated answer instead of invoking the model again, cutting token‑based charges and compute load.
Why LLM response caching matters for engineers
LLM APIs bill per token, and most providers treat every request as new, even when the inputs have not changed. In production pipelines this leads to repeated work that appears harmless because the output is correct, yet it inflates costs and consumes resources that could be allocated elsewhere. Detecting unchanged inputs and skipping the inference step directly reduces spend and frees compute for other workloads.
Cache design patterns
Three practical tiers emerge from the source:
- Exact‑match cache: Normalize the request body, compute a cryptographic hash (e.g.,
SHA‑256), and look it up in an in‑memory store such as Redis. A hit returns the stored answer instantly. This works well for batch jobs, CI runs, and repetitive summarisation tasks where the request shape is stable. - Semantic‑match cache: When exact matches are too strict, embed the query with an embedding model, store the vector in a vector database, and on a miss perform a nearest‑neighbour search. A cosine‑similarity threshold (commonly 0.90–0.95, but tuned per model and data) decides if a close answer can be reused. Practitioners must verify the distance metric semantics of their vector store to avoid false positives.
- Hybrid approach: First attempt an exact‑match lookup; on a miss, fall back to semantic search. If a semantic hit is accepted, promote the result into the exact‑match store under the new query’s hash, turning future paraphrases into cheap exact hits.
Both tiers must incorporate more than the raw query text: the prompt context, document versions, model identifier, settings, and caller scope. Two identical questions that reference different documents or run under different permissions must never share a cache entry.
def cached_completion(query, ctx):
# ctx bundles everything that changes the correct answer
key = sha256(normalize(query, ctx))
if hit := redis.get(key):
return hit
emb = embed(query)
match = vector_db.search(emb, top_k=1, filter=scope_of(ctx))
if match and same_scope(match, ctx) and match.score >= threshold_for(category(query)):
remaining = match.expires_at - now()
redis.set(key, match.response, ttl=remaining)
return match.response
# fallback to real LLM call (omitted)
Operational and security considerations
Implementing a cache introduces state that must be kept consistent with upstream data. Practitioners should define a freshness policy—TTL or explicit invalidation when source documents change—to avoid serving stale answers. Monitoring cache hit‑rate and miss‑rate helps gauge effectiveness and spot unexpected drift.
From a security perspective, the cache key must include the caller’s access scope. Without this, a user with broader permissions could retrieve a response generated for a more restricted user, violating confidentiality. Additionally, any stored responses should respect data‑retention policies; sensitive outputs may need encryption at rest or short TTLs.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
Start by instrumenting a hash of the full request payload and storing responses in Redis for low‑latency exact matches. Extend the pattern with a vector store if your workload shows many paraphrased queries. Ensure the cache key incorporates context, model version, and caller scope, and define clear invalidation rules tied to source data changes. Track cost savings and cache metrics to justify the added complexity and to fine‑tune similarity thresholds.
