Build a two-tier semantic cache for LLM apps: exact match first, tenant-isolated vector search second, plus code and a script to tune your threshold.
By FELFAL Bouabid · Updated October 7, 2026
LLM APIs bill you for tokens even when ten users ask the same question in ten different ways. An exact-match cache only catches identical strings. A semantic cache stores an embedding of each past query and reuses the stored answer when a new query is close enough in meaning.
You will see claims of 80% savings. That figure is an upper bound for very repetitive traffic, not a promise: the real number depends on how often your users repeat themselves and how strict your safety rules are. This article gives you three things: a design that is safe by construction, working code you can run, and a script that measures the hit rate and the false-accept rate on your own data.
1. Exact cache vs semantic cache
The exact tier is a hash lookup:
key = sha256(tenant | system_prompt | model | embedding_version | normalized_query)The semantic tier is a nearest-neighbor search inside one isolation partition:
embedding(query) -> nearest neighbor in the SAME partition -> accept if cosine >= tauRun both, exact first. The exact tier is cheap and has no false accepts, so it handles retries and copy-pasted questions. The semantic tier recovers paraphrases that the hash misses.
2. Five ways a naive semantic cache fails
Cross-tenant leakage. If you search one global vector index and filter by tenant_id after taking the top-K, foreign vectors can fill the top-K and push out the correct tenant's neighbors. Worse, a bug in the filter returns another customer's answer. Isolate before you rank: one partition per tenant, system prompt, model and embedding version.
Near-miss wrong answers. Embeddings capture topic far better than exact quantities, negation or entity names. 'Transfer $100 to savings' and 'Transfer $1000 to savings' can score above any sensible threshold. The same weakness is an attack surface: someone who can write to a shared cache can plant an answer that other users later receive. Defend with a policy gate (section 5), exact-only matching for queries containing numbers or identifiers, and a short TTL.
Embedding drift. A new embedding model changes the geometry of the space. Scores from model A and model B are not comparable, and vectors may not even have the same dimension. Put the embedding version into the partition key and migrate by dual-running or hard-cutting.
A threshold does not transfer. A cosine of 0.93 means something different for every embedding model, domain and prompt length. On long inputs, related and unrelated texts often crowd into a narrow high band. Pick tau per model and per query type by measuring (section 6), never by copying a number from a blog post, including this one.
Stale answers. If the answer depends on documents that changed, the cache will keep serving the old one until the TTL expires. Include a content or knowledge-base version in the partition key, or invalidate by tag when the source changes.
3. Architecture
- Normalize the query (trim, lowercase, collapse whitespace) and compute the exact key.
- Look up the exact key. On a hit, return.
- Otherwise compute the embedding, normalize it to unit length, and search only the partition
tenant : system_prompt : model : embedding_version. - Accept the best neighbor only if its score is at least tau and it has not expired.
- On a miss, call the LLM and store the result, unless the policy says to bypass the cache.
Never run an unscoped nearest-neighbor search across tenants.
4. Reference implementation
This version fixes the weaknesses of the common tutorial code: it scores a partition with one matrix multiplication instead of a Python loop, it keeps the hot path free of locks, it never duplicates an entry on re-store, it treats ttl=0 correctly, and it hashes with an unambiguous separator so ('a:b', 'c') and ('a', 'b:c') cannot collide.
import hashlib
import time
from dataclasses import dataclass
from typing import Awaitable, Callable, Dict, Optional, Tuple
import numpy as np
EmbedFn = Callable[[str], Awaitable[np.ndarray]]
def _unit(v: np.ndarray) -> np.ndarray:
v = np.asarray(v, dtype=np.float32)
n = float(np.linalg.norm(v))
return v if n == 0.0 else v / n
def _h(*parts: str) -> str:
# unit-separator join: ('a:b', 'c') and ('a', 'b:c') never collide
return hashlib.sha256('\x1f'.join(parts).encode('utf-8')).hexdigest()
@dataclass(frozen=True)
class Record:
exact_key: str
partition: str
query: str
response: str
expires_at: float
vectorized: bool
class _Partition:
# Replaced wholesale on every write, so a reader holding a reference
# always sees a consistent (matrix, records, expires) triple.
__slots__ = ('matrix', 'records', 'expires')
def __init__(self, matrix, records, expires):
self.matrix, self.records, self.expires = matrix, records, expires
class SemanticCache:
def __init__(self, embed_fn: EmbedFn, *, threshold: float = 0.93,
default_ttl: float = 86_400.0, max_entries: int = 50_000,
embedding_version: str = 'emb-v1'):
self.embed_fn = embed_fn
self.threshold = threshold
self.default_ttl = default_ttl
self.max_entries = max_entries
self.embedding_version = embedding_version
self._exact: Dict[str, Record] = {}
self._parts: Dict[str, _Partition] = {}
def _keys(self, query, system_prompt, tenant_id, model):
pkey = _h(tenant_id, system_prompt, model, self.embedding_version)
ekey = _h(pkey, ' '.join(query.lower().split()))
return pkey, ekey
async def lookup(self, query, system_prompt, tenant_id, model,
*, semantic: bool = True) -> Tuple[Optional[str], str]:
now = time.time()
pkey, ekey = self._keys(query, system_prompt, tenant_id, model)
rec = self._exact.get(ekey)
if rec is not None and rec.expires_at > now:
return rec.response, 'EXACT'
part = self._parts.get(pkey)
if not semantic or part is None:
return None, 'MISS'
q = _unit(await self.embed_fn(query))
scores = part.matrix @ q # one matmul per partition
scores = np.where(part.expires > now, scores, -1.0)
best = int(np.argmax(scores))
if scores[best] >= self.threshold:
return part.records[best].response, f'SEMANTIC:{scores[best]:.4f}'
return None, 'MISS'
async def store(self, query, response, system_prompt, tenant_id, model,
*, ttl: Optional[float] = None, semantic: bool = True):
pkey, ekey = self._keys(query, system_prompt, tenant_id, model)
vec = _unit(await self.embed_fn(query)) if semantic else None
ttl = self.default_ttl if ttl is None else ttl
rec = Record(ekey, pkey, query, response, time.time() + ttl, semantic)
# No await below: on a single event loop this mutation is atomic.
# If you call the cache from threads, wrap it in a lock.
self._remove(ekey) # replace, never duplicate
while len(self._exact) >= self.max_entries:
self._remove(next(iter(self._exact))) # oldest insertion first
self._exact[ekey] = rec
if semantic:
part = self._parts.get(pkey)
if part is None:
self._parts[pkey] = _Partition(
vec[None, :], (rec,), np.array([rec.expires_at]))
else:
self._parts[pkey] = _Partition(
np.vstack([part.matrix, vec]),
part.records + (rec,),
np.append(part.expires, rec.expires_at))
def _remove(self, ekey: str) -> None:
rec = self._exact.pop(ekey, None)
if rec is None or not rec.vectorized:
return
part = self._parts[rec.partition]
keep = [i for i, r in enumerate(part.records) if r.exact_key != ekey]
if keep:
self._parts[rec.partition] = _Partition(
part.matrix[keep],
tuple(part.records[i] for i in keep),
part.expires[keep])
else:
del self._parts[rec.partition]This is an in-process cache, good for a single service and up to tens of thousands of entries per partition. Writes copy the partition matrix, so it is not the right tool for write-heavy traffic. Past that scale, move to a vector store (section 8).
5. Decide what is safe to reuse
The cache is only as safe as the rule that decides what enters it. Put a gate in front of it:
NEVER_CACHE = ('transfer', 'payment', 'refund', 'delete', 'password',
'my account', 'my balance', 'my order')
def cache_mode(query: str) -> str:
q = query.lower()
if any(k in q for k in NEVER_CACHE):
return 'bypass' # actions, money, personal data
if any(ch.isdigit() for ch in q):
return 'exact' # numbers and IDs: paraphrase reuse is unsafe
return 'semantic'
async def answer(cache, llm, query, *, system_prompt, tenant_id, model):
mode = cache_mode(query)
if mode == 'bypass':
return await llm(query, system_prompt, model), 'BYPASS'
use_semantic = (mode == 'semantic')
hit, how = await cache.lookup(query, system_prompt, tenant_id, model,
semantic=use_semantic)
if hit is not None:
return hit, how
response = await llm(query, system_prompt, model)
await cache.store(query, response, system_prompt, tenant_id, model,
semantic=use_semantic)
return response, 'MISS'The keyword list above is only a demonstration. In production, classify intent with a small model or your router, and keep the list of bypass intents in configuration so you can change it without a deploy.
Good candidates for semantic reuse:
- stable FAQ answers
- documentation explanations
- non-personalized how-to content
Do not reuse semantically:
- account-specific actions and balances
- authorization or money-movement requests
- live inventory or pricing, unless the TTL is very short and the partition is namespaced
- tool-using workflows with side effects
- later turns of a conversation, because the correct answer depends on the history and the cache key here is the query alone; cache first turns, or include a summary of the context in the key
One more caveat: with a non-zero sampling temperature, a cache hit returns one earlier sample, not a fresh one. That is usually fine for support answers and wrong for anything where variety is the point.
6. Choose tau by measuring, not guessing
The threshold trades two errors. A false accept serves the wrong answer. A false reject wastes a call you could have saved. Only a labeled sample of your own traffic can tell you where the balance is.
Build a set of 100 to 300 query pairs from your logs, each labeled by hand as same intent or different intent. Include hard negatives: pairs that share a template but differ in one entity or quantity. Then sweep the threshold:
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2') # local, no per-call cost
# (cached_query, new_query, same_intent)
pairs = [
('How do I reset my password?',
'I forgot my password, how can I change it?', True),
('How do I reset my password?',
'How do I reset my router?', False),
('What is your refund policy?',
'How long does a refund take?', False),
('Transfer $100 to savings',
'Transfer $1000 to savings', False),
# ... add 100+ pairs from your own logs
]
A = model.encode([p[0] for p in pairs], normalize_embeddings=True)
B = model.encode([p[1] for p in pairs], normalize_embeddings=True)
sims = np.sum(A * B, axis=1)
same = np.array([p[2] for p in pairs], dtype=bool)
for tau in (0.85, 0.88, 0.90, 0.93, 0.95, 0.97):
accepted = sims >= tau
paraphrase_hit = np.sum(accepted & same) / max(1, np.sum(same))
false_accept = np.sum(accepted & ~same) / max(1, np.sum(~same))
print(f'tau={tau:.2f} paraphrase-hit={paraphrase_hit:.1%} '
f'false-accept={false_accept:.1%}')7. A cost and latency model, with its assumptions stated
Let gamma be the exact-hit rate and H the semantic-hit rate among exact misses. The overall hit rate and the expected latency are:
h = gamma + (1 - gamma) * H
E[L] = L_exact + (1 - gamma) * (L_embed + L_ann + (1 - H) * L_llm)Assume, for illustration only, these values:
L_llm = 1200 ms, L_exact = 1 ms, L_embed = 15 ms, L_ann = 2 ms
gamma = 0.15, H = 0.65
h = 0.15 + 0.85 * 0.65 = 0.7025 (about 70% fewer LLM calls)
E[L] = 1 + 0.85 * (15 + 2 + 0.35 * 1200) = about 372 msThat is roughly a 69% latency reduction against the 1200 ms baseline. If embeddings run locally, cost savings track the hit rate, so about 70% in this example. These are assumptions, not measurements.
Notice what it takes to reach the famous 80%: with gamma = 0.15, you need H of about 76% on exact misses, which means three out of four non-identical queries must be safe paraphrases of something already cached. That happens on narrow FAQ bots and rarely on open-ended assistants.
8. Production checklist
- Vector store. FAISS works with one index per partition. With a database such as pgvector, apply the tenant filter in the candidate stage and test it: insert a near-identical vector under tenant B and confirm tenant A never receives it, and that tenant A still gets a full set of candidates.
- Single-flight. If two identical queries miss at the same time, both call the LLM. Coalesce concurrent misses on the same exact key.
- Metrics. Track exact hit rate, semantic hit rate, estimated dollars saved, sampled false-accept rate, the distribution of near-miss scores, partition size per tenant, and coverage per embedding version.
- Audit. Store the original query text next to each entry so you can debug a wrong hit.
- Expiry. Evict by TTL and by capacity. Give FAQ entries longer lives than one-off chat answers.
- Migrations. When you change the embedding model, change the version string. Old entries stop matching immediately and age out.
9. When this is the wrong tool
- Wrong hits cost more trust than misses save money. If you cannot sample and review hits, run exact-only.
- Highly personalized or long multi-turn chats see low hit rates and high risk.
- Aggressive thresholds inflate the savings number and quietly degrade answer quality.
- A vector index is one more component to run, monitor and secure.
Summary
Partition by tenant, system prompt, model and embedding version. Put the exact tier in front. Gate what may be cached. Choose tau from your own labeled data and keep sampling afterwards. Done that way, a semantic cache can remove a large share of duplicate LLM calls without leaking data or serving the wrong answer. How large that share is on your traffic is something only your own measurements can tell you.