Semantic Caching for LLMs: Architecture, Thresholds, Risks

Semantic caching for LLMs looks like the easiest cost win in AI engineering: put a vector store in front of the model and stop paying twice for “how do I reset my password” and “how can I reset my password?”. The trouble starts in production. In a banking FAQ test published on InfoQ, a customer who said “I don’t want this card anymore” was routed to investment account closure steps with 88.7% confidence. The cache was fast, cheap and wrong.

This guide treats the semantic cache as a system design problem: where it sits next to prompt caching, how to scope keys, why threshold units trip people up, and which traffic should skip it. The Node.js code below was executed in our sandbox on Node v22.23.4, and its output is shown verbatim. If you already run an AI gateway with token-bucket rate limiting, the cache slots in right before the provider call.

What is semantic caching for LLMs, and how is it different from prompt caching?

Semantic caching for LLMs returns a stored response without calling the model; prompt caching still calls the model but bills the repeated prefix at a discount. They solve different problems and work well together.

Provider prompt caching is automatic or near-automatic. OpenAI’s cookbook says caching activates for prompts longer than 1024 tokens with no request changes. The same page cites up to an 80% latency reduction for prompts over 10,000 tokens. Anthropic’s documentation prices cache reads at 0.1x the base input price for most models, with a 5-minute default lifetime and a 1-hour option at extra cost.

A semantic cache works one layer higher. It answers “has a user already asked this, in any wording?” and, on a hit, there is no output-token bill at all. The cost is an embedding call, a vector search and the risk of a false hit. That risk does not exist with prompt caching, because the model still reads the full prompt.

My recommendation: structure prompts for the provider cache first (static system prompt and tools at the front), then add semantic caching only for traffic that repeats by intent. Agent workloads, covered in our single vs multi-agent orchestration guide, are where long repeated system prompts make the provider cache shine.

How does a semantic cache architecture work in an LLM gateway?

A production semantic cache is a pipeline of cheap checks before expensive ones, ending in the LLM only when every layer misses.

1. Bypass check: skip the cache for multi-turn requests, personalised prompts and very short queries.
2. Scope key: build a key from tenant, model and prompt-template version.
3. Exact layer: hash the normalised prompt within the scope and look it up. No embedding needed.
4. Semantic layer: embed the prompt and run a nearest-neighbour search inside the same scope only.
5. Threshold gate: serve the cached answer only if similarity clears a tuned threshold.
6. Single-flight miss: if identical requests miss together, let one call the LLM and make the rest wait for it.
7. Write-back: store response, vector and a TTL, then emit hit/miss metrics.

This matches the published GPT Semantic Cache design, which stores query embeddings in Redis and uses an HNSW index for approximate nearest-neighbour search. Its authors evaluated it on 8,000 question-answer pairs and 2,000 test queries. Step 6 is the same concurrency problem covered in our cache stampede prevention guide: a popular new question can trigger dozens of identical LLM calls at once.

How do you build a semantic cache in Node.js?

You can build a working semantic cache in Node.js with a hash map, a cosine function and a promise map for single-flight. Swap the in-memory arrays for Redis or a vector database in production.

import { createHash } from 'node:crypto';

const cosine = (a, b) => {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] ** 2; nb += b[i] ** 2; }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
};
export const similarityToDistance = (s) => 1 - s;

export class SemanticCache {
  constructor({ embed, minSimilarity = 0.9, ttlMs = 3_600_000 }) {
    this.embed = embed; this.minSimilarity = minSimilarity; this.ttlMs = ttlMs;
    this.exact = new Map(); this.vectors = new Map(); this.inflight = new Map();
  }
  scopeKey({ tenantId, model, promptVersion }) { return `${tenantId}|${model}|${promptVersion}`; }
  normalise(q) { return q.trim().toLowerCase().replace(/\s+/g, ' '); }
  cacheable(q, ctx) {
    if (ctx.hasHistory || ctx.personalised) return false;
    return this.normalise(q).split(' ').length >= 3;
  }
  async get(q, ctx, callLLM) {
    if (!this.cacheable(q, ctx)) return { source: 'llm-bypass', text: await callLLM(q) };
    const scope = this.scopeKey(ctx), now = Date.now();
    const h = createHash('sha256').update(scope + '\n' + this.normalise(q)).digest('hex');
    const ex = this.exact.get(h);
    if (ex && ex.expires > now) return { source: 'exact', text: ex.text };

    const vec = await this.embed(q);
    let best = null, bestSim = -1;
    for (const row of this.vectors.get(scope) ?? []) {
      if (row.entry.expires <= now) continue;
      const s = cosine(vec, row.vec);
      if (s > bestSim) { bestSim = s; best = row; }
    }
    if (best && bestSim >= this.minSimilarity)
      return { source: 'semantic', similarity: +bestSim.toFixed(3), text: best.entry.text };

    if (this.inflight.has(h)) return { source: 'coalesced', text: await this.inflight.get(h) };
    const p = callLLM(q).finally(() => this.inflight.delete(h));
    this.inflight.set(h, p);
    const text = await p, entry = { text, expires: now + this.ttlMs };
    this.exact.set(h, entry);
    if (!this.vectors.has(scope)) this.vectors.set(scope, []);
    this.vectors.get(scope).push({ vec, entry });
    return { source: 'llm', similarity: bestSim < 0 ? null : +bestSim.toFixed(3), text };
  }
}

What happened when we ran it

We ran a test script on Node v22.23.4 using hand-made 4-dimensional vectors instead of a real embedding model, so the output is deterministic. The vectors are illustrative, not real model scores. The raw output:

1 acme: how do i reset my password           {"source":"llm","similarity":null}
2 acme: HOW do I reset my password           {"source":"exact","similarity":null}
3 acme: how can i reset my password?         {"source":"semantic","similarity":0.999}
4 acme: how do i reset my api key            {"source":"llm","similarity":0.851}
5 globex: how do i reset my password         {"source":"llm","similarity":null}
6 acme, multi-turn follow-up                 {"source":"llm-bypass","similarity":null}
7 five concurrent identical misses -> llm,coalesced,coalesced,coalesced,coalesced

Line 2 never touched the embedding model, line 5 proves tenant isolation, and line 7 shows five simultaneous misses producing one LLM call. In Python, the same flow maps onto RedisVL’s SemanticCache class, covered below.

What similarity threshold should a semantic cache use?

There is no universal semantic caching threshold; it depends on the embedding model, the domain and how costly a wrong answer is. Start strict and loosen only with labelled data.

The GPT Semantic Cache authors swept cosine similarity from 0.6 to 0.9 in 0.05 steps and chose 0.8 for their all-MiniLM-L6-v2 setup. The InfoQ banking study shows why copying that number is dangerous. At a 0.7 threshold, two of seven models produced a 99% false positive rate, and even after per-model tuning, false positives stayed between 13.4% and 27.2%.

Our sandbox run reproduces the failure mode in miniature. With the threshold at 0.8, the “reset my API key” question scored 0.851 against “reset my password” and was served the password answer:

minSimilarity 0.8: source=semantic similarity=0.851 served="answer(how do i reset my password)"
minSimilarity 0.9: source=llm similarity=0.851 served="answer(how do i reset my api key)"

The unit mismatch nobody warns you about

Papers and blog posts quote similarity (higher is stricter). Redis quotes distance. RedisVL’s SemanticCache sets distance_threshold in Redis COSINE units from 0 to 2, defaulting to 0.1, where lower is stricter. A similarity of 0.8 is a distance of 0.2. Paste “0.8” into a distance setting and you have accepted almost everything. Convert explicitly with 1 - similarity and log both numbers.

Why does semantic caching return wrong answers?

Semantic caching returns wrong answers because embeddings measure topical closeness, not intent, and short or messy queries sit close to many unrelated answers.

The InfoQ study lists three repeat failure modes: confusing related concepts such as credit and debit cards, missing intent (“can I skip a payment” vs “what happens if I miss one”), and averaging away qualifiers such as “after hours”. Its fix was architectural. Pre-loading 100 gold FAQs plus 300 distractors raised cache hit rates to a 68.4%–84.9% range while cutting false positives. Adding a quality filter for tiny, typo-heavy and vague queries took the best model to a 3.8% false positive rate.

The GPT Semantic Cache paper is candid too. Its limitations section states that the system does not handle context-dependent multi-turn interactions and that TTL alone is insufficient for rapidly changing data. That is why the code above bypasses any request carrying chat history.

One detail top-ranking tutorials skip: the paper’s abstract reports positive hit rates “exceeding 97%”, but its own results table puts per-category accuracy between 92.5% and 97.3%. On the Python category, 25 of 335 cache hits were judged wrong. Budget for that error rate, not the headline.

How do you scope and invalidate a semantic cache safely?

Scope every lookup by tenant, model and prompt version, and invalidate by bumping the version rather than hunting for stale vectors.

• Tenant: never search across customers. RedisVL exposes filterable_fields and a filter_expression on check() for exactly this kind of filtered retrieval.
• Model and prompt version: a new system prompt or model changes the right answer, so it must change the key.
• TTL per entry: RedisVL allows a per-entry TTL override on store(); use short TTLs for anything tied to prices, stock or policy.
• Data classification: do not cache responses built from a user’s private documents unless the scope includes that user.

Provider caches already isolate this way. Anthropic states that caches are isolated between organizations and, on the Claude API, between workspaces. Your own cache deserves the same discipline. Pair it with the idempotency key pattern if cached responses ever trigger side effects.

When is semantic caching for LLMs worth it?

Semantic caching for LLMs pays off when the same intents repeat across many users and a wrong answer is cheap to correct. It loses money on unique, personal or fast-changing prompts.

The published hit rates come from repetitive domains. GPT Semantic Cache reached 61.6% to 68.8% hit rates on programming, network support, shipping and shopping questions. Agentic coding sessions, document Q&A on private files and multi-turn chats rarely look like that. For those, the provider prompt cache does the heavy lifting. As a reference point, Anthropic lists Claude Sonnet 4.6 input at $3 per million tokens and cache hits at $0.30.

Exact-match cache vs provider prompt cache vs semantic cache (sources linked in each row)

Property	Exact-match cache	Provider prompt cache	Semantic cache
LLM still called on hit?	No	Yes, prefix reused (OpenAI)	No
Matches paraphrases?	No	No, prefix must match	Yes, by embedding similarity
Cost on hit	Lookup only	0.1x input for most Claude models (Anthropic)	Embedding call + vector search
Minimum size / lifetime	Your TTL	>1024 tokens (OpenAI); 5 min default (Anthropic)	Your TTL (RedisVL default: none)
Wrong-answer risk	None	None	Real: 3.8%–99% FP in the InfoQ study
Best for	Identical FAQ, retries	Long static system prompts, tools	High-volume paraphrased FAQs

• Semantic caching skips the LLM call on a hit; provider prompt caching still runs the model but discounts the repeated prefix.
• GPT Semantic Cache reported 61.6%–68.8% hit rates on repetitive support-style questions using a 0.8 cosine similarity threshold.
• An InfoQ banking study saw false positive rates as high as 99% at a 0.7 threshold before cache design changes brought them to 3.8%.
• RedisVL’s distance_threshold is cosine distance (default 0.1, lower is stricter), so similarity 0.8 equals distance 0.2.
• Every semantic cache lookup should be scoped by tenant, model and prompt version, and bypassed for multi-turn or personalised requests.
• Single-flight on cache misses stops a burst of identical new questions from triggering parallel LLM calls.

Frequently Asked Questions

What is semantic caching in LLM?

Semantic caching in LLM applications stores prompts as embedding vectors alongside their responses. When a new prompt is close enough in meaning to a stored one, the cached response is returned and the model is not called. It reduces cost and latency on repetitive traffic but can return wrong answers if the similarity threshold is too loose.

What is the difference between prompt caching and semantic caching?

Prompt caching is done by the LLM provider: the model still runs, but a repeated prompt prefix is billed at a discount and processed faster. Semantic caching is done by your application: it matches prompts by meaning and returns a stored answer without calling the model. Most production systems use both, with prompt caching first.

What similarity threshold should I use for semantic caching?

There is no universal value. One published Redis-based system chose 0.8 cosine similarity after testing 0.6 to 0.9, while a banking study still saw double-digit false positives after tuning. Start strict, label real queries, and convert carefully between similarity and distance, since RedisVL uses cosine distance where lower is stricter.

When should you not use semantic caching?

Avoid semantic caching for multi-turn conversations, personalised prompts, answers built from a user’s private data, and fast-changing facts such as prices or stock levels. It also adds little on workloads where most prompts are unique, such as coding agents, because hit rates stay low while every request still pays for an embedding and a vector search.

Is Redis good for semantic caching?

Redis is a common choice because it keeps vectors and responses in memory and supports vector search. RedisVL provides a SemanticCache class with a configurable cosine distance threshold, per-entry TTLs and filterable fields for scoping by tenant. The GPT Semantic Cache paper also used Redis with an HNSW index for its experiments.

Semantic caching for LLMs is worth building when your traffic repeats by intent, and dangerous when it does not. The pattern that holds up is layered: bypass rules, scoped keys, an exact-match layer, a strict semantic layer and single-flight misses, all sitting above the provider’s own prompt cache. Copy the Node.js class above, swap the arrays for Redis, and log similarity on every hit so you tune with real data.

If you are wiring this into an existing AI gateway, start by measuring how many of last week’s prompts were near-duplicates. That one query tells you whether the semantic layer will earn its keep.

Sources

• arXiv — GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching (Regmi & Pun, v3)
• InfoQ — Reducing False Positives in RAG Semantic Caching: a Banking Case Study
• Redis — RedisVL 0.28.0 LLM Cache API reference (SemanticCache)
• OpenAI Cookbook — Prompt Caching 101
• Anthropic — Claude Docs: Prompt caching