Our LLM API invoice was rising 30% month-over-month. Visitors was rising, however not that quick. Once I analyzed our question logs, I discovered the true downside: Customers ask the identical questions in numerous methods.
"What's your return policy?," "How do I return something?", and "Can I get a refund?" had been all hitting our LLM individually, producing almost an identical responses, every incurring full API prices.
Precise-match caching, the plain first answer, captured solely 18% of those redundant calls. The identical semantic query, phrased in a different way, bypassed the cache completely.
So, I applied semantic caching based mostly on what queries imply, not how they're worded. After implementing it, our cache hit charge elevated to 67%, lowering LLM API prices by 73%. However getting there requires fixing issues that naive implementations miss.
Why exact-match caching falls quick
Conventional caching makes use of question textual content because the cache key. This works when queries are an identical:
# Precise-match caching
cache_key = hash(query_text)
if cache_key in cache:
return cache[cache_key]
However customers don't phrase questions identically. My evaluation of 100,000 manufacturing queries discovered:
Solely 18% had been actual duplicates of earlier queries
47% had been semantically much like earlier queries (identical intent, totally different wording)
35% had been genuinely novel queries
That 47% represented large value financial savings we had been lacking. Every semantically-similar question triggered a full LLM name, producing a response almost an identical to 1 we'd already computed.
Semantic caching structure
Semantic caching replaces text-based keys with embedding-based similarity lookup:
class SemanticCache:
def __init__(self, embedding_model, similarity_threshold=0.92):
self.embedding_model = embedding_model
self.threshold = similarity_threshold
self.vector_store = VectorStore() # FAISS, Pinecone, and so forth.
self.response_store = ResponseStore() # Redis, DynamoDB, and so forth.
def get(self, question: str) -> Non-obligatory[str]:
"""Return cached response if semantically similar query exists."""
query_embedding = self.embedding_model.encode(question)
# Discover most related cached question
matches = self.vector_store.search(query_embedding, top_k=1)
if matches and matches[0].similarity >= self.threshold:
cache_id = matches[0].id
return self.response_store.get(cache_id)
return None
def set(self, question: str, response: str):
"""Cache query-response pair."""
query_embedding = self.embedding_model.encode(question)
cache_id = generate_id()
self.vector_store.add(cache_id, query_embedding)
self.response_store.set(cache_id, {
'question': question,
'response': response,
'timestamp': datetime.utcnow()
})
The important thing perception: As a substitute of hashing question textual content, I embed queries into vector area and discover cached queries inside a similarity threshold.
The edge downside
The similarity threshold is the important parameter. Set it too excessive, and also you miss legitimate cache hits. Set it too low, and you come back flawed responses.
Our preliminary threshold of 0.85 appeared affordable; 85% related needs to be "the same question," proper?
Mistaken. At 0.85, we obtained cache hits like:
Question: "How do I cancel my subscription?"
Cached: "How do I cancel my order?"
Similarity: 0.87
These are totally different questions with totally different solutions. Returning the cached response could be incorrect.
I found that optimum thresholds differ by question sort:
Question sort
Optimum threshold
Rationale
FAQ-style questions
0.94
Excessive precision wanted; flawed solutions injury belief
Product searches
0.88
Extra tolerance for near-matches
Help queries
0.92
Stability between protection and accuracy
Transactional queries
0.97
Very low tolerance for errors
I applied query-type-specific thresholds:
class AdaptiveSemanticCache:
def __init__(self):
self.thresholds = {
'faq': 0.94,
'search': 0.88,
'assist': 0.92,
'transactional': 0.97,
'default': 0.92
}
self.query_classifier = QueryClassifier()
def get_threshold(self, question: str) -> float:
query_type = self.query_classifier.classify(question)
return self.thresholds.get(query_type, self.thresholds['default'])
def get(self, question: str) -> Non-obligatory[str]:
threshold = self.get_threshold(question)
query_embedding = self.embedding_model.encode(question)
matches = self.vector_store.search(query_embedding, top_k=1)
if matches and matches[0].similarity >= threshold:
return self.response_store.get(matches[0].id)
return None
Threshold tuning methodology
I couldn't tune thresholds blindly. I wanted floor fact on which question pairs had been truly "the same."
Our methodology:
Step 1: Pattern question pairs. I sampled 5,000 question pairs at numerous similarity ranges (0.80-0.99).
Step 2: Human labeling. Annotators labeled every pair as "same intent" or "different intent." I used three annotators per pair and took a majority vote.
Step 3: Compute precision/recall curves. For every threshold, we computed:
Precision: Of cache hits, what fraction had the identical intent?
Recall: Of same-intent pairs, what fraction did we cache-hit?
def compute_precision_recall(pairs, labels, threshold):
"""Compute precision and recall at given similarity threshold."""
predictions = [1 if pair.similarity >= threshold else 0 for pair in pairs]
true_positives = sum(1 for p, l in zip(predictions, labels) if p == 1 and l == 1)
false_positives = sum(1 for p, l in zip(predictions, labels) if p == 1 and l == 0)
false_negatives = sum(1 for p, l in zip(predictions, labels) if p == 0 and l == 1)
precision = true_positives / (true_positives + false_positives) if (true_positives + false_positives) > 0 else 0
recall = true_positives / (true_positives + false_negatives) if (true_positives + false_negatives) > 0 else 0
return precision, recall
Step 4: Choose threshold based mostly on value of errors. For FAQ queries the place flawed solutions injury belief, I optimized for precision (0.94 threshold gave 98% precision). For search queries the place lacking a cache hit simply prices cash, I optimized for recall (0.88 threshold).
Latency overhead
Semantic caching provides latency: You have to embed the question and search the vector retailer earlier than figuring out whether or not to name the LLM.
Our measurements:
Operation
Latency (p50)
Latency (p99)
Question embedding
12ms
28ms
Vector search
8ms
19ms
Complete cache lookup
20ms
47ms
LLM API name
850ms
2400ms
The 20ms overhead is negligible in comparison with the 850ms LLM name we keep away from on cache hits. Even at p99, the 47ms overhead is appropriate.
Nevertheless, cache misses now take 20ms longer than earlier than (embedding + search + LLM name). At our 67% hit charge, the maths works out favorably:
Earlier than: 100% of queries × 850ms = 850ms common
After: (33% × 870ms) + (67% × 20ms) = 287ms + 13ms = 300ms common
Internet latency enchancment of 65% alongside the price discount.
Cache invalidation
Cached responses go stale. Product info modifications, insurance policies replace and yesterday's right reply turns into as we speak's flawed reply.
I applied three invalidation methods:
Time-based TTL
Easy expiration based mostly on content material sort:
TTL_BY_CONTENT_TYPE = {
'pricing': timedelta(hours=4), # Adjustments steadily
'coverage': timedelta(days=7), # Adjustments hardly ever
'product_info': timedelta(days=1), # Day by day refresh
'general_faq': timedelta(days=14), # Very steady
}
Occasion-based invalidation
When underlying information modifications, invalidate associated cache entries:
class CacheInvalidator:
def on_content_update(self, content_id: str, content_type: str):
"""Invalidate cache entries related to updated content."""
# Discover cached queries that referenced this content material
affected_queries = self.find_queries_referencing(content_id)
for query_id in affected_queries:
self.cache.invalidate(query_id)
self.log_invalidation(content_id, len(affected_queries))
Staleness detection
For responses which may turn into stale with out express occasions, I applied periodic freshness checks:
def check_freshness(self, cached_response: dict) -> bool:
"""Verify cached response is still valid."""
# Re-run the question towards present information
fresh_response = self.generate_response(cached_response['query'])
# Evaluate semantic similarity of responses
cached_embedding = self.embed(cached_response['response'])
fresh_embedding = self.embed(fresh_response)
similarity = cosine_similarity(cached_embedding, fresh_embedding)
# If responses diverged considerably, invalidate
if similarity < 0.90:
self.cache.invalidate(cached_response['id'])
return False
return True
We run freshness checks on a pattern of cached entries day by day, catching staleness that TTL and event-based invalidation miss.
Manufacturing outcomes
After three months in manufacturing:
Metric
Earlier than
After
Change
Cache hit charge
18%
67%
+272%
LLM API prices
$47K/month
$12.7K/month
-73%
Common latency
850ms
300ms
-65%
False-positive charge
N/A
0.8%
—
Buyer complaints (flawed solutions)
Baseline
+0.3%
Minimal enhance
The 0.8% false-positive charge (queries the place we returned a cached response that was semantically incorrect) was inside acceptable bounds. These circumstances occurred primarily on the boundaries of our threshold, the place similarity was simply above the cutoff however intent differed barely.
Pitfalls to keep away from
Don't use a single international threshold. Completely different question sorts have totally different tolerance for errors. Tune thresholds per class.
Don't skip the embedding step on cache hits. You is perhaps tempted to skip embedding overhead when returning cached responses, however you want the embedding for cache key era. The overhead is unavoidable.
Don't overlook invalidation. Semantic caching with out invalidation technique results in stale responses that erode person belief. Construct invalidation from day one.
Don't cache every little thing. Some queries shouldn't be cached: Customized responses, time-sensitive info, transactional confirmations. Construct exclusion guidelines.
def should_cache(self, question: str, response: str) -> bool:
"""Determine if response should be cached.""
# Don't cache personalised responses
if self.contains_personal_info(response):
return False
# Don't cache time-sensitive info
if self.is_time_sensitive(question):
return False
# Don't cache transactional confirmations
if self.is_transactional(question):
return False
return True
Key takeaways
Semantic caching is a sensible sample for LLM value management that captures redundancy exact-match caching misses. The important thing challenges are threshold tuning (use query-type-specific thresholds based mostly on precision/recall evaluation) and cache invalidation (mix TTL, event-based and staleness detection).
At 73% value discount, this was our highest-ROI optimization for manufacturing LLM methods. The implementation complexity is reasonable, however the threshold tuning requires cautious consideration to keep away from high quality degradation.
Sreenivasa Reddy Hulebeedu Reddy is a lead software program engineer.
