Implementing redis cache in LLM

Table of contents

Intro

Caching not only improves application performance but in LLM aspect it saves huge amount of cost and also benefits the environment as we all know running a LLM in a data center consumes lot of resources, eventually it will be sustainable but we need to wait and see.

So as architect/dev we have the responsiblity to create energy efficient applications, we will see how to implement cache in LLM to make more faster responses and reducing costs.

Types of Cache available

Traditionally, we find the cache data by key, the application requests with the key which holds the cached data. But in LLM, we have the input differs every time, say the user might ask Weather today in Chennai also next time he might ask Todays weather in Chennai even though both has the same meaning but the exact string wouldn’t match.

We will be using Redis cache for our usecase, redis offers both exact string match and Semantic match to handle the question which holds the same Semantic value. It’s up to us based on the usecase we need to decide.

• Exact match
• Semantic match

Step 1 : Packages required

We have used both openai and gemini for the example and you would see those packages in the below command

uv add "google-genai>=2.22.0" "langchain>=1.4.0" "langchain-core>=0.3.0" "langchain-openai>=1.6.0" "langchain-redis>=0.2.5" "python-dotenv>=1.2.3" "redis>=5.0.0"

Step 2: Exact match cache

We are initiating the REDIS and created a method to mesaure the performance of the same query asked by the user twice.

#redis url
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")

# Initialize RedisCache
redis_cache = RedisCache(redis_url=REDIS_URL)

# Set the cache for LangChain to use
set_llm_cache(redis_cache)

#model

llm = init_chat_model(
    model="gemini-2.5-flash",
    model_provider="google_genai")

# Function to measure execution time
def timed_completion(prompt):
    start_time = time.time()
    result = llm.invoke(prompt)
    end_time = time.time()
    return result, end_time - start_time

# First call (not cached)
prompt = "Explain the concept of caching in three sentences."
result1, time1 = timed_completion(prompt)
print(f"First call (not cached):\nResult: {result1}\nTime: {time1:.2f} seconds\n")

# Second call (should be cached)
result2, time2 = timed_completion(prompt)
print(f"Second call (cached):\nResult: {result2}\nTime: {time2:.2f} seconds\n")

print(f"Speed improvement: {time1 / time2:.2f}x faster")

Below is the performance

Step 3 : Semantic cache

The exact match will work fine for the repeative same query, for the different queries with same Semantic meaning will have cache miss. Below is the code where we implemented the Semantic search.

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")

#embeddings required
embeddings = OpenAIEmbeddings()

#Initialize semantic_cache from redis
semantic_cache = RedisSemanticCache(
    redis_url=REDIS_URL, embeddings=embeddings, distance_threshold=0.2

#model
llm = init_chat_model(
    model="gpt-5.4-mini",
    model_provider="openai")

# Function to test semantic cache
def test_semantic_cache(prompt):
    start_time = time.time()
    result = llm.invoke(prompt)
    end_time = time.time()
    return result, end_time - start_time

# Original query
original_prompt = "Capital of United States?"
result1, time1 = test_semantic_cache(original_prompt)
print(
    f"Original query:\nPrompt: {original_prompt}\nResult: {result1}\nTime: {time1:.2f} seconds\n"
)

# Semantically similar query
similar_prompt = "Can you tell me the capital of United States?"
result2, time2 = test_semantic_cache(similar_prompt)
print(
    f"Similar query:\nPrompt: {similar_prompt}\nResult: {result2}\nTime: {time2:.2f} seconds\n"
)

Below is the performance results

Conclusion

We saw the performance improvement in the subsequent calls and it’s upto the business use cases to set up the ttl configuration for the cache. There are several other cache providers available. If you want to see the source code of the above example refer here.