Caching (Exact and Semantic)

TrueFoundry AI Gateway caches LLM responses so that repeated or similar queries are served instantly without calling the model provider. The gateway supports two caching strategies:

Exact-Match — returns cached responses only when the request is identical
Semantic — returns cached responses when requests are semantically similar, even if worded differently

Prompt Caching vs Gateway Caching

LLM providers and the TrueFoundry AI Gateway offer caching at different layers. Understanding the difference helps you pick the right approach — or combine both for maximum savings.

	Provider Prompt Caching	Gateway Caching (Exact-Match & Semantic)
Where it runs	Inside the model provider (e.g. Anthropic)	In the TrueFoundry AI Gateway, before the request reaches any provider
What is cached	The prompt prefix — the provider skips re-processing tokens it has already seen	The complete LLM response — a cache hit returns the response instantly with zero model invocation
How it matches	Exact token-prefix match on the prompt	Exact request hash (exact-match) or embedding cosine similarity (semantic)
Latency savings	Reduces time-to-first-token; the model still generates a new completion	Eliminates model call entirely — response is returned from cache in milliseconds
Cost savings	Cached input tokens are billed at a reduced rate (varies by provider)	Cache hit = zero model cost for that request
Provider support	Provider-specific (see Prompt Caching)	Works with every provider routed through the gateway

Cache Types

Exact-Match Cache

Exact-match caching stores responses keyed by a hash of the complete request — messages, model, and all parameters. A cached response is returned only when every part of the request matches exactly. Best for:

API calls with identical parameters
Deterministic queries that always need the same response
Development and testing environments
Applications with predictable, repetitive queries

Semantic Cache

Semantic caching uses embeddings and cosine similarity to match requests that express the same intent, even when worded differently. For example, “How do I reset my password?” and “What’s the password reset process?” would match. The gateway extracts the last message, generates an embedding, and compares it against cached embeddings. All other request parameters (model, prior messages, temperature, etc.) are hashed and must still match exactly — only the last message is compared semantically. Best for:

Customer support chatbots
FAQ systems where users phrase questions differently
Conversational AI applications
Any scenario where query variations express the same intent

Semantic cache is a superset of exact-match cache. Setting the cache type to semantic will also return results for exact-match hits, so you don’t need to configure both.

How to control the similarity required for a cache hit?

The similarity_threshold parameter (0 – 1.0) controls how close two queries must be for the cached response to be returned. A higher value demands closer matches; a lower value allows broader matching.

Range	Behaviour	Recommended for
0.95 – 1.0	Very strict — only nearly identical queries match	High-precision use cases where incorrect cache hits are costly
0.85 – 0.95	Balanced — works well for most conversational applications	General-purpose chatbots and FAQ systems
< 0.85	Broad — may return results for loosely related queries	Exploratory or low-risk scenarios

Start with a threshold of 0.9 and monitor cache hit rates. Adjust up for precision or down for coverage.

How is similarity computed?

The gateway converts the last message of each request into a vector embedding using an embedding model. When a new request arrives, its embedding is compared against cached embeddings using cosine similarity — a score between 0 and 1.0 that measures how close two vectors are in meaning. If the score meets or exceeds the configured similarity_threshold, the cached response is returned.All other request parameters (model, prior messages, temperature, etc.) are hashed separately and must match exactly — only the last message is compared semantically.

Namespacing and Cache Isolation

The gateway isolates cache entries at two levels to prevent data leaking across boundaries.

Level 1 — User / Virtual Account (automatic)

Every cache entry is scoped to the user or virtual account that created it. This happens automatically — you don’t need to configure anything. A cache entry created by User A is never visible to User B, even if they send the exact same request.

Level 2 — Custom Namespace (optional)

Within a user’s or virtual account’s cache, you can further partition entries by providing a namespace string. Entries in one namespace are invisible to requests with a different namespace (or no namespace). This is useful when a single virtual account serves multiple downstream end-users or application contexts and you want per-context cache isolation. For example, an application that serves many tenants through a single virtual account can set namespace to each tenant’s ID so that cached responses never cross tenant boundaries.

Scenario	Namespace value	Effect
Single application, shared cache	omit or set to `"default"`	All requests under the same user/virtual account share a single cache pool
Multi-tenant application	Set to the tenant or end-user ID (e.g. `"tenant-123"`)	Each tenant gets its own isolated cache within the same virtual account
Multiple environments	Set to the environment name (e.g. `"staging"`, `"production"`)	Prevents staging queries from returning production-cached responses

{
  "type": "semantic",
  "similarity_threshold": 0.9,
  "ttl": 600,
  "namespace": "tenant-123"
}

Configuration

Enable caching by adding the x-tfy-cache-config header to your requests.

Examples

Replace {GATEWAY_BASE_URL} with the base URL of the TrueFoundry AI Gateway and your-truefoundry-api-key with your API key.

curl {GATEWAY_BASE_URL}/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-truefoundry-api-key" \
  -H "x-tfy-cache-config: {\"type\": \"exact-match\", \"ttl\": 600}" \
  -d '{
    "model": "openai-main/gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ]
  }'

curl {GATEWAY_BASE_URL}/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-truefoundry-api-key" \
  -H "x-tfy-cache-config: {\"type\": \"semantic\", \"similarity_threshold\": 0.9, \"ttl\": 600}" \
  -d '{
    "model": "openai-main/gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "How do I reset my password?"
      }
    ]
  }'

curl {GATEWAY_BASE_URL}/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-truefoundry-api-key" \
  -H "x-tfy-cache-config: {\"type\": \"semantic\", \"similarity_threshold\": 0.9, \"ttl\": 600, \"namespace\": \"tenant-123\"}" \
  -d '{
    "model": "openai-main/gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "How do I reset my password?"
      }
    ]
  }'

Response Headers

The gateway returns headers indicating cache status on every response:

Header	Description	Example Values
`x-tfy-cache-status`	Whether the cache was hit	`hit`, `miss`, `error`
`x-tfy-cached-trace-id`	Trace ID of the original request that populated the cache (only on hits)	`trace_abc123`
`x-tfy-cache-similarity-score`	Cosine similarity score (semantic cache hits only)	`0.95`

How It Works

Exact-Match
Semantic

Hash the request

The gateway generates a hash of the complete request (messages, model, parameters).

Look up the cache

If a cached response exists for this hash, it is returned immediately.

Forward on miss

On a cache miss, the request is forwarded to the model provider. The response is cached before being returned.

Infrastructure Setup

For SaaS customers, caching infrastructure is fully managed — no additional setup is required. For On-Premise deployments, caching requires two infrastructure components:

Redis — cache store for both exact-match and semantic caching
Embedding Model — required for semantic caching to generate vector embeddings (see Embedding Model)

Redis Setup

You can either enable the bundled Redis instance in the tfy-llm-gateway Helm chart, or connect your own Redis (or Redis-compatible store such as Valkey).

Bundled Redis
Bring Your Own Redis

Enable Redis directly in the tfy-llm-gateway chart values:

tfy-llm-gateway:
  redis:
    enabled: true

Disable the bundled Redis and configure connection details via environment variables:

tfy-llm-gateway:
  redis:
    enabled: false
  env:
    REDIS_HOST: "<your-redis-host>"
    REDIS_PORT: 6379
    REDIS_USERNAME: "<your-redis-username>"
    REDIS_PASSWORD: "<your-redis-password>"

Variable	Required	Type	Description
`REDIS_HOST`	Yes	`string`	Hostname or IP of your Redis instance
`REDIS_PORT`	Yes	`number`	Port number (default: `6379`)
`REDIS_USERNAME`	No	`string`	Username for Redis authentication
`REDIS_PASSWORD`	No	`string`	Password for Redis authentication
`REDIS_DB`	No	`number`	Redis database index to use (default: `0`). Applies to both standalone and Sentinel modes

How do I connect my external Redis?

Enable an encrypted connection by setting REDIS_TLS_ENABLED: true. TLS applies to both standalone and Sentinel connections. The certificate material (REDIS_TLS_CA_CERT, REDIS_TLS_CERT, REDIS_TLS_KEY) accepts either a path to a mounted PEM file or the inline PEM contents.

tfy-llm-gateway:
  redis:
    enabled: false
  env:
    REDIS_HOST: "<your-redis-host>"
    REDIS_PORT: 6379
    REDIS_TLS_ENABLED: true
    # Trust a private CA (path to a mounted PEM file, or inline PEM)
    REDIS_TLS_CA_CERT: "/etc/redis-certs/ca.crt"
    # Optional: SNI / expected certificate hostname if it differs from REDIS_HOST
    REDIS_TLS_SERVERNAME: "redis.internal"
    # Mutual TLS (client certificate) — CERT and KEY must be set together
    REDIS_TLS_CERT: "/etc/redis-certs/client.crt"
    REDIS_TLS_KEY: "/etc/redis-certs/client.key"
    REDIS_TLS_KEY_PASSPHRASE: "<key-passphrase>"

Variable	Required	Type	Description
`REDIS_TLS_ENABLED`	Yes	`boolean`	Set to `true` to connect over TLS (default: `false`)
`REDIS_TLS_CA_CERT`	No	`string`	CA certificate to trust, as a path to a mounted PEM file or inline PEM. Use when Redis presents a certificate signed by a private CA
`REDIS_TLS_REJECT_UNAUTHORIZED`	No	`boolean`	Whether to reject a server whose certificate can’t be verified (default: `true`). Only set to `false` for testing
`REDIS_TLS_SERVERNAME`	No	`string`	Server name (SNI) to send and hostname to verify against, if it differs from `REDIS_HOST`
`REDIS_TLS_CERT`	No	`string`	Client certificate for mutual TLS, as a path or inline PEM. Must be set together with `REDIS_TLS_KEY`
`REDIS_TLS_KEY`	No	`string`	Client private key for mutual TLS, as a path or inline PEM. Must be set together with `REDIS_TLS_CERT`
`REDIS_TLS_KEY_PASSPHRASE`	No	`string`	Passphrase for the client private key, if it is encrypted

PEM material (REDIS_TLS_CA_CERT, REDIS_TLS_CERT, REDIS_TLS_KEY) can be a path to a Kubernetes-mounted file or the inline PEM string. A value that is neither a readable file nor valid PEM fails loudly at startup.
Mutual TLS requires both REDIS_TLS_CERT and REDIS_TLS_KEY — setting only one is a configuration error.
These TLS settings apply to both standalone and Sentinel connections.

How do I connect to a highly available Redis using Sentinel?

For high-availability Redis deployments that use Redis Sentinel for automatic failover, point the gateway at your Sentinel nodes instead of a single Redis host. In Sentinel mode the gateway discovers the current master through the Sentinels and automatically follows failovers.Enable Sentinel by setting REDIS_SENTINEL_ENABLED: true along with the Sentinel node list and master name:

tfy-llm-gateway:
  redis:
    enabled: false
  env:
    REDIS_SENTINEL_ENABLED: true
    REDIS_SENTINEL_NODES: "sentinel-0:26379,sentinel-1:26379,sentinel-2:26379"
    REDIS_SENTINEL_MASTER_NAME: "mymaster"
    # Optional: credentials for the Sentinel processes themselves
    REDIS_SENTINEL_USERNAME: "<your-sentinel-username>"
    REDIS_SENTINEL_PASSWORD: "<your-sentinel-password>"
    # Data-node (master/replica) auth reuses the standard Redis credentials
    REDIS_USERNAME: "<your-redis-username>"
    REDIS_PASSWORD: "<your-redis-password>"

Variable	Required	Type	Description
`REDIS_SENTINEL_ENABLED`	Yes	`boolean`	Set to `true` to use Sentinel mode (default: `false`)
`REDIS_SENTINEL_NODES`	Yes	`string`	Comma-separated list of Sentinel nodes as `host:port` (e.g. `sentinel-0:26379,sentinel-1:26379`). If a port is omitted, `26379` is assumed
`REDIS_SENTINEL_MASTER_NAME`	Yes	`string`	Name of the master group monitored by Sentinel (e.g. `mymaster`)
`REDIS_SENTINEL_USERNAME`	No	`string`	Username for authenticating to the Sentinel nodes
`REDIS_SENTINEL_PASSWORD`	No	`string`	Password for authenticating to the Sentinel nodes

Sentinel mode activates only when REDIS_SENTINEL_ENABLED is true and both REDIS_SENTINEL_NODES and REDIS_SENTINEL_MASTER_NAME are set. If any of these is missing, the gateway falls back to standalone mode using REDIS_HOST.
Data-node authentication (to the Redis master and replicas) uses REDIS_USERNAME / REDIS_PASSWORD. The REDIS_SENTINEL_USERNAME / REDIS_SENTINEL_PASSWORD variables authenticate to the Sentinel processes only.
TLS applies to both the Sentinel and data-node connections — enable it with the same REDIS_TLS_* variables used for standalone Redis.

Embedding Model

For embedding model, here is the configuration required based on your deployment mode:

TrueFoundry SaaS: No setup required, we use OpenAI’s text-embedding-3-small model by default. This model is not configurable.
Self Hosted (Control Plane + Gateway Plane): You can configure the model by going to “Settings” → “Semantic Cache” and select the embedding model. (should already be added as in integration)
Hybrid: (TrueFoundry Control Plane + Self Hosted Gateway Plane): For this, you need add an environment variable in the values of tfy-llm-gateway helm chart.
You can add your embedding model (already registered on TrueFoundry) using the following change:
tfy-llm-gateway: env: SEMANTIC_CACHE_MODEL_IDENTIFIER: "openai-account/embedding-model-name" #model identifier on TrueFoundry for the embedding model

Get Started

LLM Gateway

MCP Registry and Gateway

Skills Registry

Prompt Registry

Guardrails and Security

Observability

Deployment

Admin Guide

Chat

Messages

Embeddings

Rerank

Responses

Image

Audio

Batch

Files

Fine-tuning

Moderations

Models

Prompt Caching vs Gateway Caching

Cache Types

Exact-Match Cache

Semantic Cache

Namespacing and Cache Isolation

Level 1 — User / Virtual Account (automatic)

Level 2 — Custom Namespace (optional)

Configuration

Examples

Response Headers

How It Works

Infrastructure Setup

Redis Setup

Embedding Model

​Prompt Caching vs Gateway Caching

​Cache Types

​Exact-Match Cache

​Semantic Cache

​Namespacing and Cache Isolation

​Level 1 — User / Virtual Account (automatic)

​Level 2 — Custom Namespace (optional)

​Configuration

​Examples

​Response Headers

​How It Works

​Infrastructure Setup

​Redis Setup

​Embedding Model

Prompt Caching vs Gateway Caching

Cache Types

Exact-Match Cache

Semantic Cache

Namespacing and Cache Isolation

Level 1 — User / Virtual Account (automatic)

Level 2 — Custom Namespace (optional)

Configuration

Examples

Response Headers

How It Works

Infrastructure Setup

Redis Setup

Embedding Model