> ## Documentation Index > Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt > Use this file to discover all available pages before exploring further. # Caching (Exact and Semantic) > Speed up responses and cut costs with gateway-level exact-match and semantic caching. TrueFoundry AI Gateway caches LLM responses so that repeated or similar queries are served instantly without calling the model provider. The gateway supports two caching strategies: * **Exact-Match** — returns cached responses only when the request is identical * **Semantic** — returns cached responses when requests are semantically similar, even if worded differently ## Prompt Caching vs Gateway Caching LLM providers and the TrueFoundry AI Gateway offer caching at different layers. Understanding the difference helps you pick the right approach — or combine both for maximum savings. | | Provider Prompt Caching | Gateway Caching (Exact-Match & Semantic) | | -------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | **Where it runs** | Inside the model provider (e.g. Anthropic) | In the TrueFoundry AI Gateway, before the request reaches any provider | | **What is cached** | The prompt prefix — the provider skips re-processing tokens it has already seen | The complete LLM response — a cache hit returns the response instantly with zero model invocation | | **How it matches** | Exact token-prefix match on the prompt | Exact request hash (exact-match) or embedding cosine similarity (semantic) | | **Latency savings** | Reduces time-to-first-token; the model still generates a new completion | Eliminates model call entirely — response is returned from cache in milliseconds | | **Cost savings** | Cached input tokens are billed at a reduced rate (varies by provider) | Cache hit = zero model cost for that request | | **Provider support** | Provider-specific (see [Prompt Caching](/ai-gateway/chat-completions-prompt-caching)) | Works with every provider routed through the gateway | ## Cache Types ### Exact-Match Cache Exact-match caching stores responses keyed by a hash of the complete request — messages, model, and all parameters. A cached response is returned only when every part of the request matches exactly. **Best for:** * API calls with identical parameters * Deterministic queries that always need the same response * Development and testing environments * Applications with predictable, repetitive queries ### Semantic Cache Semantic caching uses embeddings and cosine similarity to match requests that express the same intent, even when worded differently. For example, *"How do I reset my password?"* and *"What's the password reset process?"* would match. The gateway extracts the last message, generates an embedding, and compares it against cached embeddings. All other request parameters (model, prior messages, temperature, etc.) are hashed and must still match exactly — only the last message is compared semantically. **Best for:** * Customer support chatbots * FAQ systems where users phrase questions differently * Conversational AI applications * Any scenario where query variations express the same intent Semantic cache is a superset of exact-match cache. Setting the cache type to `semantic` will also return results for exact-match hits, so you don't need to configure both. The `similarity_threshold` parameter (0 – 1.0) controls how close two queries must be for the cached response to be returned. A higher value demands closer matches; a lower value allows broader matching. | Range | Behaviour | Recommended for | | --------------- | ---------------------------------------------------------- | -------------------------------------------------------------- | | **0.95 – 1.0** | Very strict — only nearly identical queries match | High-precision use cases where incorrect cache hits are costly | | **0.85 – 0.95** | Balanced — works well for most conversational applications | General-purpose chatbots and FAQ systems | | **\< 0.85** | Broad — may return results for loosely related queries | Exploratory or low-risk scenarios | Start with a threshold of **0.9** and monitor cache hit rates. Adjust up for precision or down for coverage. The gateway converts the last message of each request into a vector embedding using an embedding model. When a new request arrives, its embedding is compared against cached embeddings using **cosine similarity** — a score between 0 and 1.0 that measures how close two vectors are in meaning. If the score meets or exceeds the configured `similarity_threshold`, the cached response is returned. All other request parameters (model, prior messages, temperature, etc.) are hashed separately and must match exactly — only the last message is compared semantically. *** ## Namespacing and Cache Isolation The gateway isolates cache entries at two levels to prevent data leaking across boundaries. ### Level 1 — User / Virtual Account (automatic) Every cache entry is scoped to the **user or virtual account** that created it. This happens automatically — you don't need to configure anything. A cache entry created by User A is never visible to User B, even if they send the exact same request. ### Level 2 — Custom Namespace (optional) Within a user's or virtual account's cache, you can further partition entries by providing a `namespace` string. Entries in one namespace are invisible to requests with a different namespace (or no namespace). This is useful when a single virtual account serves multiple downstream end-users or application contexts and you want per-context cache isolation. For example, an application that serves many tenants through a single virtual account can set `namespace` to each tenant's ID so that cached responses never cross tenant boundaries. | Scenario | Namespace value | Effect | | -------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------- | | Single application, shared cache | *omit or set to `"default"`* | All requests under the same user/virtual account share a single cache pool | | Multi-tenant application | Set to the tenant or end-user ID (e.g. `"tenant-123"`) | Each tenant gets its own isolated cache within the same virtual account | | Multiple environments | Set to the environment name (e.g. `"staging"`, `"production"`) | Prevents staging queries from returning production-cached responses | ```json theme={"dark"} { "type": "semantic", "similarity_threshold": 0.9, "ttl": 600, "namespace": "tenant-123" } ``` *** ## Configuration Enable caching by adding the `x-tfy-cache-config` header to your requests. ### Examples Replace `{GATEWAY_BASE_URL}` with the base URL of the TrueFoundry AI Gateway and `your-truefoundry-api-key` with your API key. ```bash Exact-Match Cache theme={"dark"} curl {GATEWAY_BASE_URL}/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer your-truefoundry-api-key" \ -H "x-tfy-cache-config: {\"type\": \"exact-match\", \"ttl\": 600}" \ -d '{ "model": "openai-main/gpt-4o", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' ``` ```bash Semantic Cache theme={"dark"} curl {GATEWAY_BASE_URL}/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer your-truefoundry-api-key" \ -H "x-tfy-cache-config: {\"type\": \"semantic\", \"similarity_threshold\": 0.9, \"ttl\": 600}" \ -d '{ "model": "openai-main/gpt-4o", "messages": [ { "role": "user", "content": "How do I reset my password?" } ] }' ``` ```bash Semantic Cache with Namespace theme={"dark"} curl {GATEWAY_BASE_URL}/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer your-truefoundry-api-key" \ -H "x-tfy-cache-config: {\"type\": \"semantic\", \"similarity_threshold\": 0.9, \"ttl\": 600, \"namespace\": \"tenant-123\"}" \ -d '{ "model": "openai-main/gpt-4o", "messages": [ { "role": "user", "content": "How do I reset my password?" } ] }' ``` *** ## Response Headers The gateway returns headers indicating cache status on every response: | Header | Description | Example Values | | ------------------------------ | ------------------------------------------------------------------------ | ---------------------- | | `x-tfy-cache-status` | Whether the cache was hit | `hit`, `miss`, `error` | | `x-tfy-cached-trace-id` | Trace ID of the original request that populated the cache (only on hits) | `trace_abc123` | | `x-tfy-cache-similarity-score` | Cosine similarity score (semantic cache hits only) | `0.95` | *** ## How It Works The gateway generates a hash of the complete request (messages, model, parameters). If a cached response exists for this hash, it is returned immediately. On a cache miss, the request is forwarded to the model provider. The response is cached before being returned. The gateway extracts the last message and generates an embedding vector. All other request parameters are hashed. The embedding is compared against cached embeddings using cosine similarity. The parameter hash must also match exactly. If the similarity score exceeds the configured threshold and the parameter hash matches, the cached response is returned. Otherwise, the request is forwarded to the model provider and both the response and embedding are cached. *** ## Infrastructure Setup For **SaaS** customers, caching infrastructure is fully managed — no additional setup is required. For **On-Premise** deployments, caching requires two infrastructure components: 1. **Redis** — cache store for both exact-match and semantic caching 2. **Embedding Model** — required for semantic caching to generate vector embeddings (see [Embedding Model](#embedding-model)) ### Redis Setup You can either enable the bundled Redis instance in the `tfy-llm-gateway` Helm chart, or connect your own Redis (or Redis-compatible store such as [Valkey](https://valkey.io/)). Enable Redis directly in the `tfy-llm-gateway` chart values: ```yaml theme={"dark"} tfy-llm-gateway: redis: enabled: true ``` Disable the bundled Redis and configure connection details via environment variables: ```yaml theme={"dark"} tfy-llm-gateway: redis: enabled: false env: REDIS_HOST: "" REDIS_PORT: 6379 REDIS_USERNAME: "" REDIS_PASSWORD: "" ``` | Variable | Required | Type | Description | | ---------------- | -------- | -------- | ----------------------------------------------------------------------------------------- | | `REDIS_HOST` | Yes | `string` | Hostname or IP of your Redis instance | | `REDIS_PORT` | Yes | `number` | Port number (default: `6379`) | | `REDIS_USERNAME` | No | `string` | Username for Redis authentication | | `REDIS_PASSWORD` | No | `string` | Password for Redis authentication | | `REDIS_DB` | No | `number` | Redis database index to use (default: `0`). Applies to both standalone and Sentinel modes | Enable an encrypted connection by setting `REDIS_TLS_ENABLED: true`. TLS applies to both standalone and [Sentinel](#how-do-i-connect-to-a-highly-available-redis-using-sentinel) connections. The certificate material (`REDIS_TLS_CA_CERT`, `REDIS_TLS_CERT`, `REDIS_TLS_KEY`) accepts **either a path to a mounted PEM file or the inline PEM contents**. ```yaml theme={"dark"} tfy-llm-gateway: redis: enabled: false env: REDIS_HOST: "" REDIS_PORT: 6379 REDIS_TLS_ENABLED: true # Trust a private CA (path to a mounted PEM file, or inline PEM) REDIS_TLS_CA_CERT: "/etc/redis-certs/ca.crt" # Optional: SNI / expected certificate hostname if it differs from REDIS_HOST REDIS_TLS_SERVERNAME: "redis.internal" # Mutual TLS (client certificate) — CERT and KEY must be set together REDIS_TLS_CERT: "/etc/redis-certs/client.crt" REDIS_TLS_KEY: "/etc/redis-certs/client.key" REDIS_TLS_KEY_PASSPHRASE: "" ``` | Variable | Required | Type | Description | | ------------------------------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `REDIS_TLS_ENABLED` | Yes | `boolean` | Set to `true` to connect over TLS (default: `false`) | | `REDIS_TLS_CA_CERT` | No | `string` | CA certificate to trust, as a path to a mounted PEM file or inline PEM. Use when Redis presents a certificate signed by a private CA | | `REDIS_TLS_REJECT_UNAUTHORIZED` | No | `boolean` | Whether to reject a server whose certificate can't be verified (default: `true`). Only set to `false` for testing | | `REDIS_TLS_SERVERNAME` | No | `string` | Server name (SNI) to send and hostname to verify against, if it differs from `REDIS_HOST` | | `REDIS_TLS_CERT` | No | `string` | Client certificate for mutual TLS, as a path or inline PEM. Must be set together with `REDIS_TLS_KEY` | | `REDIS_TLS_KEY` | No | `string` | Client private key for mutual TLS, as a path or inline PEM. Must be set together with `REDIS_TLS_CERT` | | `REDIS_TLS_KEY_PASSPHRASE` | No | `string` | Passphrase for the client private key, if it is encrypted | * **PEM material** (`REDIS_TLS_CA_CERT`, `REDIS_TLS_CERT`, `REDIS_TLS_KEY`) can be a path to a Kubernetes-mounted file or the inline PEM string. A value that is neither a readable file nor valid PEM fails loudly at startup. * **Mutual TLS** requires both `REDIS_TLS_CERT` and `REDIS_TLS_KEY` — setting only one is a configuration error. * These TLS settings apply to both standalone and Sentinel connections. For high-availability Redis deployments that use [Redis Sentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) for automatic failover, point the gateway at your Sentinel nodes instead of a single Redis host. In Sentinel mode the gateway discovers the current master through the Sentinels and automatically follows failovers. Enable Sentinel by setting `REDIS_SENTINEL_ENABLED: true` along with the Sentinel node list and master name: ```yaml theme={"dark"} tfy-llm-gateway: redis: enabled: false env: REDIS_SENTINEL_ENABLED: true REDIS_SENTINEL_NODES: "sentinel-0:26379,sentinel-1:26379,sentinel-2:26379" REDIS_SENTINEL_MASTER_NAME: "mymaster" # Optional: credentials for the Sentinel processes themselves REDIS_SENTINEL_USERNAME: "" REDIS_SENTINEL_PASSWORD: "" # Data-node (master/replica) auth reuses the standard Redis credentials REDIS_USERNAME: "" REDIS_PASSWORD: "" ``` | Variable | Required | Type | Description | | ---------------------------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `REDIS_SENTINEL_ENABLED` | Yes | `boolean` | Set to `true` to use Sentinel mode (default: `false`) | | `REDIS_SENTINEL_NODES` | Yes | `string` | Comma-separated list of Sentinel nodes as `host:port` (e.g. `sentinel-0:26379,sentinel-1:26379`). If a port is omitted, `26379` is assumed | | `REDIS_SENTINEL_MASTER_NAME` | Yes | `string` | Name of the master group monitored by Sentinel (e.g. `mymaster`) | | `REDIS_SENTINEL_USERNAME` | No | `string` | Username for authenticating to the Sentinel nodes | | `REDIS_SENTINEL_PASSWORD` | No | `string` | Password for authenticating to the Sentinel nodes | * **Sentinel mode activates only when `REDIS_SENTINEL_ENABLED` is `true` *and* both `REDIS_SENTINEL_NODES` and `REDIS_SENTINEL_MASTER_NAME` are set.** If any of these is missing, the gateway falls back to standalone mode using `REDIS_HOST`. * **Data-node authentication** (to the Redis master and replicas) uses `REDIS_USERNAME` / `REDIS_PASSWORD`. The `REDIS_SENTINEL_USERNAME` / `REDIS_SENTINEL_PASSWORD` variables authenticate to the Sentinel processes only. * **TLS** applies to both the Sentinel and data-node connections — enable it with the same `REDIS_TLS_*` variables used for standalone Redis. ### Embedding Model For embedding model, here is the configuration required based on your deployment mode: 1. **TrueFoundry SaaS:** No setup required, we use OpenAI's `text-embedding-3-small` model by default. This model is not configurable. 2. **Self Hosted (Control Plane + Gateway Plane):** You can configure the model by going to "Settings" → "Semantic Cache" and select the embedding model. (should already be added as in integration) 3. **Hybrid**: (TrueFoundry Control Plane + Self Hosted Gateway Plane): For this, you need add an environment variable in the values of tfy-llm-gateway helm chart.\ You can add your embedding model (already registered on TrueFoundry) using the following change: ```yaml theme={"dark"} tfy-llm-gateway: env: SEMANTIC_CACHE_MODEL_IDENTIFIER: "openai-account/embedding-model-name" #model identifier on TrueFoundry for the embedding model ```