Skip to main content

​
Prompt caching

Prompt caching lets the inference router reuse work from an earlier request when a later request uses the same model and begins with the same prompt prefix. The first request populates the cache. Later requests can reuse the matching prefix while it remains available, reducing latency and the work required before generating new tokens. See API pricing for the latest cached-token rates for selected models. For the design and security details behind this feature, read the secure prompt caching blog post. To get the most out of caching, put stable content, such as system instructions and tool definitions, near the beginning of the prompt and request-specific content later. Changing an earlier token shortens or eliminates the reusable prefix. Prompt caches are temporary and may be evicted. Your application must remain correct when a request does not produce a cache hit.

​
How the SDKs handle it

You do not need to configure anything to benefit from prompt caching. Each Tinfoil SDK automatically manages a value called user_cache_secret that scopes which requests can share cached prefixes:
  1. The SDK resolves a stable secret. By default it generates a random one and stores it at ~/.tinfoil/user_cache_secret, so SDK clients on the same machine reuse it. Browsers and other environments without usable persistence use a runtime-lifetime value instead.
  2. The SDK adds the secret to eligible inference requests: JSON POST requests for chat completions, legacy completions, and responses. Endpoints that do not use prompt-prefix caching, such as embeddings, audio, files, models, and realtime WebSockets, are left unchanged.
  3. The router combines the secret with your authenticated API identity to select a cache namespace. Requests reuse cached prefixes only when the API identity, secret, model, and prompt prefix all match.
user_cache_secret is used only for cache partitioning. It is not an API credential or encryption key, and it does not encrypt prompts or cache entries. Treat it as sensitive application data and avoid logging it, since requests sharing an identity and secret also share cache-hit timing.

​
Configuring the secret

Most applications can use the generated default. When your deployment needs different scoping, you can override the secret at three levels. The first non-empty value wins:
  1. A user_cache_secret string set in an individual request body.
  2. The client option (for example userCacheSecret in JavaScript or user_cache_secret in Python).
  3. The TINFOIL_USER_CACHE_SECRET environment variable.
Otherwise the SDK falls back to the generated secret. Empty values at any level are treated as unset. Choose the scope that matches your deployment:
  • Single-user application: use the generated default.
  • Multi-user application: provide a stable, distinct, non-empty value for each user (or group that may share a cache) on every eligible request.
  • Multiple containers or machines: use the same stable, non-empty value wherever requests should share one cache.
Use opaque random values rather than raw user identifiers or API keys. Without a per-user secret, requests under the same API identity share one cache, so a client-level or environment-level value alone does not separate users in a multi-user service. Requests sent through a raw HTTP client, or direct API access without an SDK, do not get a secret injected automatically and must set the field themselves where cache separation is required.

​
Minimum SDK versions

Use at least these SDK versions to benefit from secure prompt caching: