The response cache

The response cache answers a repeated request without a call to a provider. It is on for every project on proxium.tech. This page explains how Proxium finds a cached answer, what the answer costs, and where the cache does not apply.

The same question, in other words

Proxium looks at the last question of a request and at the conversation around it. When the conversation is the same and the question asks the same thing, a cached answer can serve it, even when the words differ. "How long does a refund take?" and "How many days until I get my refund?" can share an answer. "Refund for a card" and "refund for a gift card" do not.

Proxium makes no embedding for this. It compares the text first. When the words differ, Jev, the decision model of TypeSafe, decides. Jev answers for a project that has it included, or that added its own TypeSafe key. For other projects, a question in the same words still gets the cached answer. When your app asks for model: auto, the same Jev call also picks the tier.

Some requests use only the exact match below, for example a request whose last turn is a tool result or holds an image.

Exact match

The key of a cached answer is a SHA-256 hash of these four things:

• the project;
• the first model of the resolved tier;
• the endpoint;
• the request body, with the model field that your app sent.

A cached answer serves that request, whichever model of the plan made it. A model or vendor that your project retires keeps its cached answers until they expire. Only a body that is the same, field for field, gets a hit. An answer with no content and no tool call is not stored, so a repeat reaches the model again.

Each cached answer expires after a while, or after the time that the project sets in Settings › Cache. After that time, the same request goes to a provider again.

If the cache fails, Proxium continues as on a miss. A cache error never stops a call.

Fig. 1 · What the cache does with a request

A chat or Messages request arrives

1. 1Does an earlier question in the same conversation ask the same thing?In the same or other words
   yes
   The stored answer. No vendor call
   no
2. 2Is the same request in the cache, and younger than the cache time?Same project, model, route and body, except stream, stream_options and user
   yes
   The stored answer. No vendor call
   no

Proxium calls the models, then stores the answer

What is in the key

The model in the key is the first model of the route that Proxium resolved. If you change the routing of a tier, the new model gets new cache entries.

Proxium removes three fields from the body before it makes the key: stream, stream_options and user. All other fields stay in the key, temperature and tools included.

Because the key has no user field, two end users of one project who send the same body get the same answer. If an answer must depend on the end user, put something of that end user in the messages.

A call that sends x-proxium-memory: recall and gets memories has those memories in its body, so the key includes them. Use project memory explains recall.

One project never reads another project's cache

The project is the first part of the key. A request of one project cannot match an entry of another project.

Which calls the cache serves

The cache serves two routes: /v1/chat/completions and /v1/messages. A Messages request goes through the same chat path, so it uses the same cache.

Route	Response cache
/v1/chat/completions	Yes
/v1/messages	Yes
/v1/responses	No
/v1/embeddings, /v1/moderations, /v1/rerank	No
/v1/images/generations, /v1/audio/speech	No
/video/generations, /video/operations, /video/download	No

The routes without a cache send every request to a provider.

Streaming

stream is not in the key. A streamed request and a buffered request with the same body share one entry.

• If a streamed request gets a hit, Proxium sends the cached answer as server-sent events. A Messages client gets the Anthropic event sequence.
• If a streamed request misses, Proxium stores the answer when the stream is complete. Proxium never stores a stream that stopped before the end.

The answer in the cache always has the shape of a complete, non-streamed answer.

Where in the request the lookup happens

Proxium looks up the cache before it reserves the budget. Two results follow from that order:

• A hit does not call a provider. By default it is free and does not count against the call limits.
• When a hit is free, a project at its budget ceiling still gets hits.

What a hit costs

By default a cached answer is free.

The operator of Proxium can set a price for the hits of a project. The price is Free, Quarter, Half or Full price of the live call. The operator can also set a different price for one x-proxium-source or one virtual key. A sender price wins over a key price, and both win over the project price.

A member of the project cannot see or change this price. On proxium.tech, each project is on Free unless the operator set another price.

When the charge is above zero, a hit is also a request. It counts as one call against the call limits of the project and the key. Proxium refuses it when the project is at its ceiling.

See what the cache saved

• Overview shows Saved by cache: the money the project did not spend in the window.
• Requests shows Cache saved and a Cache panel with the hits over time.

The answer itself does not say that it came from the cache. Use the Cache panel on Requests to count hits.

Change the cache of a project

The cache is on for every project. In Settings › Cache, a project can:

• turn the cache off;
• keep an answer for a shorter or a longer time;
• turn off the match of a question in other words, where the project has it.

The change applies to the next request. No request header turns the cache off for one call.

A request gets a fresh answer from a provider when one of these is true:

• The cache of the project is off.
• No cached request asks the same thing in the same conversation.
• The cached answer is older than the time the project keeps answers.
• It goes to a route without a cache, such as /v1/responses.

When a project is erased

The cache cannot list the entries of one project, because each key is a hash. The entries of an erased project expire after a while. The erasure deletes every virtual key of the project first, so nobody can read them in that time.