Lower your costs
The Improve screen reads the traffic of your project, and lists what to change. Each finding shows the numbers that it comes from.
Read the findings
Open Improve in the console. The readouts on top count the findings:
| Readout | Shows |
|---|---|
| Findings | All findings |
| Worth doing first | The findings of high severity |
| Worth a look | The other findings |
| Worth fixing | The estimated saving of each month, in dollars, when Proxium can compute one |
What to change lists the findings, high severity first. These are the kinds:
| Kind | Proxium flags | Severity |
|---|---|---|
| Cost | One of your largest spend lines this month, when the same vendor has a chat model that costs much less | High when the saving is large |
| Cache | A project with real traffic this month, where the cache saved little | High when the saving is large |
| Reliability | A vendor where many recent attempts failed | High when most of them failed |
| Retired model | A model that your failover policy retired recently, with the last failure | High while it is retired |
| Retired vendor | A vendor that your failover policy retired recently: no credits, a rejected key, or failed calls across its models | High while it is retired |
| Unknown model | An app that asks for a model name that the vendor does not list, with its recent calls and the model that answered them | High |
If the list says Nothing to flag, your traffic has none of these patterns.
Move traffic to a cheaper model
For a cost finding:
- Read the cheaper model that the finding names, and its price.
- Open Routing › Model catalog, and compare Cost, per Mtok of the two models.
- Test the cheaper model on a few real prompts of the app.
- In Text routing, select Change on the tier of the app, and put the cheaper model first.
- Select Save for this project.
Your apps need no change, because they send the tier name. To move one app only, add a rule for that app. See Route requests to models.
Let the cache answer repeated calls
For a cache finding: the cache answers only a request that is the same, field for field, as an earlier one. These changes give more hits:
| Change | Reason |
|---|---|
| Keep the system prompt the same on each call | A changed character is a new request |
Send the same temperature and tools each time | Both fields are part of the cache key |
| Put a date or an id in a message only when the answer depends on it | Each new value is a new request |
Overview › Saved by cache shows what the cache saved. The response cache explains the key.
Fix a vendor that fails often
For a reliability finding:
- Open Requests, and read What failed and Who is unreliable for that vendor.
- If the reason is a rejected key or a used quota, fix the key at the vendor, or add it again on Providers.
- Else, move the vendor down in your chains, so that a better vendor answers first.
Each failed attempt adds time before the answer. Debug a failed call shows how to read the attempts.
Fix a retired model, a retired vendor or an unknown model
For a retired model finding: remove the model from the tiers that name it on Routing, or replace it with another model. Proxium does not change your tiers.
For a retired vendor finding, the fix depends on the reason:
- No credits: add credits at the vendor.
- A rejected key: replace the key on Providers.
- Failed calls: check the status of the vendor, or move its models down your tiers.
For an unknown model finding: fix the model name in the code of the app that the finding names. Or, to keep the name, make it a tier on Routing. Until you fix it, each call sends the name to the vendor first, until your failover policy retires it. See Retire a model or a vendor.
Related pages
- Track spend: where the money goes.
- Set budgets and limits: cap the spend of one app.