Reduce call cost

Gateway-side cost mainly comes from three places: upstream token billing, unnecessary response content, and the disk footprint of logs and log bodies. This page gives several mutually non-conflicting measures that can be stacked.

Where cost comes from

Cost source	Magnitude	Cost-reduction measure
Upstream token billing	Largest	Hit upstream prompt cache; route to cheaper models by complexity tiering
Multimodal response body	Medium	Trim unneeded image/audio content
Log and log-body disk	Small but continuous	Reduce retention period and body capture limit

Hit upstream prompt cache

Anthropic, OpenAI, and other upstreams give a prompt-cache discount for repeated system / prefix content. The gateway sticks the same caller's requests to the same upstream key (deterministic weighted distribution + sticky binding), keeping the prompt-cache prefix that the upstream sees stable.

• Don't spread the same caller across different keys on a multi-key upstream, otherwise each key has to rebuild its cache. See "Multiple keys and weights" in Upstream and model fields.
• When connecting via Claude Code via gateway, use a request_payload expression to write the x-claude-code-session-id header that Claude Code injects into the upstream private cache-key field (prompt_cache_key), so multiple requests in the same session hit the same cache — this is a request payload rule, not a script. See that page's Inject a session cache key section.

Tiered routing by input complexity

"Simple requests go to a cheap model, complex requests go to a strong model" is implemented with request_payload's switch-route rule — it determines by the client's request shape before protocol translation, and on a match redirects the request to the designated model, without reading body content and without consuming memory, and works across protocols.

Example: route to a cheap model when the request has no image block, and only go to a strong multimodal model when there is an image. The rule only does structural / small-value checks; see the switch-route mode in Upstream and model fields for configuration. For the selection decision see Request rewrite and routing: how to choose.

Trim unneeded response content

Use a response script to strip large blocks such as images and audio, keeping only text. See the "Strip image content" section of Response redaction with scripts for typical scenarios and scripts.

Log and log-body retention cost

• logging.max_body_size_mb (default 25 MB) limits the capture size of each body. Lower it to reduce disk usage, but large request bodies will be truncated.
• log_retention_days controls the retention days; lower it (e.g. 3 days) to speed up cleanup.
• Streaming body disk usage is controlled by stream_body_max_disk_mb.

See Log and log-body storage for field semantics.

How to verify the effect

• By token usage: Console → Statistics, view token trends by model / access-key dimension. See Statistics.
• By cost: after configuring a price snapshot in Pricing and billing, the monthly bill gives a per-model/key cost summary.
• By cache hits: the cache_read_input_tokens / cached_tokens fields in upstream responses show up in log details.

FAQ

Q: Does switch-route slow down requests? No. It makes structural determinations before protocol translation without reading binary content bytes; large request bodies still go to disk spooling and don't consume memory.

Q: Do response-redaction scripts slow down streaming responses? Response scripts run on the response body. Streaming responses don't retry after the first chunk is sent, and the script processes each chunk. Avoid heavy pure-routing-decision logic on large bodies; see Script performance and memory.

Next: Achieve high availability for disaster recovery; Request rewrite and routing: how to choose for selection; Pricing and billing for billing.