Troubleshoot from symptoms

The first step in troubleshooting is always to get the z-request-id — it's the single clue that runs through client, gateway, upstream, and logs.

Get the z-request-id first

Every response header carries z-request-id (format <iso8601>-<uuid4>). When a client errors, ask the client for this header from the response.

When you can't get the response header

If the client fails before receiving a response (connection refused, TLS failure), there is no z-request-id — this means the gateway never received the request or it was cut before reception, so nothing shows in the gateway-side logs; check the network layer first (port, firewall, reverse proxy).

Go to Console → logs page and paste this ID into the search box to locate it. Log details contain the complete request/response lifecycle; see the "In practice: locate client problems" section of Export billing Excel and troubleshoot logs.

Client gets 401 / 403

Status code	Common error_type values	Meaning	Handling
401	authentication_error	No access key provided or the key is invalid/disabled	Check that Authorization: Bearer is followed by an access key you issued (not an upstream sk-...); whether the key is disabled
403	model_access_denied	The key group's models list doesn't include that model	Add the model (or *) back to the key group
403	model_access_denied	The key group's load_balancers doesn't include that LB	Check it on the key group's "Load Balancers" sub-tab (* doesn't include LBs)
403	ip_banned	The IP is banned (triggered by consecutive login failures)	Wait a few minutes (default 300 seconds) for auto-unban

The LB dimension of 403 is the most common misconfiguration point; see Multi-tenant isolation.

Client gets 404

• model_not_found: the model name is misspelled, or the model has hide_name set (only accessible by alias). See "Hide the primary name" in Upstream and model fields.
• Alias vs canonical name: the key group's models only matches the canonical name, not aliases.
• See Error code reference for the complete 404 error codes; for 404 model-name alignment in the Claude Code scenario, see Claude Code via gateway → FAQ.

Client gets 400

error_type	Cause	Handling
bad_request	Request JSON parse failed / field semantic validation failed	Check the request-body JSON and required fields
unsupported_feature	This "client protocol → upstream protocol" pair has no translator	Check the Protocol interop matrix and change the protocol
invalid_request_error (Anthropic client)	Duplicate tool_use ids in the conversation history	Deduplicate tool_use ids; see Error code reference

Image-bearing requests rejected with 400 by the upstream (a text model received an image-bearing request): add a switch-route rule to the model to redirect image-bearing requests to a vision model; see Scenario 1 of Enhance model capability and Route image-bearing requests to a vision model.

Client gets 429

429 has multiple sources with different body formats and header presence; first distinguish which one it is (see Error code reference → Sources of 429 for the complete list):

Source	Body / header characteristics	Handling
Per-access-key rate limit	Plain-text body, no Retry-After	That key's requests are too frequent; split keys or slow down, and if needed adjust RATE_LIMIT_RPM
Per-IP rate limit	JSON body, no Retry-After	Too many requests from the same IP; adjust IP_RATE_LIMIT_RPM
Upstream rate limit	JSON body, with Retry-After header	The upstream itself rate-limits; back off per Retry-After, or add keys / upstreams to spread load
Overload admission (queue depth / connection budget / memory gate)	JSON body, with Retry-After header	The gateway itself is healthy but overloaded in-flight; back off per Retry-After. Persistent occurrence means insufficient capacity — scale up, or review the budget configs in Memory admission and spill and Listening and request limits

See Audit and security configuration → Rate limit for the rate-limit layer configs; for the 429 triggered by the memory gate and memory levels, see "Gateway memory rises abnormally" below.

Client gets 502 / 503

Status code	Source	Handling
502 bad_gateway	Upstream unreachable or returned something unexpected	Check the upstream base_url, API key, and upstream reachability
503	All nodes unavailable (LB retry budget exhausted) / gateway draining	In an LB scenario, check each entry's upstream; draining is a normal shutdown
503 upstream_saturated	All candidate upstreams saturated (bad-link budget full)	Carries Retry-After; back off per it; open Console Settings → System's "upstream health" block to locate the hung upstream and check its connectivity
504 outbound_deadline_exceeded	The outbound attempt-chain total budget was exhausted	The upstream had no first response for a long time; check that upstream's connectivity and timeout config

Persistent 503 means all nodes are down; check each upstream. For the Bedrock scenario, note that you should fill in the Bedrock API key, not the IAM key; see Connect AWS Bedrock.

Requests are slow or time out

• Client times out but the upstream is fine: check UPSTREAM_SEND_TIMEOUT_SECS (send request body + wait for response header, default 180) / UPSTREAM_READ_TIMEOUT_SECS (single-chunk read, default 600). The whole upstream lifecycle also has a fixed 3600-second hard cap (fixed, not exposed via environment variables). See Environment variable configuration reference.
• The upstream itself is slow (reasoning models thinking long): raise UPSTREAM_READ_TIMEOUT_SECS, which resets per chunk.
• Cross-border upstream zombie connections: for cross-border, shorten the reverse-proxy/upstream-side keepalive; that TCP keepalive parameter isn't exposed via environment variables — contact support if needed.

Streaming cuts off mid-way

• Upstream drops after the first chunk is sent: no node switch; record body_incomplete and return the received portion to the client.
• Gateway-side timeout: STREAMING_KEEPALIVE_SECONDS (default 15) sends SSE :keep-alive, and must be shorter than the reverse proxy's idle timeout (Nginx default 60). A too-short reverse-proxy timeout will cut long streams.
• Streaming duration or idle exceeds the cap: MAX_STREAM_DURATION_SECS (default 3600) is the single-stream total-duration cap, and STREAM_IDLE_TIMEOUT_SECS (default 600) is the idle cap between two chunks; exceeding either cuts the stream.

The streaming-related environment variables above (STREAMING_KEEPALIVE_SECONDS / MAX_STREAM_DURATION_SECS / STREAM_IDLE_TIMEOUT_SECS): see Environment variable configuration reference → Streaming.

Gateway memory rises abnormally

• A large-body scenario uses an after-slot script for pure routing decisions: switch to a switch-route rule instead. See the anti-patterns in Script performance and memory.
• max_request_size_mb / max_response_body_mb set too large: large multimodal conversations consume memory.
• The memory gate triggers 429: adjust memory_soft_limit_mb or check for abnormally large burst traffic.

Still can't locate it

1. Get the z-request-id and open the logs page to view the complete request/response/error details.
2. Cross-reference the error code with Error code reference.
3. For real-time observation use the logs page Live toggle (SSE stream); see Logs viewer.
4. In multi-instance, Live aggregates logs from all instances.

FAQ

Q: Can't find this z-request-id in the logs? The request may not have reached the gateway (network-layer break), or it was in an early 401 failure phase (not persisted). Check whether the client actually connected to the gateway port.

Q: What does the error body format look like? Three formats (depending on the client protocol and the error phase); see Error code reference.

Next: Error code reference for the complete error codes; Logs viewer for the UI; Script performance and memory for memory anti-patterns.