> Raw Markdown twin (generated at build time from the source Markdown). Rendered page: https://docs.gatellm.io/en/usecases/troubleshooting · Doc index: https://docs.gatellm.io/en/llms.txt


# Troubleshoot from symptoms

The first step in troubleshooting is always to get the `z-request-id` — it's the single clue that runs through client, gateway, upstream, and logs.

## Get the z-request-id first

Every response header carries `z-request-id` (format `<iso8601>-<uuid4>`). When a client errors, ask the client for this header from the response.

::: tip When you can't get the response header
If the client fails before receiving a response (connection refused, TLS failure), there is no `z-request-id` — this means the gateway never received the request or it was cut before reception, so nothing shows in the gateway-side logs; check the network layer first (port, firewall, reverse proxy).
:::

Go to Console → logs page and paste this ID into the search box to locate it. Log details contain the complete request/response lifecycle; see the "In practice: locate client problems" section of [Export billing Excel and troubleshoot logs](/en/howto/billing-export-and-logs.md).

## Client gets 401 / 403

| Status code | Common error_type values | Meaning | Handling |
|--------|------------------|------|------|
| 401 | authentication_error | No access key provided or the key is invalid/disabled | Check that `Authorization: Bearer` is followed by an access key you issued (not an upstream `sk-...`); whether the key is disabled |
| 403 | model_access_denied | The key group's `models` list doesn't include that model | Add the model (or `*`) back to the key group |
| 403 | model_access_denied | The key group's `load_balancers` doesn't include that LB | Check it on the key group's "Load Balancers" sub-tab (`*` doesn't include LBs) |
| 403 | ip_banned | The IP is banned (triggered by consecutive login failures) | Wait a few minutes (default 300 seconds) for auto-unban |

The LB dimension of 403 is the most common misconfiguration point; see [Multi-tenant isolation](/en/usecases/multi-tenant-isolation.md#lb-is-auth-unit).

## Client gets 404

- `model_not_found`: the model name is misspelled, or the model has `hide_name` set (only accessible by alias). See "Hide the primary name" in [Upstream and model fields](/en/reference/upstreams-models-fields.md).
- Alias vs canonical name: the key group's `models` only matches the canonical name, not aliases.
- See [Error code reference](/en/reference/error-codes.md) for the complete 404 error codes; for 404 model-name alignment in the Claude Code scenario, see [Claude Code via gateway → FAQ](/en/quickstart/claude-code-via-gateway.md#faq).

## Client gets 400

| error_type | Cause | Handling |
|--------|------|------|
| `bad_request` | Request JSON parse failed / field semantic validation failed | Check the request-body JSON and required fields |
| `unsupported_feature` | This "client protocol → upstream protocol" pair has no translator | Check the [Protocol interop matrix](/en/reference/protocol-matrix.md) and change the protocol |
| `invalid_request_error` (Anthropic client) | Duplicate `tool_use` ids in the conversation history | Deduplicate `tool_use` ids; see [Error code reference](/en/reference/error-codes.md) |

**Image-bearing requests rejected with 400 by the upstream** (a text model received an image-bearing request): add a `switch-route` rule to the model to redirect image-bearing requests to a vision model; see Scenario 1 of [Enhance model capability](/en/usecases/model-enhancement.md) and [Route image-bearing requests to a vision model](/en/howto/route-image-requests-to-vision-model.md).

## Client gets 429

429 has multiple sources with different body formats and header presence; first distinguish which one it is (see [Error code reference → Sources of 429](/en/reference/error-codes.md#source-of-429) for the complete list):

| Source | Body / header characteristics | Handling |
|------|------------|------|
| Per-access-key rate limit | **Plain-text** body, no `Retry-After` | That key's requests are too frequent; split keys or slow down, and if needed adjust `RATE_LIMIT_RPM` |
| Per-IP rate limit | JSON body, no `Retry-After` | Too many requests from the same IP; adjust `IP_RATE_LIMIT_RPM` |
| Upstream rate limit | JSON body, **with** `Retry-After` header | The upstream itself rate-limits; back off per `Retry-After`, or add keys / upstreams to spread load |
| Overload admission (queue depth / connection budget / memory gate) | JSON body, **with** `Retry-After` header | The gateway itself is healthy but overloaded in-flight; back off per `Retry-After`. Persistent occurrence means insufficient capacity — scale up, or review the budget configs in [Memory admission and spill](/en/reference/configuration.md#memory-admission-spill) and [Listening and request limits](/en/reference/configuration.md#listen-and-request-limits) |

> See [Audit and security configuration → Rate limit](/en/reference/audit-and-security-config.md#rate-limit) for the rate-limit layer configs; for the 429 triggered by the memory gate and memory levels, see "Gateway memory rises abnormally" below.

## Client gets 502 / 503

| Status code | Source | Handling |
|--------|------|------|
| 502 bad_gateway | Upstream unreachable or returned something unexpected | Check the upstream `base_url`, API key, and upstream reachability |
| 503 | All nodes unavailable (LB retry budget exhausted) / gateway draining | In an LB scenario, check each entry's upstream; draining is a normal shutdown |
| 503 upstream_saturated | All candidate upstreams saturated (bad-link budget full) | Carries `Retry-After`; back off per it; open Console Settings → System's "upstream health" block to locate the hung upstream and check its connectivity |
| 504 outbound_deadline_exceeded | The outbound attempt-chain total budget was exhausted | The upstream had no first response for a long time; check that upstream's connectivity and timeout config |

Persistent 503 means all nodes are down; check each upstream. For the Bedrock scenario, note that you should fill in the Bedrock API key, not the IAM key; see [Connect AWS Bedrock](/en/quickstart/aws-bedrock.md).

## Requests are slow or time out

- Client times out but the upstream is fine: check `UPSTREAM_SEND_TIMEOUT_SECS` (send request body + wait for response header, default 180) / `UPSTREAM_READ_TIMEOUT_SECS` (single-chunk read, default 600). The whole upstream lifecycle also has a fixed 3600-second hard cap (fixed, not exposed via environment variables). See [Environment variable configuration reference](/en/reference/configuration.md).
- The upstream itself is slow (reasoning models thinking long): raise `UPSTREAM_READ_TIMEOUT_SECS`, which resets per chunk.
- Cross-border upstream zombie connections: for cross-border, shorten the reverse-proxy/upstream-side keepalive; that TCP keepalive parameter isn't exposed via environment variables — contact support if needed.

## Streaming cuts off mid-way

- Upstream drops after the first chunk is sent: no node switch; record `body_incomplete` and return the received portion to the client.
- Gateway-side timeout: `STREAMING_KEEPALIVE_SECONDS` (default 15) sends SSE `:keep-alive`, and must be shorter than the reverse proxy's idle timeout (Nginx default 60). A too-short reverse-proxy timeout will cut long streams.
- Streaming duration or idle exceeds the cap: `MAX_STREAM_DURATION_SECS` (default 3600) is the single-stream total-duration cap, and `STREAM_IDLE_TIMEOUT_SECS` (default 600) is the idle cap between two chunks; exceeding either cuts the stream.

The streaming-related environment variables above (`STREAMING_KEEPALIVE_SECONDS` / `MAX_STREAM_DURATION_SECS` / `STREAM_IDLE_TIMEOUT_SECS`): see [Environment variable configuration reference → Streaming](/en/reference/configuration.md#streaming).

## Gateway memory rises abnormally

- A large-body scenario uses an after-slot script for pure routing decisions: switch to a `switch-route` rule instead. See the anti-patterns in [Script performance and memory](/en/practices/script-performance.md).
- `max_request_size_mb` / `max_response_body_mb` set too large: large multimodal conversations consume memory.
- The memory gate triggers 429: adjust `memory_soft_limit_mb` or check for abnormally large burst traffic.

## Still can't locate it

1. Get the `z-request-id` and open the logs page to view the complete request/response/error details.
2. Cross-reference the error code with [Error code reference](/en/reference/error-codes.md).
3. For real-time observation use the logs page Live toggle (SSE stream); see [Logs viewer](/en/console/logs-viewer.md).
4. In multi-instance, Live aggregates logs from all instances.

## FAQ

**Q: Can't find this z-request-id in the logs?**
The request may not have reached the gateway (network-layer break), or it was in an early 401 failure phase (not persisted). Check whether the client actually connected to the gateway port.

**Q: What does the error body format look like?**
Three formats (depending on the client protocol and the error phase); see [Error code reference](/en/reference/error-codes.md).

**Next**: [Error code reference](/en/reference/error-codes.md) for the complete error codes; [Logs viewer](/en/console/logs-viewer.md) for the UI; [Script performance and memory](/en/practices/script-performance.md) for memory anti-patterns.