Achieve high availability

The core of high availability is eliminating single points of failure: single model multi-node → single provider multi-copy → multi-instance shared state. This page ties together the gateway's relevant capabilities without repeating the step-by-step LB setup (see Multi-upstream load balancing).

Goals and failure surfaces

Failure surface	Typical cause	Countermeasure
Single upstream key invalidated	Rate-limited, revoked	Configure multiple upstream keys with automatic rotation on failure
Single upstream instance unavailable	Upstream down, network partition	LB across multiple upstreams of the same model
Single provider unavailable	Provider-level failure	Cross-provider LB
Single gateway process unavailable	Process crash, host failure	Multi-instance + shared storage

Single model multi-node: load balancer

Combine multiple upstream+model combinations of the same model into one load balancer's entries, and have clients call only the LB name. When a node returns a retryable error (5xx, connection/read timeout), the gateway automatically switches nodes and retries. See Create a load balancer and case A of Multi-upstream load balancing for setup.

Canary rollout of a new model

When introducing a new model (or new node), ramp traffic gradually using weights rather than switching all at once:

1. Add the new node to the existing load balancer's entries with a small weight — e.g. old node weight=10, new node weight=1; the gateway distributes proportionally by weight, so the new node carries only about 1/11 of traffic.
2. Observe the new node's success rate and latency (Console → Load Balancers → Entries panel, or the statistics page by model); once stable, gradually increase its weight.
3. On problems, lower the new node's weight or set it to 0 to roll back; traffic automatically returns to the old node.

Difference from a weight=0 standby node: a standby node normally carries no traffic and only steps in when all normal nodes are unavailable; a canary node is given a small real weight and actually carries a portion of traffic to validate quality. See Load-balancing fields for weight semantics.

Provider-level disaster recovery

An LB with OpenAI as primary and Azure OpenAI as backup: two entries with different upstreams and different model names, retry_on_different_node=true. When all primary nodes fail it switches to backup. See case B of Multi-upstream load balancing.

Retry and streaming boundaries

• The retry count is controlled by MAX_UPSTREAM_RETRIES (see Load-balancing fields).
• Streaming retries are only attempted before the first SSE data chunk is sent. Once the client starts receiving data, you can't switch nodes mid-stream — doing so would corrupt the partial state already emitted. So the failover window in streaming scenarios is "before the first byte".

Probes and canary shutdown

• /health returns 200 when alive.
• /ready returns 503 when the database is unreachable, the gateway is draining (drain mode), or the entry disk volume hangs — point your external load balancer's (Nginx / ALB) health check at /ready to automatically stop distributing traffic when the gateway is draining or the disk fails, achieving canary shutdown, zero-downtime restart, and automatic removal of a hung volume.

Multi-instance shared state

Single machine uses SQLite; multi-instance uses PostgreSQL to share statistics and configuration (Console changes sync across instances in 30s), plus Redis to share sessions/rate limiting/IP bans/log broadcast. See the STORAGE_MODE / POSTGRES_URL / REDIS_URL fields in Environment variable configuration reference.

Multi-instance without Redis

Without Redis in a multi-instance setup, sessions, rate limiting, and IP bans each degrade to per-instance independence — a caller's failure count is spread across different instances, making the ban threshold useless. Always configure Redis for multi-instance.

Integrate monitoring (Prometheus / OpenTelemetry)

The gateway exposes runtime metrics to plug into your existing observability stack:

• Prometheus: GET /metrics exposes Prometheus metrics. First set an auth token via the METRICS_AUTH_TOKEN environment variable, then access with Authorization: Bearer <token>; without a token this endpoint returns 403 (preventing unauthorized scraping). In Prometheus's scrape config, add the gateway address with a Bearer header, and Grafana uses Prometheus as a data source to chart.
• OpenTelemetry: set OTEL_ENDPOINT to point at your OTLP collector and the gateway reports traces/spans; OTEL_SERVICE_NAME (default protoflux) distinguishes the source across services. Without OTEL_ENDPOINT, nothing is reported.

See Environment variable configuration reference for fields and defaults, and Endpoints · auth · protocol interop for the endpoint list.

Pre-launch self-check

See Production checklist.

FAQ

Q: What if the upstream drops mid-stream? If the first chunk has already been sent, it doesn't switch nodes; the gateway records body_incomplete and returns the received portion to the client. The client handles the interruption per the SSE protocol.

Q: When is the standby node (weight=0) used? Only when all normal nodes with weight > 0 are unavailable. Once a normal node recovers, traffic automatically switches back.

Q: How are sessions shared across multiple instances? Sessions are stored in Redis (REDIS_URL) and shared by all instances; without Redis each instance is independent and login only applies to the current instance.

Next: Multi-tenant isolation for isolation; Production checklist for pre-launch self-check; Load-balancing fields for fields.