{% raw %} # Configuration [English](configuration.md) | [简体中文](configuration.zh-CN.md) ## Entry Point Protoflux separates **infrastructure** and **business** configuration: | Source | Content | Management | |--------|---------|------------| | `server.toml` | Infrastructure (server, web_console, cors, streaming, logging, rate_limit, retry, routing) | Manual file edit | | **Database** (SQLite/PostgreSQL) | Business (access_keys, upstreams, models, access_key_groups) | Console UI / API | Start from `server.example.toml` for infrastructure settings. Business config is managed entirely through the Console. **Storage modes**: - **SQLite** (default): business config is stored in structured per-row tables (`config_access_key`, `config_upstream`, `config_model`, `config_access_key_group`) within the SQLite database. Console edits are persisted in the DB. A 30-second polling loop detects changes for hot-reload. - **PostgreSQL**: business config is stored in the database. Console edits are shared across all instances via 30-second polling. `server.toml` always provides infrastructure settings. **Startup flow**: 1. Load `server.toml` (via `--config`) → infrastructure config 2. Load business config from the database (SQLite or PostgreSQL) 3. Merge → complete config ## Architecture Protoflux has two storage modes that share the same configuration management design: ### Design Philosophy Business configuration (access_keys, upstreams, models, access_key_groups) is managed through the **Web Console** (`/console`). The Console provides a visual interface to: - Configure upstream endpoints (protocol, base URL, API keys, credentials rotation) - Declare client-facing models and route them to upstreams - Manage access key access control (API keys, groups, model ACLs) - View request statistics and system health Infrastructure configuration (server binding, CORS, logging, streaming, rate limiting, retry policy) comes from `server.toml` and is **never** modified by the Console. This separation ensures infrastructure settings remain under operator control (version-controlled files, environment variables) while business configuration is dynamically managed at runtime. ### SQLite Mode (Single Instance) - Business config stored in structured per-row tables (`config_access_key`, `config_upstream`, `config_model`, `config_access_key_group`) of the local SQLite database - Request statistics stored in the same SQLite database - A 30-second polling loop detects config changes for hot-reload - Ideal for development, edge deployments, or single-node production ### PostgreSQL Mode (Multi-Instance) - Business config and statistics stored in a shared PostgreSQL database - Multiple gateway instances poll the same database (30-second interval) - Config edits from any instance propagate to all instances automatically - Refinery migrations auto-create the required tables on first boot - Ideal for container deployments with horizontal scaling ### Redis Distributed State When `redis_url` is configured, runtime state is shared across instances via Redis: - **Session**: users stay logged in when switching between instances - **Rate Limit**: distributed rate limiting prevents users from bypassing limits by rotating instances - **Log Broadcast**: Console SSE log stream aggregates request logs from all instances - **IP Ban**: brute-force IP bans are synchronized across all instances Redis only handles runtime state. Business configuration (access_keys/upstreams/models) is still persisted via PostgreSQL/SQLite. When `redis_url` is not set, all state is in-memory with zero overhead. Redis connection failure gracefully falls back to in-memory mode. ```toml [server] redis_url = "redis://redis-host:6379" redis_key_prefix = "protoflux:" # use different prefixes for multiple clusters sharing one Redis ``` ### Server Configuration The `[server]` section controls the HTTP server and storage backend: | Field | Type | Default | Description | |-------|------|---------|-------------| | `host` | string | `0.0.0.0` | Listen address | | `port` | u16 | `7890` | Listen port | | `max_request_size_mb` | f64 | `10` | Maximum request body size in MB (0.01–100) | | `upstream_idle_connections` | u32 | `32` | HTTP connection pool size per upstream host | | `upstream_user_agent` | string | `"Protoflux"` | `User-Agent` header for upstream requests. Supports env var: `${UPSTREAM_USER_AGENT}` | | `worker_threads` | u32 | `4` | Tokio async runtime worker thread count. Useful for containers with limited CPU (e.g., set to 2 for 0.5 CPU) | | `storage_mode` | string | `sqlite` | Storage backend: `sqlite` or `postgresql` | | `postgres_url` | string? | none | PostgreSQL connection URL (required when `storage_mode = "postgresql"`) | | `sqlite_path` | string | `protoflux.sqlite` | SQLite file path (only when `storage_mode = "sqlite"`) | | `redis_url` | string? | none | Redis connection URL. When set, enables distributed state sharing (sessions, rate limiting, log broadcast, IP bans). Whitespace-only values are trimmed and treated as not configured | | `redis_key_prefix` | string | `""` | Redis key prefix (empty string defaults to `"protoflux:"` at runtime). Use different prefixes when multiple gateway clusters share a Redis instance. Only relevant when `redis_url` is set; whitespace values are trimmed | | `log_queue_dir` | string? | none | Directory for the append-only file log queue (single-instance mode only). When not set, defaults to the system temp directory. For Docker, consider mounting a volume path | | `script_max_operations` | u64 | `10000` | Maximum JavaScript (QuickJS) script operations per evaluation. The sandbox engine uses this as a per-evaluation resource limit (no race condition). 10000 operations typically execute in < 5ms, providing a tight bound against malicious scripts | | `metrics_auth_token` | string? | none | Bearer token for `/metrics` endpoint authentication. When set, requests must include `Authorization: Bearer ` header. **Security**: the endpoint is disabled by default (returns 403) when this is empty or not configured | | `request_timeout_secs` | u64 | `120` | Global request timeout in seconds. Requests exceeding this duration are terminated with 408 Request Timeout. **Note**: SSE streaming requests are exempt from this timeout once the stream is established | | `graceful_shutdown_timeout_secs` | u64 | `30` | Maximum time to wait for in-flight requests to complete during graceful shutdown. After this timeout, the server forcefully terminates even if requests are still active | | `pre_stop_delay_secs` | u64 | `15` | Delay before initiating graceful shutdown after receiving SIGTERM. In Kubernetes deployments, this allows time for the pod to be removed from service endpoints before the server stops accepting new connections. Set to 0 to disable | | `ip_rate_limit_rpm` | u64? | none | Per-IP rate limit (requests per minute). When set, each client IP is limited to this many requests per minute using a sliding window algorithm. IPs are extracted from `X-Forwarded-For` header (first IP) or connection info. Complements per-user rate limiting | | `trusted_proxies` | string? | none | Comma-separated list of trusted proxy CIDRs (e.g., `"10.0.0.0/8,172.16.0.0/12"`). When configured, `X-Forwarded-For` header is parsed to extract the real client IP. When not set, only direct connection IPs are used for rate limiting | | `upstream_read_timeout_secs` | u64 | `360` | Maximum time to wait between consecutive data chunks from the upstream (applies to both the initial response and each subsequent stream read). Must be greater than `stream_idle_timeout_secs` so the gateway-level idle timeout fires first. Increase for extended thinking models (e.g., reasoning models that may pause 5+ minutes between tokens) | | `upstream_total_timeout_secs` | u64 | `3600` | Maximum total time for an upstream request, including response body reading. For SSE streams, `read_timeout` resets on each chunk, while `total_timeout` is an absolute ceiling. Protects against extremely long streams | | `tcp_keepalive_secs` | u64 | `50` | TCP keepalive interval for upstream connections. Controls how often the OS sends keepalive probes on idle connections. Shorter values detect zombie connections (silently dropped by NAT/LB) faster. For cross-border deployments, set to 10-15s; for same-region, the default 50s is sufficient | | `postgres_pool_size` | u32 | `16` | Maximum number of concurrent connections to PostgreSQL. Only relevant when `storage_mode = "postgresql"` | | `postgres_wait_timeout_secs` | u64 | `10` | Deadpool pool acquire timeout in seconds. When a slow query saturates the pool, requests past this timeout fail fast instead of hanging indefinitely — prevents one slow query from cascading to all DB-backed console pages. 0 = no timeout (wait forever, not recommended). Only when `storage_mode = "postgresql"` | | `postgres_statement_timeout_secs` | u64 | `30` | PostgreSQL per-statement timeout in seconds (stats pool only). Caps a single SQL statement via `SET statement_timeout` so a large range aggregation (e.g. 12h time-series scan) cannot hold a connection and saturate the RDS. 0 = no timeout. Only when `storage_mode = "postgresql"` | | `script_pool_size` | usize? | `4` | Number of concurrent script execution slots. The script engine maintains a pool of independent QuickJS contexts. Each slot consumes ~10MB memory. Increase if profiling shows mutex contention under high load | | `max_response_body_mb` | u32 | `25` | Maximum upstream response body size in MB for non-streaming responses (buffered in memory). Streaming responses are NOT affected — they are forwarded chunk-by-chunk. Increase for large non-streaming responses (e.g., base64 images, large embeddings) | | `max_global_concurrency` | u32? | `1000` | Maximum concurrent requests across all proxy routes. When the limit is reached, new requests receive HTTP 503. Protects against resource exhaustion under high load. Set to `None` or `0` to disable | | `encryption_key` | string? | none | Encryption key for sensitive data at rest (access keys, upstream API keys). AES-256-GCM key in 64-character hex format, or a passphrase (derived via SHA-256). Typically set via `${ENCRYPTION_KEY}` environment variable | | `encryption_key_deprecated` | string[] | `[]` | Deprecated encryption keys, retained **only for decryption** during key rotation. Data encrypted under any deprecated key can still be decrypted, but all new writes use the active `encryption_key`. Supports comma-separated hex strings via `${ENCRYPTION_KEY_DEPRECATED}` | | `request_log_capacity` | usize | `1000` | Request log ring buffer capacity. Number of recent request/response entries kept in the in-memory ring buffer exposed via `/console/api/requests`. When full, oldest entries are overwritten. ~1KB per entry | | `histogram_buckets` | f64[]? | none | Custom Prometheus histogram bucket boundaries for request duration metrics (in seconds). Built-in: `[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0]`. Override for workloads with different latency profiles (e.g., add 60.0/120.0/300.0 for reasoning models). Must be sorted ascending | | `ebp_cooloff_secs` | u64 | `30` | EBP congestion control cooloff period in seconds. After a multiplicative decrease (halving the probe ceiling), the controller enters a cooloff period where additive increase is suppressed. Increase for more conservative recovery; decrease for faster recovery on stable upstreams | | `sidecar_ready_timeout_secs` | u64 | `10` | Maximum time to wait for plugin sidecar gRPC servers to become ready during startup. Each sidecar is probed with exponential backoff (100ms initial, 2s max). Increase for slow-starting plugins | | `memory_soft_limit_mb` | usize | `0` | Memory cap in MB (ADR-005) — the maximum intended memory usage and the Critical hard-reject line. Watermark levels are 70/80/90/100% of this cap (Warning/Pressure/Reclaim/Critical); request gating (spill to disk + queue) activates at Warning (70%), and at the cap (100%) requests are hard-rejected with 429. `0` = auto-detect: cgroup v2/v1 → ECS Container Metadata API (Fargate) → `/proc/meminfo` (Linux bare metal/VM) → `sysctl hw.memsize` (macOS), using `limit - headroom` (headroom = clamp(15% × limit, 128MB, 512MB)). Set manually to override auto-detection | | `memory_queue_timeout_secs` | u64 | `300` | Maximum time (seconds) a request will wait in the memory-pressure queue before being rejected with 429 (ADR-005). The 300s default matches the queuing semantics — the user-perceived queueing time should be ≥ the gateway queue timeout, so an overly short timeout would defeat the purpose of queueing | | `memory_queue_max_depth` | u32 | `0` | Cap on requests resident in the slow-path permit-wait queue at once. Beyond it, new requests are rejected immediately with 429 + `Retry-After` (reason `queue_full`), without waiting for `memory_queue_timeout_secs`. Bounding the backlog lets a burst drain in finite time, avoiding 429 storms and 499 client disconnects. `0` = disabled (legacy: queue until timeout). Size from the tolerable backlog and the drain rate; a good starting point is ~`handler_permits_ceiling` (one round of in-flight capacity queued). Sustained overload is handled by scaling ECS on `protoflux_admission_reject_reasons_total{reason="queue_full"}` (ADR-005) | | `spill_write_concurrency` | usize | `4` | **Producer**: concurrent spill file write slots (spawn_blocking), each ~2.3MB. Not hot-reloadable (ADR-005) | | `memory_consume_max_concurrent` | usize | `100` | **Consumer**: drain rate at Warning/Pressure/Reclaim (concurrent bodies read back from spill files). Must be > 0; validated at startup. Capped by shared budget (ADR-005) | | `memory_consume_critical_max_concurrent` | usize | `2` | **Consumer**: drain rate at Critical level. Only a few bodies can be read back when memory is exhausted. Capped by shared budget (ADR-005) | | `memory_admission_enabled` | bool | `true` | Enable feed-forward admission (ADR-005): per-request decision BEFORE the body resides in memory, using real-time RSS + in-flight admitted body budget + this request's Content-Length — over the admit line, spill to queue so the body never enters memory, closing the burst blind spot between watchdog ticks where Normal was zero-overhead passthrough. The feed-forward layer only admits or spills; 429 remains the slow path's last resort (queue full / spill quota / timeout). `false` = legacy feedback-only (Normal = passthrough) | | `memory_admission_threshold_pct` | u8 | `70` | Feed-forward admit line = cap × pct / 100. Spill-to-queue when `rss + content_length > admit line`. Default 70% aligns with the Warning watermark so feed-forward and feedback watchdog trip at the same RSS. Range 1..=100 (validated at startup); lower = more conservative (spill sooner) | | `memory_admission_fresh_read_pct` | u8 | `70` | Threshold to re-read RSS synchronously when close to the admit line. The `last_memory_usage_bytes` snapshot is at most ~1s stale (one watchdog tick); when `rss > pct×cap` or the projection already exceeds 60%cap, the decision path calls `read_usage()` to refresh RSS before projecting, so a rising RSS that crossed mid-tick is not under-counted. Range 1..=100 | | `memory_force_spill_body_mb` | usize | `0` | Force-spill threshold in MB (`0` = disabled). Bodies above this size are spilled to queue regardless of the RSS projection, bounding the per-request memory cost of the one-shot `consume` read-back. When non-zero, must be ≤ `max_request_size_mb` (validated) | | `memory_admission_handler_permit` | usize | `30` | Worst-case memory footprint per handler (MB, default 30 = 30MB). Used to derive the **adaptive** permit bounds: `handler_permits_initial = soft_budget / (handler_permit × 1MB)` (conservative starting point) and `handler_permits_ceiling = cgroup_budget / (handler_permit × 1MB)` (AIMD increase ceiling, scaling with the machine; `usize::MAX` when no cgroup). AIMD climbs the target from initial toward ceiling — additive increase while RSS is below the low watermark, multiplicative decrease above the high watermark, settling at the observed runtime operating point. Covers request body + response buffer + reasoning content + serde deserialization overhead. Tune from observed peak RSS / active handlers. Must be > 0 (validated at startup). Setting `memory_admission_handler_permits > 0` explicitly pins initial = ceiling and disables the adaptive climb | | `health_probe_enabled` | bool | `true` | Enable TCP health probing for upstream hosts (ADR-012). When enabled, a background task periodically probes all registered upstream hosts via TCP connect, enabling automatic pool recovery without penalizing healthy upstreams | | `health_probe_interval_secs` | u64 | `10` | TCP health probe interval in seconds (ADR-012). Shorter intervals detect network issues faster but generate more probe traffic | | `health_probe_timeout_secs` | u64 | `3` | TCP health probe connect timeout in seconds (ADR-012). Should be shorter than the probe interval to avoid probe overlap | | `dns_refresh_interval_secs` | u64 | `60` | DNS re-resolution interval for health probe targets in seconds (ADR-012). Handles CDN failover, IP changes, and DNS TTL expiration | #### Memory Admission Tuning Scenarios The memory admission system's core parameters are interrelated and should be tuned based on actual workload characteristics. Below are common scenarios and recommended configurations: **Scenario 1: High-concurrency small requests (chat API, short prompts)** Characteristics: request body < 10KB, response body < 50KB, 100+ concurrent requests ```toml memory_admission_handler_permit = 10 # Each handler only needs 10MB memory_admission_threshold_pct = 75 # Allow higher RSS utilization memory_force_spill_body_mb = 0 # Small requests don't need forced spill ``` Effect: With `budget=300MB`, `handler_permits_initial=30`, supporting higher concurrency. **Scenario 2: Large context requests (document analysis, code review)** Characteristics: request body 100KB-2MB, response body 50-200KB, 10-30 concurrent ```toml memory_admission_handler_permit = 80 # Reasoning content can reach 50MB+ memory_admission_threshold_pct = 65 # More conservative, spill earlier memory_force_spill_body_mb = 5 # Force-spill very large requests ``` Effect: With `budget=300MB`, `handler_permits_initial=3`, preventing OOM. **Scenario 3: Multimodal requests (image + text)** Characteristics: request body contains base64 images (1-10MB), response body < 100KB ```toml memory_admission_handler_permit = 50 # Image decoding + reasoning buffer memory_force_spill_body_mb = 8 # Force-spill large image requests max_request_size_mb = 20 # Allow larger request bodies ``` Effect: Image requests auto-spill, preventing single requests from filling memory. **Scenario 4: CPU-constrained environments (0.25-0.5 vCPU)** Characteristics: Low container CPU quota, slow processing ```toml worker_threads = 1 # Match CPU cores memory_admission_handler_permit = 30 # Default value memory_queue_timeout_secs = 600 # Extend queue timeout (slow CPU; default 300 is not enough) ``` Effect: Prevent handlers from holding permits too long, queued requests don't timeout. **Scenario 5: Low memory utilization but frequent queuing** Symptoms: RSS < 60% cap, but logs show `handler_full` or `spill` Diagnosis: ```bash grep "handler_permits\|handler_full\|AIMD" protoflux.log | tail -20 ``` Solution: Reduce `memory_admission_handler_permit` (e.g., 30→20→15), which increases `handler_permits_initial` / `handler_permits_ceiling` (both ∝ budget / handler_permit). If the target stays pinned low and does not climb, check for an AIMD deadlock (the `initial < ceiling` invariant; see ADR-005). **Parameter Relationship** ``` cap (memory_soft_limit_mb) ├─ baseline (60MB) ├─ spill_overhead (20MB) ├─ reserved (50MB + cap/16) └─ budget = cap - baseline - spill - reserved ├─ handler_permits_initial = soft_budget / handler_permit (conservative start) └─ handler_permits_ceiling = cgroup_budget / handler_permit (AIMD increase ceiling) └─ AIMD target adapts within [initial, ceiling]; excess active requests spill to queue admit line = cap × threshold_pct% ├─ RSS + Content-Length < admit line → Direct admission └─ RSS + Content-Length > admit line → Spill to disk, wait for handler permit ``` **Tuning Workflow** 1. **Observe baseline**: Run with default config, monitor RSS and handler concurrency 2. **Identify bottleneck**: - RSS approaching cap → Increase `threshold_pct` or increase `cap` - Frequent spill but low RSS → Reduce `handler_permit` - Queue timeouts → Increase `queue_timeout` or add CPU 3. **Stress test**: Use `scripts/stress_test.sh` to simulate real load 4. **Iterate**: Change one parameter at a time, observe effects Validate changes with: ```bash cargo run -- --config server.example.toml config-validate ``` ### Streaming Configuration The `[streaming]` section controls SSE streaming connection behavior: | Field | Type | Default | Description | |-------|------|---------|-------------| | `keepalive_seconds` | u64 | `15` | SSE keep-alive heartbeat interval in seconds. The gateway sends `:keep-alive` comments at this interval to prevent proxy/browser timeouts. Should be shorter than your reverse proxy's idle timeout (e.g., Nginx default is 60s, so 15s is safe) | | `bootstrap_retries` | u32 | `1` | Number of retries before the first byte. If the upstream connection fails before the first SSE chunk is sent, the gateway retries this many times. No retries are possible once the first chunk reaches the client | | `stream_log_timeout_secs` | u64 | `3600` | Timeout for the SSE stream log background task in seconds. After a stream ends, a background task waits for upstream stream completion to record final stats. If the stream terminates abnormally (client disconnect + sender lost), this timeout releases the zombie task. 3600s (1 hour) far exceeds any legitimate LLM streaming response and only covers pathological cases | | `stream_idle_timeout_secs` | u64 | `300` | Maximum time (seconds) the upstream can go without sending any data before the gateway closes the stream with an idle timeout error. Covers both the first-byte wait and mid-stream pauses (not just the first token) — reasoning models that pause for long thinking may trigger it, so increase as needed. Prevents zombie connections where the upstream silently stops sending. Set to 0 to disable. Must be less than `upstream_read_timeout_secs` so the gateway-level timeout fires first. Env: `STREAM_IDLE_TIMEOUT_SECS` | | `max_stream_duration_secs` | u64 | `3600` | Maximum total duration (seconds) for a single streaming response. After this duration, the gateway closes the stream regardless of upstream activity. Prevents infinitely-long streams from consuming resources. Set to 0 to disable | | `max_concurrent_streams` | u32? | `200` | Maximum concurrent SSE streams across all clients. Global semaphore limit for streaming responses. When the limit is reached, new stream requests receive HTTP 503. Each SSE stream spawns a Tokio task and holds an upstream connection. Set to `None` or `0` to disable | ### Logging Configuration The `[logging]` section controls structured logging and observability: | Field | Type | Default | Description | |-------|------|---------|-------------| | `level` | string | `"info"` | Log level: `trace`, `debug`, `info`, `warn`, `error`. Controls verbosity of structured logs | | `format` | string | `"json"` | Log format: `json` (machine-parseable), `compact` (single-line human-readable), `pretty` (multi-line with colors) | | `max_body_size_mb` | f64 | `25` | Maximum megabytes per body capture across all logging — terminal + SSE/webconsole. Bodies exceeding this are truncated | | `log_body_to_terminal` | bool | `false` | Whether to log request/response bodies to terminal output. When false, bodies are only captured in SSE/webconsole logs | | `otel_endpoint` | string? | none | OpenTelemetry OTLP gRPC endpoint for distributed tracing export (requires `--features otel` build). Example: `"http://jaeger:4317"` | | `otel_service_name` | string | `"protoflux"` | Service name reported to OpenTelemetry collector. Used to identify this service in traces and spans | | `persist_request_logs` | bool | `true` | Whether to persist request logs to the database. When false, log events still flow through FileLogQueue for SSE broadcast but are not written to DB | | `log_retention_days` | u32 | `7` | Number of days to retain request logs in the database. PostgreSQL uses DROP PARTITION (DDL, instant), SQLite uses DELETE. 0 disables automatic retention | | `stream_body_max_disk_mb` | u64 | `256` | Maximum disk space (MB) for SSE streaming response body disk spill. When exceeded, new chunks are dropped and `body_incomplete` is flagged | | `stream_body_batch_size_kb` | usize | `64` | Batch size (KB) for stream body writer before flushing to disk. Larger values reduce I/O operations but increase memory usage per stream | | `stream_body_linger_ms` | u64 | `5` | Maximum wait time (ms) before flushing incomplete batches. Shorter values reduce latency but increase I/O frequency | | `stream_body_channel_capacity_chunks` | usize | `2048` | Bounded mpsc channel capacity for stream body chunks (**chunk count, not bytes**). When full, new chunks are dropped (non-blocking). Default 2048 chunks ≈ 2 MB buffer (estimated at 1 KB/chunk) | **OpenTelemetry**: When `otel_endpoint` is configured and the binary is built with `--features otel`, Protoflux exports traces via OTLP gRPC protocol. This enables distributed tracing across microservices. The gateway automatically propagates W3C Trace Context (`traceparent` header) to upstream providers, allowing end-to-end request tracking. ### Pricing Configuration The `[pricing]` section controls price resolution and price-snapshot versioning for model billing. Prices are **append-only snapshots**; each carries a `created_at` timestamp, and cost is computed on the fly against each per-minute bucket's timestamp (see the Pricing & Billing section of the architecture doc). Prices are resolved through a **two-tier** field-level overlay (low → high): Tier1 `meta.toml` baseline → Tier2 remote pricing service (configured here); field-level semantics — `Some` overrides, `None` preserves the lower tier, `Some(0)` is an explicit free price, and `extra_pricing` is a key-level map-merge. `PriceResolver` writes the resolved currently-effective price as an appended snapshot into `config_entity_version` (`entity_type='model_pricing'`, `created_at` taken from DB server time, `source` recording the winning tier `tier1:meta.toml` / `tier2:remote`). The unit is micro-USD per 1M tokens (i64); the console human unit is USD/1M tokens (×1e6/÷1e6 conversion at the console boundary). Console manual prices are a **bypass pin**, not a resolver tier: when an admin creates or edits a model's price snapshot via console, the row carries a fixed `source="console:manual"`, and the resolver skips that model on seeing `source.starts_with("console:")` (no diff, no append) — the manual price is authoritative and is not overwritten by remote or baseline; only after the pin row is deleted does the resolver re-append per Tier1/Tier2. | Field | Type | Default | Description | |-------|------|---------|-------------| | `remote_source` | table? | none | Remote pricing service config. When unset, Tier2 remote fetching is disabled and prices come only from the Tier1 meta.toml baseline (not overwritten while a console pin is in effect); the price refresh timer task still spawns but skips fetch/refresh, keeping the managed task count stable (Memory=R7 / Redis=R14) | | `retention_days` | u64? | none | Price-snapshot history retention in days. When unset (default) no automatic cleanup; when set to N, the housekeeping periodic task rolling-deletes non-latest snapshot rows with `created_at < now - N*86400` (**never the latest currently-effective row, never the 6 config-type rows**, batch-throttled to avoid long transactions). Deleting old snapshots causes that period's latest-before to fall back to an earlier snapshot, recomputing historical cost; enable only when remote pricing is expected to fluctuate frequently | #### Remote Pricing Service `[pricing.remote_source]` configures the Tier2 remote pricing service fetch. A synchronous fetch runs once at startup (failure falls back to Tier1 without blocking startup), followed by periodic refresh. Remote responses use lenient deserialization: items are parsed one by one, a single bad item (missing `model` / illegal field type) is skipped with a warn log and never fails the whole batch; unknown numeric dimensions land in the `extra_pricing` extension map, non-numeric fields are ignored. | Field | Type | Default | Description | |-------|------|---------|-------------| | `url` | string | — | Remote pricing service endpoint. GET request; the response body is a JSON array, each item containing `model` (required) + `input_price`/`output_price`/`cache_read_price` (micro-USD per 1M tokens, numeric) + optional `extra_pricing` nested map + optional `upstream` | | `auth_header` | string? | none | Request authorization header value (e.g. `"Bearer "`). When set, an `Authorization` header is sent with the request | | `refresh_interval_secs` | u64 | `300` | Periodic fetch interval (seconds). Each tick first fetches remote prices to update the in-memory Tier2 cache, then triggers `refresh_and_persist` to resolve the two tiers and decide whether to append a new snapshot based on the diff | | `timeout_secs` | u64 | `10` | Single HTTP request timeout (seconds) | | `min_rewrite_interval_secs` | u64 | `21600` | Tier2 write-coalescing window (seconds, default 6 hours). Applies only to Timer-triggered writes: when the current row's `source` originates from tier2 and `now - current_row.created_at < this window`, the append is skipped (the old current row is preserved); after the window expires the next refresh appends one new snapshot with the latest remote price | **Tier2 write coalescing**: Under the coalescing window (default 6 hours), even if Tier2 remote prices fluctuate every minute, each model_key appends at most 4 snapshot rows per day (one append per window expiry with the latest price), keeping write growth predictable and preventing `config_entity_version` table bloat. **Bootstrap first fetch (no current row) and ConfigReload (console manual price change / pin) are not subject to coalescing** — candidates execute immediately. When the current row's `source` is not tier2 (the last write was a console manual price change, i.e. a pin), a Timer trigger appends even if the diff has changes (the remote rebound after a manual change should be recorded). The refresh timer task wraps each tick in `catch_unwind`; on panic/error it logs a warn and waits for the next tick, never terminating the task. #### Price Intervals and Snapshot Time Snapshots are **price data points**, not state transitions that "take effect at a moment": one snapshot covers `[its own created_at, the next newer snapshot's created_at)`, the earliest back-fills to `-∞`, the newest is open to `+∞`; with a single snapshot it is the sole price for all time. `created_at` is the snapshot timestamp, settable on console create/edit (back-fill a past price, preset a future price, correct a time) — **not an "effective from" concept**: the interval is determined by the ordering of snapshots' `created_at`, not declared by a field. A future-dated (`created_at > now`) snapshot is dormant until it matures — it is not counted as the current price, and the cost join does not hit historical buckets. #### Configuration Examples `[pricing]` is a top-level infra section (peer of `[server]`/`[console]`, etc.): ```toml [pricing] # Remote pricing service (Tier2); when unset, prices come only from the meta.toml baseline (Tier1) [pricing.remote_source] url = "https://pricing.example.internal/v1/models" auth_header = "Bearer " refresh_interval_secs = 300 # fetch every 5 minutes timeout_secs = 10 min_rewrite_interval_secs = 21600 # Tier2 write-coalescing window, 6h # Optional: price-snapshot history retention (no cleanup by default) # retention_days = 365 ``` Tier1 baseline prices live in the `[[models]]` entries of `crates/core/src/metadata/meta.toml` (compiled in, overridable via the Console API). All price fields are `Option`; field-level overlay — fill only the dimensions you want to override, the rest preserve the lower tier: ```toml [[models]] patterns = ["gpt-5*"] priority = 10 # …(other metadata fields)… # Prices (micro-USD per 1M tokens; console human unit is USD/1M tokens, ×1e6 conversion) input_price = 5000000 # $5/1M input tokens output_price = 15000000 # $15/1M output tokens cache_read_price = 500000 # $0.5/1M cache-read tokens # Extension dimensions (key-level map-merge); declare cache_creation separately when it differs from input [models.extra_pricing] cache_creation = 6250000 # $6.25/1M cache-creation tokens # Per-call billing (mutually exclusive with token prices): per_call mode charges per_call_price × request count # billing_mode = "per_call" # per_call_price = 10000 # $0.01/request (micro-USD) # Original-currency tag (default "USD", no conversion) # currency = "USD" ``` The console human unit is USD/1M tokens (entering `5` = $5/1M); the backend stores micro-USD (`5_000_000`); the console boundary converts ×1e6/÷1e6 automatically. The remote pricing service (Tier2) JSON response also carries prices as micro-USD per 1M tokens (numeric). ### Disk Configuration Overview Protoflux uses multiple **independent shared append logs** (`SegmentedStore`, sequential write + cursor-driven segment reclaim) for disk buffering. Each store has its own directory + independent cap, so they do not compete. Core OOM defense: request/response bodies never reside in memory — they spill to disk and only a 24-byte `BodyRef` pointer is held, read back via mmap zero-copy at consume time. The table below summarizes all disk-related config items (caps + writer tuning) for at-a-glance review; each field is also documented in its owning config section. | Field | Env var | Default | Dockerfile | What it stores / what happens when full | |------|---------|---------|------------|------------------------------------------| | `log_queue_max_disk_size_mb` | `LOG_QUEUE_MAX_DISK_SIZE_MB` | `1024` | `1024` | LogEvent NDJSON queue total disk cap. When full → new events are dropped (one log entry lost, not a body) | | `stream_body_max_disk_mb` | `LOG_STREAM_BODY_MAX_DISK_MB` | `256` | `1024` | SSE streaming response body segment total cap. When full → new chunks dropped + `body_incomplete` flagged | | `request_body_max_disk_mb` | `LOG_REQUEST_BODY_MAX_DISK_MB` | `256` | `1024` | Redacted request body spill (held full-stream + drain). When full → spill fails, body stays inline in memory (OOM risk) | | `accumulator_max_disk_mb` | `LOG_ACCUMULATOR_MAX_DISK_MB` | `128` | `256` | Stream accumulator (reasoning / text / tool_call aggregation) overflow. When full → degrades to inline (per-stream resident) | | `stream_body_batch_size_kb` | `LOG_STREAM_BODY_BATCH_SIZE_KB` | `64` | `64` | Writer batch size (**not a cap**, tuning param). Flushed when full; larger = fewer I/O ops, more per-stream memory | | `stream_body_channel_capacity_chunks` | `LOG_STREAM_BODY_CHANNEL_CAPACITY_CHUNKS` | `2048` | `2048` | Stream body mpsc channel capacity (**chunk count, not bytes**, not a cap). When full, chunks are dropped + `body_incomplete` flagged; default 2048 ≈ 2 MB buffer | **Tuning guidance**: - In production `StoreFull` alerts, first check the `role` field in logs (`request body` / `response body` / accumulator / log queue) to locate which store is full, then raise its cap or speed up drain (drain lag pins disk). - Caps are **tourniquet buffers** (provide headroom, delay saturation), **not real fixes** — a `completion_tx` that exits non-EOS pins extents for 3600s; no matter how large the cap, it eventually saturates. The real fix is the `CompletionOnDrop` gate (see `body-store-completion-gate-and-cap-tuning`). - `log_queue_segment_size_mb` (default 256) is the per-segment rollover size (not a cap); paired with `log_queue_max_disk_size_mb`: segments roll when full, total over cap triggers cleanup. - Redaction contract: `request_body_store` / `response_body_store` / accumulator store only redacted bodies (28 credential classes redacted before spill); the ingress request body is spilled raw to a per-request temp file (brief ephemeral, RAII `Drop`-deleted at request end, independent dir, **no aggregate disk cap** — transient footprint bounded by in-flight requests × `max_request_size_mb`). ### Observability Metric Unit Layering Protoflux's disk/memory observability follows industrial **unit layering** (separation of concerns, not a single rule): | Layer | Suffix | Reader | Conversion point | Rationale | |-------|--------|--------|------------------|-----------| | Collection (Prometheus metric) | `_bytes` | Machine (scrape) | none (base unit) | Prometheus "Use base units" convention; `rate()`/`increase()` math is correct in bytes; Grafana `unit: bytes` auto-converts to human-readable; community dashboards (node_exporter etc.) are compatible | | API (/stats JSON + `InstanceMemorySummary` + `LogQueueDiagnostics`) | `_mb` | Human (ops) | `bytes / 1024 / 1024` at backend output | Frontend reads directly, no mental math; Redis-stored JSON uses this layer too | | Code (internal struct fields + cross-fn params) | `bytes` | Engineer (source) | none (precision) | Cross-fn bytes avoid repeated conversion error; conversion only at boundaries (config load / JSON output) | | Display (console UI) | `MB`/`GB` | Human (user) | `formatMB` at UI edge | `< 1024` → `N MB`, `≥ 1024` → `X.X GB` | | Channel capacity | `_chunks` | all layers | none | mpsc message count, not bytes (2048 chunks ≈ 2 MB at 1 KB/chunk) | Metric names and JSON field names are **decoupled but correlated**: metric `protoflux_log_queue_disk_used_bytes` (machine scrape) ↔ JSON `log_queue_disk_mb` (human-readable). The gauge name carries `_used` (`disk_used_bytes`) while the JSON field does not (`disk_mb`) — this is intentional semantic layering: the gauge emphasizes "used" to distinguish from cap gauges, while the JSON field doesn't repeat "used" in a `usage / cap` pairing context. Mapping table: | Prometheus gauge (`_bytes`) | /stats JSON field (`_mb`) | Description | |----|----|------| | `protoflux_process_rss_bytes` | `process_rss_mb` | Process RSS | | `protoflux_memory_limit_bytes` | `memory_limit_mb` | cgroup memory limit | | `protoflux_log_queue_disk_used_bytes` | `log_queue_disk_mb` | Log queue disk usage | | `protoflux_response_body_store_disk_used_bytes` | `response_body_store_disk_mb` | Response body store usage | | `protoflux_request_body_store_disk_used_bytes` | `request_body_store_disk_mb` | Request body store usage | | `protoflux_accumulator_store_disk_used_bytes` | `accumulator_store_disk_mb` | Accumulator store usage | | `protoflux_ingress_spill_disk_used_bytes` | `ingress_spill_disk_mb` | Temp body usage (per-request temp file transient footprint) | In Redis multi-instance mode, `/stats` returns `instances[]` (one `InstanceMemorySummary` per instance, with the `*_mb` fields above + `body_store_config`); cluster-level aggregation uses `total_rss_mb` / `total_limit_mb`. ### Disk Observability Metrics The `memory` object in `/stats` JSON outputs these fields (Memory mode inlines them; Redis mode aggregates from `instances[]`): | Field | Unit | Description | |----|----|------| | `process_rss_mb` | MB | Process RSS (Prometheus gauge `process_rss_bytes / 1024 / 1024`) | | `memory_limit_mb` | MB | cgroup memory limit (`memory_limit_bytes / 1024 / 1024`) | | `log_queue_disk_mb` | MB | Log queue disk usage | | `response_body_store_disk_mb` | MB | Response body store disk usage | | `request_body_store_disk_mb` | MB | Request body store disk usage | | `accumulator_store_disk_mb` | MB | Accumulator store disk usage | | `ingress_spill_disk_mb` | MB | Temp body disk usage (per-request temp file, transient, no cap) | | `body_store_config` | — | Resolved caps for the three drain-persistent stores (see below) | | `log_queues[].write_pos_mb` | MB (offset) | Write offset position, **not disk usage** | | `log_queues[].disk_usage_mb` | MB | Current disk usage | | `log_queues[].max_disk_size_mb` | MB | Queue total disk cap | | `log_queues[].segment_size_mb` | MB | Per-segment rollover size (default 256 MB, configurable, not a cap) | | `log_queues[].segment_count` | count | Number of live segments (unitless) | | `log_queues[].retention_floor_mb` | MB (offset) | Reclaim floor (min cursor across DbDrain + non-stale SSE), **not usage** | | `log_queues[].retention_floor_no_stale_mb` | MB (offset) | Reclaim floor ignoring stale consumers (3× segment size behind = stale) | | `log_queues[].db_drain_cursor_mb` | MB (offset) | DB drain consumer cursor (null when no DbDrain) | | `log_queues[].sse_consumer_count` | count | Active SSE consumer count (unitless) | | `log_queues[].sse_min_cursor_mb` | MB (offset) | Min cursor across SSE consumers (null when no SSE) | **Offset fields**: `write_pos_mb` / `retention_floor_mb` / `retention_floor_no_stale_mb` / `db_drain_cursor_mb` / `sse_min_cursor_mb` are **position offsets** (u64 bytes / 1024 / 1024 truncated to 1 MB granularity), not disk usage — they locate "where written / where consumed" to drive segment reclaim, not bytes on disk. To judge disk saturation, read `disk_usage_mb / max_disk_size_mb`. `body_store_config` fields (resolved caps for the three drain-persistent stores; the console uses these for usage/cap ratios; the ingress temp body has no cap — see "Temp body store" below): | Field | Unit | Description | |----|----|------| | `stream_body_max_disk_mb` | MB | Response body store cap (`stream_body_max_disk_mb` resolved) | | `request_body_max_disk_mb` | MB | Request body store cap | | `accumulator_max_disk_mb` | MB | Accumulator store cap | | `body_segment_size_mb` | MB | Segment rollover size shared by the three body stores (64 MB hardcoded, not configurable) | | `stream_body_channel_capacity_chunks` | chunks | Stream body mpsc capacity (message count, not bytes) | ### Frontend Memory Metrics Explained Each metric shown in the OverviewPage instance card + memory hover popover, with how to read it and what anomalies mean. Field definitions are in the table above; this section walks them in hover layout order. **Instance card (visible without hover)**: - **Watermark label** (`watermark_level`): Normal → Warning → Pressure → Reclaim → Critical, classified every 1s tick by watchdog from RSS as a percentage of `memory_limit` (70/80/90/100% thresholds). Normal = healthy; ≥ Pressure (orange) = RSS nearing cap, AIMD starts pulling handler/stream permits; Critical (red + ⚠) = escape hatch rejects new requests to keep the process alive. Sustained ≥ Pressure → investigate leak or raise cgroup / `memory_soft_limit_mb`. - **RSS / cap** (`process_rss_mb / memory_limit_mb`): process resident memory / cgroup limit. RSS rising without falling = leak (check moka capacity / accumulator inline degradation); RSS near cap = OOM risk, check watermark. In Redis mode the top-level `total_rss_mb / total_limit_mb` is the sum across all instances. - **Disk summary** (`diskUsage`): sum of five disk footprints (log_queue + response + request + accumulator + temp body); hover to break down which store is full. **hover popover — log_queues detail** (one block per queue; Memory mode has 1 queue, Redis mode has drain + live dual queues): - **Usage / cap** (`disk_usage_mb / max_disk_size_mb`): this is what to check for disk-full. Usage near cap = consumers behind, StoreFull about to drop logs / 429. - **Write offset** (`write_pos_mb`): how far the append-log has been written (a position offset, **not usage**). `write_pos − retention_floor` = backlog awaiting reclaim. - **Segment count × size** (`segment_count × segment_size_mb`): live segments × per-segment rollover size. Segment count growing = consumers behind, segments rolling without reclaim. - **Retention floor** (`retention_floor_mb`): the slowest consumer's position (including stale). Segments below it are deletable. `floor ≪ write_pos` = consumers behind, disk accumulating. - **Retention floor ignoring stale** (`retention_floor_no_stale_mb`): floor ignoring stale SSE consumers (>3 segments behind). **A large gap vs `retention_floor_mb` = stale SSE clients pinning disk** (disconnected but registry not cleaned); hover highlights this. Actual cleanup uses this value — stale consumers don't block reclaim. - **DB drain cursor** (`db_drain_cursor_mb`): how far DbDrain has persisted. `null` = no DbDrain (Memory mode without DB persistence). Cursor stalled = drain stuck, logs land on disk but never enter the DB. - **SSE consumer count + min cursor** (`sse_consumer_count` / `sse_min_cursor_mb`): active console live-log connections + the slowest one's cursor. `count > 0` but `sse_min_cursor ≪ write_pos` = console clients can't keep up; `count = 0` = no live subscribers. **hover popover — three drain-persistent body stores** (each shows usage / cap, for the occupancy ratio): - **Response body store** (`response_body_store_disk_mb / stream_body_max_disk_mb`): SSE streaming response bodies spilled to disk. Full = stream body chunks dropped + flagged `body_incomplete` (response body truncated). Typical cause: client disconnect but `completion_tx` exits non-EOS, pinning extents for 3600s (see Disk Store Release Mechanism). - **Request body store** (`request_body_store_disk_mb / request_body_max_disk_mb`): redacted request bodies spilled (for audit/replay). Full = spill fails, body stays inline in memory (OOM risk). Typical cause: drain lag (DB persistence slow). - **Accumulator store** (`accumulator_store_disk_mb / accumulator_max_disk_mb`): streaming reasoning / text / tool_call aggregation overflow. Full = degrades to inline (per-stream resident, OOM risk). Typical cause: long thinking streams + high concurrency. **hover popover — Temp body store** (a standalone block, showing only transient usage, **no cap**): - **Temp body store** (`ingress_spill_disk_mb`): ingress request bodies stream to per-request temp files (raw unredacted; see "Temp body store architecture" below). Transient footprint is bounded by in-flight requests × `max_request_size_mb`; each file is deleted at request end via RAII `Drop`. **No aggregate cap, no 429 backpressure** — this is the result of the 5→1 refactor eliminating the shared quota counter: the old model reserved quota by Content-Length at admission and held it across `admit → upload → TTFB → first byte`, and the TTFB (model think) window overlap saturated the 240 MB transient cap, causing a 429 storm; under the new model each body's bytes live in its own file and are deleted on Drop, with no shared counter and no hold window. **Diagnostic cheat sheet**: disk-full → `disk_usage / max`; backlog → `write_pos − retention_floor`; stale pinning → gap between `retention_floor` and `retention_floor_no_stale`; drain stuck → `db_drain_cursor` stalled; SSE slow → `sse_min_cursor ≪ write_pos`; OOM risk → watermark ≥ Pressure + RSS near cap. ### Disk Store Release Mechanism The four disk stores each have an independent directory + cap + release gate (the ingress temp body is a per-request file with no cap, deleted via RAII; see "Temp body store architecture" below). Core OOM defense: bodies never reside in memory — they spill to disk and only a 24-byte `BodyRef` pointer is held, read back via mmap zero-copy at consume time; segment reclaim is cursor-driven (consumers keeping up → reclaim to active segment; consumers behind → keep back to cursor; stale consumers 3× segment size behind are ignored to avoid pinning). | Store | Directory | Cap config | Release gate | |----|----|----|----| | `response_body_store` | `body_segments/` | `stream_body_max_disk_mb` | `StreamEndOnDrop` (flush writer) + aggregator abandoned-receiver release (publish 10s timeout fallback) + orphan sweep + pre-publish resolve-release | | `request_body_store` | `request_bodies/` | `request_body_max_disk_mb` | drain-cursor reclaim (after DB persistence) | | `accumulator_store` | `accumulator/` | `accumulator_max_disk_mb` | `SpillingAccumulator::Drop` + `.done()` read-back | | `log_queue` | `log_queue_dir` | `log_queue_max_disk_size_mb` | retention_floor reclaim (drain + live dual queue, cursor-driven) | In Redis multi-instance mode, `log_queue` splits into drain + live dual queues: drain queue (local consume → DB persistence) + live queue (all instances subscribe → SSE broadcast). `compute_cleanup_min_consumed` reclaims to `retention_floor_ignoring_stale`; with no consumers, floor = `u64::MAX` and all history segments are reclaimed (only the active segment remains). **Temp body store architecture** (per-request temp file, no cap): every admitted request body is streamed to **its own** temp file (body frame → mpsc capacity=4 → `spawn_blocking` → `TempBodyWriter` sequential write to a flat file), so peak memory = a single frame (~8-64 KB), decoupled from body size; the handler holds an `IngressBodyRef` (`Arc`, owning an `Arc` + size). Read-back takes two paths: small bodies via `mmap_body` zero-copy page mapping (page-cache reclaimable), large bodies (image base64 and other large strings) via `reader()` 64 KiB pread sliding windows (bounded RSS, decoupled from body size; see the architecture section "Streaming request-body parse and send (hybrid BlobRef)"). Per-flow residency drops from body size to an algorithmic constant (100 concurrent × 1.3 MB burst = 130 MB → ~2.4 KB). **No aggregate disk cap** — this is the core of the 5→1 refactor eliminating the 429 storm. The old model (shared append-log + CAS quota counter) reserved quota by Content-Length at admission and held it across `admit → upload → TTFB → first byte`, and the TTFB (model think time) window overlap saturated the ~240 MB transient cap → admission `StoreFull` → 429 storm. Under the new model each body's bytes live in **its own file**, with no shared counter and no hold window: the single-body upper bound is just `max_request_size_mb` (Axum `DefaultBodyLimit` enforced outside the store; exceeding it mid-stream raises `LengthLimitError` → 413), transient footprint is bounded by in-flight requests × `max_request_size_mb`, and each file is deleted at request end via RAII `Drop`. There is no admission 429 (a disk-full becomes a fatal IO error → 500, rare and acceptable). `IngressBodyRef` is an `Arc` refcount with 2 clones: `request.extensions()` (handler read) and `response.extensions()` (access_log post-phase reads the already-consumed body to log it); the last clone dropping triggers `Arc`'s `Drop` to delete the file, so `access_log` can still read the body after `next.run` consumes the request. **Cancel safety relies entirely on RAII `Drop`** — no `CancelGuard` / outstanding-set / shared-records mirror / `release_*` machinery: (1) when the `spill_ingress` future is dropped mid-spill (client disconnect, hyper drops the future), `tx` drops → the writer's `blocking_recv` returns `None` → break, and `TempBodyWriter`'s `Drop` deletes the partial file (no shared state to roll back, no quota CAS to reverse); (2) normal `Drop` → the last `Arc` clone dropping deletes the file; (3) orphan sweep — when the owner of an already-built ref is cancelled/forgotten so `Drop` never runs, stale leaked files past a `max_stream_duration × 2` threshold are deleted (a legitimate in-flight stream has age < threshold and is never swept, SIGBUS-safe). Mid-stream rejection: body over `max_request_size_mb` → 413 (writer `Drop` deletes the partial file + `body_incomplete`). Difference from `request_body_store`: the temp body store stores the **raw unredacted** body (ephemeral — deleted via RAII at request end / the whole directory wiped on restart); request_body_store stores the **redacted** body (drain-persistent, for audit/replay). The metrics-only `current_usage` counter is solely for the frontend "Temp body store" row to display transient usage (reconciled by the orphan sweep to actual disk usage) and **gates no admission**. mmap'd page cache counts toward `memory.current` (the OOM killer's view), but temp files are unlinked on Drop, so the page cache is reclaimed immediately with the inode — read-back uses mmap (reclaimable) or pread (peak RSS ≈ 64 KiB, independent of body size), never a whole-object `read_all` of a large body. See ADR-005 "mmap-spill page-cache bounds". ### Web Console Configuration The `[web_console]` section controls the administrative console behavior: | Field | Type | Default | Description | |-------|------|---------|-------------| | `session_auto_renew` | bool | `true` | Enable sliding-window session auto-renewal. When `true`, each authenticated API call extends the session TTL back to 24 hours if less than 50% lifetime remains. When `false`, sessions have a fixed 24-hour lifetime regardless of activity | ## Key Sections - `access_keys`: structured client access control with per-access-key API keys and optional group membership - `access_key_groups`: model-level access control lists (ACLs) for access key groups - `upstreams`: named upstreams with protocol, base URL, and credentials - `models`: public model IDs mapped to upstream names via `upstream_model_id`, with optional per-model `base_url` override - `routing`: optional tuning for session-based upstream affinity (automatic by default) - `web_console`: control-plane auth and admin UI behavior - `cors`: browser access policy for the admin plane Each upstream entry is intentionally small. In practice you only need: - `name` — a label for this upstream (e.g. `openai`, `aliyun-anthropic`) - `protocol` — one of the supported protocol keys (see below) - `base_url` - `api_keys` — inline array: `api_keys: [{"key": "...", "weight": 1}]`. With multiple keys, selection is **deterministic weighted distribution** (the same caller sticks to the same key; `weight=0`=backup) — see [Single upstream multi-key selection](architecture.md#single-upstream-multi-key-selection) in the Architecture reference. - `headers` — custom HTTP headers attached to every upstream request (optional, JSON object) - `dns_servers` — per-upstream DNS servers (optional, array of IPs or URI-scheme-prefixed addresses; default: system resolver). Protocol and TLS SNI are auto-inferred from the URI scheme. See "Per-Upstream DNS Resolver" below. - `request_payload` — request body modifications applied before sending upstream (optional, inline array) - `enabled` — whether this upstream is active (default: `true`). Disabled upstreams are excluded from routing and their credentials are not loaded. ### Upstream Custom Headers You can attach custom headers to all requests sent through an upstream using the `headers` field: Console → Upstreams → New upstream `openai` → Protocol `openai` / base_url `https://api.openai.com/v1` / headers `{"X-Custom-Header":"my-value","X-Organization":"my-org"}`. Custom headers are merged into the upstream request after protocol-specific headers (auth, content-type, etc.), so they can override defaults if needed. This is useful for routing through proxies that require specific headers, canary deployments, or passing operational metadata. ### Per-Upstream User-Agent Each upstream can override the global `server.upstream_user_agent` setting with its own `user_agent` field. When set, requests to this upstream use the per-upstream value instead of the global default. Console → Upstreams → New upstream `custom-proxy` → Protocol `openai` / base_url `https://proxy.example.com/v1` / API Keys `sk-...` (weight `1`) / user_agent `MyGateway/2.0`. When `user_agent` is not set (or omitted), the global `upstream_user_agent` (default: `"Protoflux"`) is used. This is useful when routing through upstream proxies or vendors that require a specific User-Agent for authentication or routing. ### Per-Upstream DNS Resolver Each upstream can configure its own DNS server list (IPv4 or IPv6). The gateway resolves the `base_url` host **directly** against these servers via an embedded hickory-resolver, **completely bypassing container/OS DNS caches** (nscd, systemd-resolved, Docker 127.0.0.11, VPC DNS, CoreDNS, etc.). DNS servers accept both plain IP addresses and URI-scheme-prefixed addresses: | Scheme | Protocol | Default Port | |--------|----------|--------------| | (none) | UDP (standard DNS) | 53 | | `udp://` | Standard DNS (UDP + TCP fallback) | 53 | | `tls://` | DNS-over-TLS (DoT) | 853 | | `https://` | DNS-over-HTTPS (DoH) | 443 | URI schemes allow per-server protocol overrides within a single resolver. For plain IPs without a scheme prefix, UDP is assumed. TLS SNI hostnames required by `tls` and `https` protocols are auto-detected for well-known providers (Cloudflare, Google, Alibaba, Quad9); for hostnames in the URI, the hostname itself is used as SNI. - **AliDNS public (UDP)**: Console → Upstreams → New upstream `aliyun-china` → Protocol `anthropic` / base_url `https://dashscope.aliyuncs.com/apps/anthropic` / API Keys `sk-...` (weight `1`) / DNS Servers `223.5.5.5, 223.6.6.6`. - **DNS-over-TLS (DoT)**: Console → Upstreams → New upstream `cloudflare-dot` → Protocol `openai` / base_url `https://api.openai.com` / API Keys `sk-...` (weight `1`) / DNS Servers `tls://1.1.1.1, tls://1.0.0.1` (dns_server_name auto-detected as `1dot1dot1dot1.cloudflare-dns.com`). - **DNS-over-HTTPS (DoH)**: Console → Upstreams → New upstream `google-doh` → Protocol `openai` / base_url `https://api.openai.com` / API Keys `sk-...` (weight `1`) / DNS Servers `https://8.8.8.8` (dns_server_name auto-detected as `dns.google`). - **No DNS**: Console → Upstreams → New upstream `openai` → Protocol `openai` / base_url `https://api.openai.com` / API Keys `sk-...` (weight `1`) (no DNS Servers → system default resolver). **Root cause it fixes:** container/OS DNS caches a stale IP from the upstream ALB; after ALB IP rotation the stale IP no longer works but the cache hasn't expired, so the gateway's zombie-pool flush self-healing chain rebuilds connections that resolve back to the same bad IP → sustained 503/504/timeout with no recovery. Per-upstream DNS gives each upstream **fresh** IPs, restoring self-healing. **Runtime mechanics:** - DNS server list persists in the DB `extra_config.dns_servers` extension field (no DB column additions). Protocol and server name are auto-inferred from the first URI on each load — no separate storage needed. Console UI edits land in the same field. - Upstreams sharing the same inferred `(dns_servers, dns_ttl_secs, protocol, server_name)` 4-tuple reuse a single resolver instance (`resolver_cache`) — avoids duplicate hickory construction. Different protocol or server name yields separate resolver instances. - Gateway-managed TTL-capped DashMap cache (configurable via `dns_ttl_secs`, default 30s); hickory's own `DnsCachingClient` is **disabled** via `cache_size = 0` to prevent double-caching (effective staleness would otherwise be `our_ttl + alb_dns_ttl`, worsening the root cause). - Singleflight per-host lookups: concurrent cache misses for the same host share a single upstream DNS query (per-host mutex), preventing thundering-herd duplicate queries. - Stale-while-revalidate-lite: on DNS query failure, a stale cached entry is served as fallback (with a warn log), preventing a brief upstream DNS outage from taking down requests. - HTTP clients are bucketed by the composite key `(proxy, dns_servers, dns_ttl_secs, protocol, server_name)` — upstreams sharing a proxy but differing DNS configs do not share a connection pool (which would cross-contaminate resolutions). - Health probing (`health_prober`) uses the same `dns_servers`, so zombie-flush decisions and rebuilt connections resolve to the same IP set. **Observability logs** (trace/debug): - `per-upstream DNS resolver initialized` (info, on construction) - `get_or_create_resolver: cache HIT/MISS` (trace/debug, resolver reuse) - `client_for_auth: resolved ClientKey` (trace, per-request composite key) - `dns_resolver: cache HIT (fresh) / MISS / fresh lookup succeeded` (trace/debug, host-level resolution with elapsed time and IP list) - `dns_resolver: lookup failed, serving STALE cache` (warn, stale fallback) **Fallback DNS pool for unconfigured upstreams:** an upstream without `dns_servers` defaults to reqwest's system `GaiResolver` (getaddrinfo), bypassing `GatewayDnsResolver` entirely — no SWR cache, SSRF guard, failure tracking, or fallback when the system resolver is slow/dead. When the infra config `fallback_dns_servers` (default `["1.1.1.1", "8.8.8.8", "119.29.29.29", "223.5.5.5"]`) is non-empty, such an upstream instead uses a merged-pool resolver whose server set = [system `/etc/resolv.conf` nameservers] + [this list]: a healthy local resolver wins by latency, and a slow/wedged system resolver is transparently backed by the public roots, while protoflux's full DNS machinery (SWR / SSRF / failover / failure counting) applies. Set `fallback_dns_servers = []` to disable (fall back to `GaiResolver`, a backward-compatible escape hatch). In Docker, override via the `FALLBACK_DNS_SERVERS` env var (comma-separated; the Dockerfile default carries these four roots, and empty disables). The merged pool does not read `/etc/hosts` and cannot resolve `localhost`/`.local` (system nameservers are queried via hickory UDP, not getaddrinfo) — for a local upstream, use the per-upstream escape `dns_servers = ["127.0.0.1"]` or disable globally. See the architecture doc "Upstream DNS Resolver". **Known constraints:** - No background async revalidation (SWR only does stale-fallback, not proactive refresh). - Plain DNS servers (no URI scheme) accept IPs only; URI-scheme entries (`tls://`, `https://`) support both IPs and hostnames (e.g. `tls://dot.pub`, `https://doh.pub/dns-query`). - Default transport is UDP 53 + TCP fallback; open outbound UDP 53 if your egress firewall blocks it. For DoT open outbound TCP 853; for DoH open outbound TCP 443. - Invalid configuration (malformed IP or unresolvable hostname) degrades to system resolver with a warn — never panics. ### Header and User-Agent Template Variables Both upstream `headers` values and the `user_agent` field support built-in template variables using the `{{variable}}` format. Variables are substituted at request time with per-request context, enabling upstream log correlation and user-aware routing. | Variable | Source | Description | |----------|--------|-------------| | `{{request_access_key_name}}` | Authenticated access key name | The human-readable name from the matched `AccessKey` entry | | `{{request_access_key_group}}` | Access key group name | The group the authenticated access key belongs to | | `{{request_id}}` | `Protoflux-Request-ID` header | The unique request ID from the client's `Protoflux-Request-ID` header | | `{{access_key_hash}}` | SHA-256 hash of access key name | First 16 bytes of the access key name's SHA-256 hash, base64-encoded (22 chars, deterministic) | | `{{header:Key}}` | Client request header | Value of the client's `Key` header (case-insensitive lookup, e.g. `{{header:X-Trace-Id}}`) | | `{{system_prompt_hash}}` | SHA-256 of system prompt content | Lowercase sha256 hex digest of the system prompt content, protocol-agnostic, covering Anthropic `system`, OpenAI `messages[role=system]`, DashScope `input.messages[role=system]`, and Google `system_instruction`. Computed over the upstream request body, so an upstream header injected here matches a `request_payload` `value` writing the same hash (see [Upstream Request Payload Modifications](#upstream-request-payload-modifications)) | If a variable is not available (e.g. no authenticated access key), it is replaced with an empty string. When the substituted value is entirely empty, the header is omitted. **Fallback chain syntax**: `{{var1|var2|var3}}` separates multiple candidate variables with `|`, taking the first available one in order — if a prior one is unavailable (unauthenticated / missing header), the next is tried; if all are unavailable, an empty string is substituted. Example: `{{header:X-Trace-Id|request_id}}` prefers the client's `X-Trace-Id` header, falling back to the gateway request ID if absent. `{{header:Name}}` can also be a member of a fallback chain. Unknown variables (not in the table above, no `header:` prefix) are skipped in the chain rather than erroring. Template validation runs at config load (`assemble_config`, the DB→runtime merge point) as warn-don't-block — a DB-loaded entity with a bad template still loads, only logging a warning, never blocking startup; the Console write path already rejects bad templates upstream. **Example — include access key name in User-Agent for upstream log correlation:** Console → Upstreams → New upstream `litellm-proxy` → Protocol `openai` / base_url `https://proxy.example.com/v1` / user_agent `Protoflux/key-{{request_access_key_name}}` / headers `{"X-Request-Access-Key":"{{request_access_key_name}}","X-Request-Group":"{{request_access_key_group}}"}`. **Example — multi-tier upstream cache-key header:** `{{system_prompt_hash}}` works in header templates just as it does in `request_payload` `value` expressions. The same fallback-chain syntax applies, and all the hash/identity variables are available in this context, so a single header can carry a stable cache key that degrades gracefully tier by tier: Console → Upstreams → New upstream `dashscope-multimodal` → Protocol `dashscope` / base_url `https://dashscope.aliyuncs.com/api/v1` / headers `{"X-Conversation-Id":"{{header:x-claude-code-session-id|access_key_hash|system_prompt_hash}}"}`. The chain is evaluated left-to-right, taking the first non-empty value: | Tier | Variable | When it wins | Cache-key granularity | |------|----------|--------------|----------------------| | 1 | `{{header:x-claude-code-session-id}}` | client sends the session header | same session → same key (finest, upstream cache hit within one conversation) | | 2 | `{{access_key_hash}}` | no session header, but authenticated | same access key → same key (medium, shares cache across one user's sessions) | | 3 | `{{system_prompt_hash}}` | no session header and no access key | same system prompt → same key (coarsest, degrades to `sha256("")` when there is no system prompt) | The hashes are computed over the upstream request body — the same body a `request_payload` `value` rule would see — so a header and a body rule writing the same variable produce identical values. When the request has neither a session header, nor an authenticated key, nor a system prompt, the whole chain resolves to empty and the header is **omitted** (rather than sent as a degenerate constant); to force an omission earlier, leave `system_prompt_hash` out of the chain, since on its own it degrades to the fixed `sha256("")`. > ℹ️ `system_prompt_hash` and `access_key_hash` are only resolved when a template actually references them, so adding them to a fallback chain never costs a hash on templates that don't use them. ### Upstream Request Payload Modifications You can modify the JSON request body before it is sent to the upstream using the `request_payload` field. **Rewrite** rules (`overwrite` / `add-if-absent` / `append` / `append-if-missing` / `remove` / `remove-matching` / `strip-lines` / `filter-content-types` / `filter-tools` / `inject-system-prompt`) are applied **after** protocol translation, so they affect the final upstream (target-protocol) body format. **The sole exception is `switch-route`** — it does not rewrite the body and is evaluated **before** protocol translation, against the **client** body shape, so that cross-protocol switches work (see the mode table and the "Conditional route switching" section below). Each rule has the following fields: | Field | Type | Description | |-------|------|-------------| | `path` | String | Dot-separated JSON path (e.g. `temperature`, `response_format.type`, `messages.0.content`). Unused by `switch-route` (which decides via `when`); may be left empty | | `value` | JSON | The value to set, append, or remove. A string containing `{{...}}` is resolved as an expression (see below); otherwise it is a literal. Non-string values are written verbatim | | `mode` | String | **Required.** See mode table below | | `condition` | Object | Condition for `append-if-missing`, `remove-matching`, `strip-lines`, and `filter-tools` modes. Required for those modes; ignored for others | | `when` | Array | Predicate list for `switch-route` (OR semantics: fires if any predicate holds). Each predicate is `{"path": "...", "eq": }`: omit `eq` (or `eq: null`) for a **field-existence** check, supply `eq` for an equality check. `path` supports the `*` wildcard, where a wildcard segment is existential (true if any array element has that sub-path). Ignored by other modes | | `use_model` | String | Route target model name written when `switch-route` fires. Empty is a no-op. The target may share the **same protocol or be cross-protocol**: `switch-route` triggers an eager swap before translation, so a cross-protocol target is re-translated correctly (matching the before-slot script's `context.useModel`). The target is treated as **admin-configured internal routing**: the gateway does **not** re-run the caller's group model ACL against the `use_model` target (the model ACL is checked once at request entry against the client's originally requested model); this lets a user who only holds access to model A be **transparently re-routed** to model B — that is the feature's intent, not a permission hole (the rule can only be authored by an admin; there is no untrusted input). Ignored by other modes | **`value` expressions:** In value-writing modes (`overwrite`/`add-if-absent`/`append`/`append-if-missing`), a string `value` **containing `{{...}}`** is resolved as an expression template; a string without `{{` is written verbatim as a literal, and non-string JSON values (number/boolean/object/array) are never parsed. Alternatives joined by `|` form a fallback chain whose first non-empty result wins; if the whole chain resolves to empty the write is **omitted** (the path is not created, rather than written as an empty string or constant). Variables available in this context: `header:Name` (client request header, case-insensitive; leading/trailing whitespace in the value counts as absent), `request_id`, `request_access_key_name`, `request_access_key_group`, and `system_prompt_hash` (the sha256 lowercase-hex digest of the system prompt content, protocol-agnostic, covering Anthropic `system`, OpenAI `messages[role=system]`, DashScope `input.messages[role=system]`, and Google `system_instruction`). The hash is computed from the same routine and the same upstream body the header-template context uses (see [Header and User-Agent Template Variables](#header-and-user-agent-template-variables)), so a header and a `value` rule writing `{{system_prompt_hash}}` produce identical values. Note that `access_key_hash` is always empty here — the request-body rewrite stage never holds the raw key, so cache-key fallbacks should use `system_prompt_hash`. The typical use is injecting a dynamic key — "session header first, system hash as fallback" — into an upstream-private cache field such as Kimi's `prompt_cache_key`, with no script: Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `overwrite` / path `prompt_cache_key` / value `{{header:x-claude-code-session-id|system_prompt_hash}}`. In this example the header value is used when the request carries a session header (same session → same key → upstream cache hit), and the system hash otherwise (same system → same key). If a request has neither a session header nor a system prompt, `system_prompt_hash` degrades to the fixed `sha256("")` — an informed trade-off of putting the hash in the chain; to inject nothing in that case, leave `system_prompt_hash` out of the chain (an all-empty chain omits the write). > ℹ️ Rewrite rules run **after** protocol translation, so writing an upstream-private field like `prompt_cache_key` is not dropped by translation — exactly the "cache key must be injected post-translation" requirement (`switch-route` is not among them: it is evaluated before translation and writes no body field). **Modes:** | Mode | Target | Description | |------|--------|-------------| | `overwrite` | Any | Replace the value at the target path. Creates intermediate paths if needed | | `remove` | Any | Delete the key at the target path. No-op if path doesn't exist | | `add-if-absent` | Object field | Set the value only if the key does not already exist | | `append` | Array | Push to array unconditionally; creates array if path missing | | `append-if-missing` | Array | Push to array only if **no element** in the scope array matches the `condition.where` clauses | | `remove-matching` | Array | Remove all elements in the scope array that match the `condition.where` clauses | | `strip-lines` | String | Remove lines matching the `match` clauses from text content. Use special paths (`"system"`, `"messages"`, `"input"`) to auto-iterate over all messages. Action: `remove-line`, `remove-line-cleanup`, `remove-block` | | `filter-content-types` | Array | Remove content blocks of specified types. Use special paths to target a protocol, or leave `path` empty to auto-expand to all known paths. `value` is a type string (e.g. `"image"`) or array (e.g. `["image", "video"]`) | | `filter-tools` | Array | Filter tools matching `condition.where` clauses from the `tools` array. `value` is optional: empty/null removes matched tools; non-empty removes matched tools then appends the listed tools. Also cleans up `tool_choice` when referenced tools are removed | | `switch-route` | Control plane | **Does not modify the body.** When any `when` predicate holds, switches this request's route target to `use_model`. **Unlike every other mode, it is evaluated before protocol translation**, with predicate paths written against the **client** body shape (e.g. an Anthropic client uses `messages.*.content.*.type` equal to `"image"`); a match triggers an eager swap that **supports cross-protocol targets** (matching the before-slot script's `context.useModel`). Predicates perform only structural / small-value checks and never read binary content bytes, so even when a `when` path targets a binary key like `image`/`image_url`, the streaming spill gate for large bodies stays ON (the body remains on disk instead of being fully materialized into memory). Use it for pure routing decisions such as "switch model when multimodal / web-search features are detected" — the native, memory-conscious replacement for a script's `useModel(...)` | **Path syntax:** The `path` field uses dot-separated segments to navigate the JSON body. Each segment is either an object key or a numeric array index. | Syntax | Example | Description | |--------|---------|-------------| | Object key | `temperature` | Top-level field | | Nested keys | `response_format.type` | Drill into nested objects | | Array index | `messages.0.content` | Access element at index 0 (bare number, no brackets) | | Wildcard | `messages.*.content` | Match **all** elements of an array (works with all modes) | | Deep nesting | `messages.0.content.1.text` | Mix object keys and array indices freely | | Nested wildcards | `messages.*.content.*.text` | Multiple `*` segments for deeply nested arrays | > ⚠️ `messages[].content`, `messages[*].content`, `messages[0].content` are **invalid**. Use `messages.*.content` for wildcard or `messages.0.content` for specific indices. **Wildcard behavior:** - `*` iterates over all elements of an array. If the current node is not an array, it is a no-op. - Wildcard paths only operate on **existing** nodes — they do not create new array elements or intermediate paths. - Works with all modes: `overwrite`, `remove`, `add-if-absent`, `append`, `strip-lines`. **Special paths for `strip-lines` and `filter-content-types`:** These reserved path values auto-iterate over array elements (equivalent to using `*` but shorter): | Path | Iterates over | Protocol | `strip-lines` | `filter-content-types` | |------|--------------|----------|:---:|:---:| | `"system"` | `body.system` (string or content block array) | Anthropic | ✅ | ✅ | | `"messages"` | `body.messages[].content` | Anthropic / OpenAI Chat / Bedrock | ✅ | ✅ | | `"input"` | `body.input[].content` or `body.input` as string | OpenAI Responses API | ✅ | ✅ | | `"contents"` | `body.contents[].parts` | Google Gemini | ❌ | ✅ | | `"input.messages"` | `body.input.messages[].content` | DashScope | ❌ | ✅ | > ℹ️ `"contents"` and `"input.messages"` only work with `filter-content-types`. For `strip-lines` with these protocols, use wildcard paths (e.g., `contents.*.parts.*.text`). For non-wildcard paths in modes like `overwrite`, `remove`, `append`, the path resolves to a single value — use explicit indices like `messages.0.content` to target specific elements. Non-wildcard paths that do not exist are created automatically (intermediate objects and arrays padded with null). Wildcard paths skip missing nodes. **Simple overwrite:** Console → Upstreams → New upstream `openai` → base_url `https://api.openai.com/v1` → Request Payload Rules → **+ Add Rule** → mode `overwrite` / path `temperature` / value `0.7`. **Nested overwrite:** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `overwrite` / path `response_format.type` / value `json_schema`. **Remove a field:** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `remove` / path `temperature` (value unused). **Add if absent (preserves client value):** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `add-if-absent` / path `top_p` / value `0.9`. **Append to array:** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `append` / path `tools` / value `{"type":"function","function":{"name":"web_search","description":"Search the web"}}`. **Append to messages array:** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `append` / path `messages` / value `{"role":"system","content":"You are a helpful assistant"}`. **Combined rules (multiple modes):** Upstream (or Model) → Request Payload Rules → add one rule per row below: | # | Mode | Path | Value | |---|------|------|-------| | 1 | `overwrite` | `temperature` | `0.3` | | 2 | `overwrite` | `max_tokens` | `4096` | | 3 | `overwrite` | `response_format.type` | `json_schema` | | 4 | `append` | `tools` | `{"type":"function","function":{"name":"calculator"}}` | Rules are applied in order, so later entries can overwrite earlier ones for the same path. ##### Strip Lines from Content The `strip-lines` mode removes matching lines from text content. The `match` field specifies how to match lines (`starts_with`, `contains`, or `regex`). **Remove billing header lines from all messages (Anthropic):** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `strip-lines` / path `messages` / match `starts_with` `x-anthropic-billing-header:`. **Remove billing header lines from OpenAI Responses API input:** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `strip-lines` / path `input` / match `starts_with` `x-anthropic-billing-header:`. **Remove lines containing a substring from system prompt:** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `strip-lines` / path `system` / match `contains` `internal-debug-flag`. **Target a specific message by index (not all messages):** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `strip-lines` / path `messages.0.content` / match `regex` `^\[INTERNAL\]` / action `remove-line`. **Using wildcard paths (all modes):** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `overwrite` / path `messages.*.role` / value `user`. Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `remove` / path `messages.*.metadata`. Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `append` / path `messages.*.tags` / value `reviewed`. ##### Filter Content Types The `filter-content-types` mode removes content blocks of specified types (e.g. `image`, `video`) from message content arrays. **Remove image blocks from messages:** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `filter-content-types` / path `messages` / value `image`. **Remove multiple content types from all known paths (auto-expand):** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `filter-content-types` / path *(leave empty to auto-expand all known paths)* / value `["image","video"]`. **Remove image blocks from OpenAI Responses API input:** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `filter-content-types` / path `input` / value `image`. **Placeholder hint text:** The placeholder text injected when blocks are removed can be customized via the `hint` field. When `hint` is unset and the filtered types include `image`, a default text is used: > [Note: The user sent image(s) in this message, but they were removed because the current model does not support image input. Please respond based on the text content only.] - When `hint` is set, a text block with that content replaces the first removed block per message. - Setting `hint` to an empty string `""` injects an empty text block (it does not disable injection). - The placeholder is injected only when blocks are actually removed; images nested inside `tool_result` are removed without a placeholder. - The injected block format adapts to the target protocol: `{"type":"text","text":...}` for Anthropic / OpenAI / Bedrock / DashScope, `{"text":...}` for Google. - The Console API does not support the `hint` field (only the default text is available); to set a custom placeholder, write the `request_payload` rule directly in the DB. ##### Conditional Array Operations The `append-if-missing` and `remove-matching` modes operate on arrays conditionally. They require a `condition` object with two fields: | Field | Type | Description | |-------|------|-------------| | `scope` | String | Dot-separated path to the array whose elements are tested (e.g. `messages`) | | `where` | Array of objects | Clauses that must all match (AND logic). Each clause has `path` (relative to array element) and `eq` (value to match) | **Conditionally append thinking block (only if absent):** Console → Models → New model `kimi-2.6` → upstream `anthropic` → Request Payload Rules → **+ Add Rule** → mode `append-if-missing` / path `messages` / value `{"type":"thinking","thinking":{"budget_tokens":10000}}` / condition scope `messages`, where `type` = `thinking`. This ensures the upstream request always has a `thinking` block in the messages array, but won't duplicate it if the client already sent one. **Remove thinking blocks:** Console → Models → New model `kimi-2.6-lite` → upstream `anthropic` → Request Payload Rules → **+ Add Rule** → mode `remove-matching` / path `messages` / condition scope `messages`, where `type` = `thinking`. **Conditionally append system message:** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `append-if-missing` / path `messages` / value `{"role":"system","content":"You are a helpful assistant."}` / condition scope `messages`, where `role` = `system`. **Multiple `where` clauses (AND logic):** Upstream (or Model) → Request Payload Rules → **+ Add Rule** → mode `append-if-missing` / path `messages` / value `{"type":"text","role":"system","text":"You are helpful."}` / condition scope `messages`, where `type` = `text` AND `role` = `system`. This only appends if no element has **both** `type = "text"` **and** `role = "system"`. **Conditional route switching (`switch-route`, memory-conscious and protocol-agnostic):** In one sentence: **"if the request body contains X, send this request to a different model instead"** — it changes none of the body's content, only *who this request goes to*. The most common use is "the user attached an image / enabled web search, so switch to a model that can see images / browse the web". > ⚙️ **Evaluation timing and path shape (important):** `switch-route` is evaluated **before** protocol translation, and the `when` paths are written against the **client** body shape — i.e. "whatever protocol the caller speaks, write the path in that protocol's field structure". For example, an Anthropic client's image block is `messages.*.content.*.type` equal to `"image"`, an OpenAI Chat client's is `messages.*.content.*.image_url` (existence), and a DashScope client's is `input.messages.*.content.*.image` (existence). Because evaluation happens before translation, a match triggers an eager swap that **supports cross-protocol targets** — the gateway re-translates the client body with the target protocol's translator, matching the before-slot script's `context.useModel`. If a single rule must tolerate **several client protocols**, list each client shape's path in `when` with OR (see the example below). This is the opposite of rewrite rules, which run **after** translation with paths in the **target** protocol shape. To configure it in the Console (no hand-written JSON): open the **model detail** (or the upstream's *default request payload rules*) → find **Request Payload Rules** → click **+ Add Rule** → pick **`switch-route`** in the middle mode dropdown → that row then hides *path / value* and reveals two dedicated controls: - **Switch to model (use_model)**: the model name to reroute to when matched. ✅ It may share the **same protocol or be cross-protocol** with the current model — `switch-route` triggers an eager swap before translation, so a cross-protocol target is re-translated. - **Match conditions (when)**: one or more *predicates*. Each predicate takes a **field path** (written in the **client** protocol shape, see the note above) and an optional **equals** value: - **equals left blank** = "matches as long as this field exists" (use it to detect "is there an image", i.e. existence). - **equals filled** = "the field's value must equal this to match" (e.g. `true`, `image`). - A `*` in the path means "any element of the array", e.g. `messages.*.content.*.type` = "the `type` field of some content block of some message" (the exact path varies with the client protocol). - Multiple predicates are OR-combined: **any one** matching triggers the switch; if none match, the original model is kept. The simplest example — "switch to the vision model when the request carries an image" (a single existence predicate; **this example assumes a DashScope client**, hence the `input.messages.*.content.*.image` shape). Console form steps: 1. Model detail → Request Payload Rules → **+ Add Rule** → pick `switch-route`; 2. **Switch to model** dropdown → `qwen-vl-max`; 3. Add one predicate: path = `input.messages.*.content.*.image`, **equals** left blank (i.e. existence check). Read as: when **any** message's **any** content block has an `image` field, reroute this request to `qwen-vl-max`; otherwise send it to this model as usual. Note that `switch-route` evaluates against the **client** shape, so this `input.messages.*` path only matches a **DashScope client**; if the caller speaks Anthropic / OpenAI, swap the path for the matching client shape, or list several client shapes with OR as in the next example. > Note: `switch-route` only checks "is the field present / does a small value equal X" and **never reads binary content** such as image base64. That is precisely why a large body carrying big images still streams to disk and is never fully read into memory — which is its key advantage over a script's `context.useModel(...)`: a script materializes the whole body (with inline base64) into the interpreter heap and can OOM on large multimodal conversations, whereas `switch-route` does not. So for pure routing decisions like "detect a feature → switch model", prefer `switch-route` over a script. `switch-route` neither reads nor writes the body content; it only re-targets this request to `use_model` when a `when` predicate holds. It natively replaces a script's `context.useModel(...)`, and because its predicates perform only structural / small-value checks and never dereference binary bytes, **the streaming spill gate for large bodies stays ON** — a multimodal body carrying base64 images remains on disk as sentinels + blobs instead of being fully materialized into the interpreter heap and OOM-ing the way a script would. `when` is an OR-combined predicate list with fully configurable paths, so a single rule can cover the image / web-search signatures of several **client** protocols (`switch-route` evaluates against the client shape, so these are the field shapes of each client protocol). Pick `switch-route`, target `qwen3.7-max`, and add four predicates (each = path + optional **equals**): | # | Path | Equals | Matches | |---|------|--------|---------| | 1 | `input.messages.*.content.*.image` | blank | DashScope client image block (existence) | | 2 | `input.messages.*.content.*.image_url` | blank | OpenAI client image block (existence) | | 3 | `input.messages.*.content.*.type` | `image` | Anthropic client image block | | 4 | `parameters.enable_search` | `true` | web-search toggle (any protocol) | These four match a DashScope client image block, an OpenAI client image block, an Anthropic client image block, and a web-search toggle under any protocol respectively — switching **client** protocol means editing JSON paths, not code; listing several client shapes with OR lets one rule work for any client protocol. The request switches to `qwen3.7-max` as soon as any predicate holds; if none hold, the original route is kept. The `use_model` target may share the current route's protocol or be cross-protocol. `switch-route` composes with rewrite rules such as `strip-lines` to fully replace an after-slot script that both "detects a feature and switches route" and "strips a system prefix". Note the two rules evaluate at different stages: `switch-route` judges **before** translation against the **client** shape, while `strip-lines` rewrites **after** translation against the **target** shape — in the retirement case below both paths happen to be written as `input.messages.*` because that case **assumes a DashScope client** (where the client shape coincides with the target shape). Configure two rules: 1. **switch-route**: target dropdown → `qwen3.7-max`; add two predicates — path `input.messages.*.content.*.image` (equals blank), path `parameters.enable_search` (equals `true`). 2. **strip-lines**: path `input.messages.*.content`; match `starts_with`, value `You are Claude Code, Anthropic's official CLI`; action `remove-line-cleanup`. > ℹ️ Once the native rules above are configured on the model, retire the original JS by clearing that model's after-slot inline script via the Console — with the script gone, the spill gate stays ON for the native rules on that route. Note that with the gate ON, `strip-lines` only acts on text blocks that were not blobified (system prompts are a few KB and are unaffected); a text block exceeding the blob threshold becomes a sentinel and is not stripped — an informed boundary versus the original script (which OOM-ed / silently skipped on large bodies), and strictly better. **Full script-retirement case (after-slot `transform(body, context)` → native rules):** The after-slot script below does three things — detect image blocks, detect the web-search toggle, switch to the multimodal model via `useModel` when either fires, and strip lines starting with a fixed prefix from the system prompt. On large multimodal conversations it OOMs the interpreter heap (the script receives a fully-materialized `body` with inline base64). It can be replaced, semantically equivalent, by `switch-route` + `strip-lines`. **Caveat**: here the target protocol is DashScope, so the after-slot script sees the translated DashScope shape and works for any client; `switch-route`, however, judges **before** translation against the **client** shape, so the native equivalence below **assumes the client also speaks DashScope** — only then does the client shape match the translated shape the script saw, letting `switch-route`'s `when` paths correspond to the script's field paths one-to-one. If the client uses another protocol, rewrite `switch-route`'s `when` to list the matching client shapes (or several with OR, as above); the `strip-lines` path stays unchanged (it still rewrites in the target shape): ```js function transform(body, context) { const msgs = body.input?.messages; // 1. Detect image blocks: DashScope multimodal blocks are {"image": ""} (no type field) let hasImage = false; if (Array.isArray(msgs)) { for (const msg of msgs) { if (!Array.isArray(msg.content)) continue; if (msg.content.some(b => b.image)) { hasImage = true; break; } } } // 2. image OR search → switch to the multimodal model (must use useModel: target is on another endpoint) if (hasImage || body.parameters?.enable_search) { context.useModel("qwen3.7-max"); } // 3. Strip lines starting with the prefix from the system prompt; drop the block if it empties const PREFIX = "You are Claude Code, Anthropic's official CLI"; if (!Array.isArray(msgs)) return body; for (let i = msgs.length - 1; i >= 0; i--) { const msg = msgs[i]; if (msg.role !== "system" || !Array.isArray(msg.content)) continue; msg.content = msg.content.filter(block => { if (block.type !== "text" || typeof block.text !== "string") return true; if (!block.text.startsWith(PREFIX)) return true; const remaining = block.text.split("\n").filter(l => !l.startsWith(PREFIX)); block.text = remaining.join("\n"); return block.text.trim().length > 0; }); if (msg.content.length === 0) msgs.splice(i, 1); } return body; } ``` Equivalent native configuration. Note the two rules' evaluation stages: `strip-lines` runs **after** protocol translation, with its path in the **target** shape (the target here is DashScope, hence `input.messages.*.content`); `switch-route` runs **before** protocol translation, with its path in the **client** shape — the client here happens to be DashScope, so its `when` paths share the `input.messages.*` shape with `strip-lines` and look identical, but the semantic source differs. Configure two rules: 1. **switch-route**: target dropdown → `qwen3.7-max`; add two predicates — path `input.messages.*.content.*.image` (equals blank), path `parameters.enable_search` (equals `true`). 2. **strip-lines**: path `input.messages.*.content`; match `starts_with`, value `You are Claude Code, Anthropic's official CLI`; action `remove-line-cleanup`. Line-by-line correspondence: - `msg.content.some(b => b.image)` (existence-only, never reads the base64) → one predicate: path `input.messages.*.content.*.image`, equals blank (existence check; `*` expands messages and content blocks). - `body.parameters?.enable_search` (the adapter already turned web_search into this boolean) → one predicate: path `parameters.enable_search`, equals `true`. - The `hasImage || enable_search` OR → the `when` list is OR-combined by definition; any match fires. - `context.useModel("qwen3.7-max")` → `use_model`. `switch-route` uses the pre-translation eager swap, which **supports cross-protocol targets**; in this case the target is the same same-protocol endpoint switch as the original after-slot script, so the two are equivalent. The original after-slot script's `useModel` is limited to same-protocol by the after position, whereas `switch-route` has no such limit — which is the extra payoff of moving "detect a feature → switch route" out of an after-slot script into `switch-route`. - The system-prefix filter → `strip-lines` + `match.starts_with` + `action: remove-line-cleanup`: match the prefix per line, remove matching lines, drop the block when it empties — equivalent to the `filter` + `splice`. It only touches blocks carrying a `text` string; image blocks (no `text`) are left intact, matching the original's `block.type !== "text"` guard. > ℹ️ After configuring the rules above, retire the script by clearing that model's after-slot inline script via the Console; with the script gone, the spill gate stays ON for the native rules on that route, and a base64-heavy body stays on disk as sentinels + blobs instead of being materialized into the interpreter heap. Minor difference: if a system text block consists **only** of prefix lines and blank lines, the original keeps an empty text block while `remove-line-cleanup` removes it — cleaner, and immaterial to the model input. #### Model-Level Request Payload `request_payload` can also be configured at the model level. Model-level rules take priority over upstream-level rules. Both sets of rules are merged with upstream rules applied first, then model rules — so model rules can override or extend upstream rules for the same paths. Console → Models → New model `qwen3.6-plus-strict` → upstream `openai` → Request Payload Rules → add one rule per row below: | # | Mode | Path | Value | |---|------|------|-------| | 1 | `overwrite` | `temperature` | `0` | | 2 | `overwrite` | `response_format.type` | `json_schema` | This is useful when different models need different body modifications while sharing the same upstream config. For example, the upstream sets a default `temperature` for all models, but a specific model overrides it to `0` for deterministic output. #### Model-Level Protocol Override Models can override the upstream's protocol using the `protocol` field. Combined with `base_url`, this allows a single upstream credential pool to serve multiple models with different API formats (e.g., an endpoint that offers both OpenAI and Anthropic compatible paths). Console → Upstreams → New upstream `aws-global` → Protocol `aws_converse` / base_url `https://bedrock-runtime.us-west-2.amazonaws.com` / API Keys `sk-...` (weight `1`). Console → Models → New model `glm-5` → upstream `aws-global` / Upstream Model ID `zai.glm-5` / Protocol Override `openai` / base_url `https://bedrock-runtime.us-west-2.amazonaws.com/openai/v1/`. ## OpenAI Images API Protoflux supports the OpenAI Images API for image generation and editing via the `openai_images` protocol with `direct_path = true`. Two endpoints are proxied: - **`POST /v1/images/generations`** — Generate images from text prompts (JSON body) - **`POST /v1/images/edits`** — Edit images using prompts and optional masks (multipart/form-data with file uploads) When `direct_path = true`, the model's `base_url` is used as the **complete endpoint URL** — no path is appended. This allows configuration of deployment-specific URLs for Azure OpenAI or standard OpenAI endpoints. ### OpenAI Direct Console → Upstreams → New upstream `openai-images` → Protocol `openai_images` / base_url `https://api.openai.com/v1` / API Keys `sk-...` (weight `1`). Console → Models → New model `qwen-vl-max` → upstream `openai-images` / Upstream Model ID `qwen-vl-max` / Protocol Override `openai_images` / direct_path `true` / base_url `https://api.openai.com/v1/images/generations`. ### Azure OpenAI Azure requires deployment-specific URLs with API version query parameters: Console → Upstreams → New upstream `azure-openai-images` → Protocol `openai_images` / base_url `https://{resource}.openai.azure.com/openai/v1` / API Keys `sk-...` (weight `1`). Console → Models → New model `qwen-vl-max` → upstream `azure-openai-images` / Upstream Model ID `qwen-vl-max` / Protocol Override `openai_images` / direct_path `true` / base_url `https://{resource}.openai.azure.com/openai/deployments/qwen-vl-max/images/generations?api-version=2025-04-01-preview`. ### Gemini (Imagen) Image Models Gemini image models (`gemini-3-pro-image-preview`, Imagen series) use the **same `generateContent` endpoint** as text models — they do not have a dedicated image API like OpenAI's `/v1/images/generations`. Image output is requested via `responseModalities: ["IMAGE"]` in `generationConfig`. Configure these models with `protocol = "google"` (not `openai_images`). The gateway automatically translates between the client-facing OpenAI Images format and Gemini's `generateContent` format: Console → Upstreams → New upstream `google-gemini` → Protocol `google` / base_url `https://generativelanguage.googleapis.com` / API Keys `sk-...` (weight `1`). Console → Models → New model `gemini-3-pro-image-preview` → upstream `google-gemini` / Upstream Model ID `gemini-3-pro-image-preview` / Protocol Override `google`. Clients call the standard `POST /v1/images/generations` with an OpenAI-format body: ```json { "model": "gemini-3-pro-image-preview", "prompt": "A cat wearing a spacesuit on Mars", "n": 1, "size": "1024x1024" } ``` Key differences from OpenAI image upstreams: | | OpenAI Image Upstream | Gemini Image Upstream | |---|---|---| | `protocol` | `openai_images` | `google` | | `direct_path` | `true` | `false` (default) | | Upstream endpoint | `/v1/images/generations` | `/v1beta/models/{model}:generateContent` | | Response format | `{data: [{b64_json}]}` | `{candidates: [{content: {parts: [{inlineData}]}}]}` | | Same model serves text chat | No | Yes (via `POST /v1/chat/completions`) | The translation is handled by the `OpenAIImagesToGoogle` translator (request) and `GoogleToOpenAIImages` translator (response), registered in the protocol translation matrix. See [Protocol Translation Matrix](../architecture.md#protocol-translation-matrix) in the architecture docs for details. #### Native Google Protocol (Passthrough) Clients can also use the **native Gemini protocol** directly. When calling `POST /v1beta/models/{model}:generateContent` (or the `/v1/models/{model}:generateContent` stable alias), the `client_protocol` and `target_protocol` are both `Google` — the pipeline resolves to an identity (passthrough) with no translation step. The request body is sent to the upstream as-is. The `/v1` path is a client-facing alias; the upstream URL is always constructed as `/v1beta/...` (a strict superset of v1, maximising model compatibility): ```json POST /v1beta/models/gemini-3-pro-image-preview:generateContent { "contents": [{ "role": "user", "parts": [{ "text": "A cat wearing a spacesuit on Mars" }] }], "generationConfig": { "responseModalities": ["IMAGE"] } } ``` This gives clients full control over Gemini-specific parameters (safety settings, aspect ratio, sample count, etc.) without going through the OpenAI Images translation layer. ### Request Parameters The images API accepts standard OpenAI Images parameters: | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `model` | string | yes | Model identifier (e.g., `qwen-vl-max`) | | `prompt` | string | yes | Text description of the desired image | | `n` | integer | no | Number of images to generate (default: 1) | | `size` | string | no | Image size (e.g., `1024x1024`, `2048x2048`) | | `quality` | string | no | Image quality (`low`, `medium`, `high`, `auto`) | | `response_format` | string | no | Response format (`png`, `jpeg`, `webp`) | | `stream` | boolean | no | Enable streaming output (default: false) | ### Streaming Both generation and edits support streaming via `stream: true`. When enabled, the gateway proxies SSE events from the upstream to the client. ### Image Editing (multipart) The `/v1/images/edits` endpoint accepts `multipart/form-data` with: - `image[]` — One or more image files to edit (PNG/JPEG, <50MB each) - `mask` — Optional mask image with alpha channel (PNG, same size as input) - `prompt` — Text description of the edit - `model`, `n`, `size`, `quality`, `response_format`, `stream`, etc. The gateway collects the multipart data, reconstructs it with the upstream model ID, and forwards it to the configured endpoint. ## DashScope API Protoflux supports Alibaba Cloud DashScope API with `protocol = "dashscope"`. DashScope has different endpoints for different model types (text generation, multimodal, image generation, video generation, image synthesis). Use `direct_path = true` and set the model-level `base_url` to the full endpoint URL. ### Endpoint Reference | API type | Example models | Full URL for `base_url` | |----------|---------------|--------------------------| | Text generation | qwen-turbo, qwen-plus, qwen-max | `https://dashscope.aliyuncs.com/api/v1/services/aigc/text-generation/generation` | | Multimodal generation | qwen-vl-chat-v1, qwen-vl-plus | `https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation` | | Image generation (sync) | wan2.6-image | `https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation` | | Image generation (async) | wan2.6-t2i | `https://dashscope.aliyuncs.com/api/v1/services/aigc/image-generation/generation` | | Video generation | wanx2.1-t2v, happyhorse | `https://dashscope.aliyuncs.com/api/v1/services/aigc/video-generation/video-synthesis` | | Image synthesis | wanx-v1 | `https://dashscope.aliyuncs.com/api/v1/services/aigc/text2image/synthesis` | | Text embedding | text-embedding-v4 | `https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding` | ### Configuration Examples **Text generation (default endpoint — no `direct_path` needed)**: Console → Upstreams → New upstream `dashscope` → Protocol `dashscope` / base_url `https://dashscope.aliyuncs.com` / API Keys `sk-...` (weight `1`). Console → Models → New model `qwen-turbo` → upstream `dashscope` / Upstream Model ID `qwen-turbo` / Protocol Override `dashscope`. **Video generation (custom endpoint with `direct_path`)**: Console → Upstreams → New upstream `dashscope` → Protocol `dashscope` / base_url `https://dashscope.aliyuncs.com` / API Keys `sk-...` (weight `1`). Console → Models → New model `wanx2.1-t2v` → upstream `dashscope` / Upstream Model ID `wanx2.1-t2v` / Protocol Override `dashscope` / direct_path `true` / base_url `https://dashscope.aliyuncs.com/api/v1/services/aigc/video-generation/video-synthesis`. **Text embedding (`kind` selects endpoint — no `direct_path` needed)**: Text embedding models use the Bailian MaaS domain (`maas.aliyuncs.com`), which differs from the text-generation `dashscope.aliyuncs.com` — configure a separate upstream. Set `kind = "dashscope-text-embedding"` to select the endpoint path; `base_url` is just the domain root and the gateway appends the endpoint path based on `kind`. Console → Upstreams → New upstream `dashscope-maas` → Protocol `dashscope` / base_url `https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com` / API Keys `sk-...` (weight `1`). Console → Models → New model `text-embedding-v4` → upstream `dashscope-maas` / Upstream Model ID `text-embedding-v4` / Protocol Override `dashscope` / kind `dashscope-text-embedding`. Clients send standard OpenAI `/v1/embeddings` requests; the gateway translates to DashScope text-embedding format (`input.texts`) and translates the response back to OpenAI format. `encoding_format: "base64"` is encoded on the response side by the translator as a base64 string of little-endian f32 bytes. `stream: true` is rejected (embedding models are non-streaming). **Rerank (OpenAI-compatible endpoints + DashScope translation)**: The gateway serves OpenAI-compatible rerank on `POST /v1/rerank` and `POST /v2/rerank` (de facto standard: `/v1/rerank` = Jina v1 / Cohere v1, `/v2/rerank` = Cohere v2). Two upstream paths: - **DashScope translation**: model `protocol = "dashscope"` + `kind = "dashscope-rerank"`. The client sends an OpenAI/Cohere/Jina rerank body (`{model, query, documents:[str|{text}], top_n, return_documents?}`); the gateway translates to DashScope text-rerank's wrapped format (`{model, input:{query, documents:[string]}, parameters:{top_n?, return_documents?, instruct?}}`, `documents` normalized to a string array — a flat body is rejected with `Field required: input.query & input.documents`) and translates the response back to a client-facing superset carrying both billing shapes: `{id, model, results:[{index, relevance_score, document?}], usage:{prompt_tokens, total_tokens}, meta:{billed_units:{input_tokens, output_tokens, search_units}, tokens:{input_tokens, output_tokens}}}` (sorted by `relevance_score` descending). Jina/vLLM clients read `usage`; Cohere v1/v2 clients read `meta.billed_units`/`meta.tokens`. v1 string documents and v2 `{text}` objects are both accepted; `rank_fields`, `max_chunks_per_doc`, and `max_tokens_per_doc` are dropped (DashScope text-rerank has no equivalent). - **OpenAI-compatible upstream**: model `protocol = "openai_rerank"`, served by the dedicated `OpenAIRerankExecutor`; the request body and response pass through unchanged (same-protocol identity translation), and the executor appends `/rerank` to `base_url` (`direct_path = true` uses `base_url` verbatim). Use this for vLLM / Cohere / Jina / SiliconFlow upstreams that are natively compatible. To target a specific version, set the version prefix in `base_url` — the executor appends `/rerank`, so `base_url = ".../v1"` yields `/v1/rerank` and `base_url = ".../v2"` yields `/v2/rerank`; use `direct_path = true` with the full URL for non-standard paths. Rerank is non-streaming; `stream: true` is rejected. `/v1/rerank` and `/v2/rerank` are not treated as separate protocols — both resolve to a single `OpenAIRerank` protocol with no path-based branching. Their contents differ (v2 adds `{text}` object documents, `rank_fields`, and an object-shaped `document` in the response), but the gateway collapses them: the DashScope translator accepts both v1 string-array and v2 `{text}`-object documents and emits a v2-canonical superset response (`document` as `{text}` objects, carrying both `usage` and `meta`); on the `openai_rerank` path the body passes through unchanged, but the executor always appends `/rerank` to `base_url` — the client's `/v1` or `/v2` path is not forwarded, so the upstream version is selected by the version prefix in `base_url` (or `direct_path = true` with a full URL), and v2-only request fields like `rank_fields` require a v2-capable upstream. - **DashScope translation path** (Bailian models like gte-rerank): Console → Models → New model `gte-rerank` → upstream `dashscope-maas` / Upstream Model ID `gte-rerank` / Protocol Override `dashscope` / kind `dashscope-rerank`. - **OpenAI-compatible upstream path** (vLLM/Cohere/Jina): Console → Models → New model `bge-reranker-v2-m3` → upstream `vllm-rerank` / Upstream Model ID `BAAI/bge-reranker-v2-m3` / Protocol Override `openai_rerank` / direct_path `false`. Client call: ```bash curl -X POST http://localhost:7890/v1/rerank \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gte-rerank", "query": "capital of France", "documents": ["Paris", "London", "Berlin"], "top_n": 2 }' ``` DashScope-native clients can still send `POST /v1/services/{*rest}` (DashScope native body) and share the same `dashscope-rerank` model. **Audio transcription (multimodal, OpenAI-compatible)**: Audio-to-text multimodal models such as `qwen-audio` use the synchronous multimodal endpoint (`multimodal-generation/generation`), selected by `kind = "dashscope-multimodal"`. This kind differs from `dashscope-audio-asr` (async paraformer) below — it is synchronous multimodal and bidirectionally compatible with OpenAI `/v1/audio/transcriptions`. `base_url` needs only the domain root; the executor appends the endpoint path from `kind`, so reuse the same `dashscope` upstream as text generation. A model configured with this kind can serve both as a DashScope-native multimodal upstream and as an OpenAI audio-transcription upstream: - Forward: a client sends an OpenAI `/v1/audio/transcriptions` (multipart `file` upload) → the gateway base64-encodes the audio into a DashScope multimodal JSON (`content: [{audio: "data:;base64,..."}, {text: transcription instruction}]`) → calls the DashScope executor → translates `{output:{choices:[{message:{content}}]}}` back to OpenAI `{text}`. The `prompt`/`language` fields are injected into the transcription instruction. - Reverse: a DashScope client sends a multimodal JSON → the gateway extracts the base64 audio from `content` → builds an OpenAI multipart transcription request → calls the OpenAI Audio (whisper) executor → wraps `{text}` back into a DashScope response. The reverse path accepts base64 inline audio only; URL input reports `unsupported_feature`. `/v1/audio/speech` (text-to-speech) is not supported against a DashScope upstream — DashScope multimodal is audio→text only, and the text→speech path reports `unsupported_feature` via the pipeline. Console → Models → New model `qwen-audio` → upstream `dashscope` / Upstream Model ID `qwen-audio` / Protocol Override `dashscope` / kind `dashscope-multimodal`. Clients can then send a standard OpenAI `/v1/audio/transcriptions` multipart upload; the gateway translates it to DashScope multimodal format and back to `{text}`. **Synchronous audio transcription (`fun-asr-flash`, OpenAI compatible)**: Synchronous ASR models such as `fun-asr-flash` share the multimodal endpoint (`multimodal-generation/generation`) with `qwen-audio` — configure them with **`kind = "dashscope-multimodal"`** directly. `kind` only selects the endpoint URL, not the request-body shape. Unlike `dashscope-audio-asr` (async paraformer, requires task polling) below, `fun-asr-flash` is a synchronous call that returns the transcript directly — no polling. `base_url` needs only the domain root; the executor appends the endpoint path from `kind`, so reuse the same `dashscope` / `dashscope-maas` upstream as text generation. The request-body shape is selected by **the client request's protocol endpoint**, not by `kind`: when the client uses OpenAI `/v1/audio/transcriptions` (multipart `file` upload), the gateway base64-encodes the audio into an OpenAI-style `input_audio` content block (`content: [{type:"input_audio", input_audio:{data:"data:;base64,..."}}]`) and fills `parameters.format` with the container format derived from the content-type (`audio/mpeg` → `mp3`, `audio/wav` → `wav`, etc.) → calls the DashScope executor → extracts the transcript from `output.text` (fallback `output.sentence.text`) → returns OpenAI `{text}`. > Difference from `qwen-audio` is in **request/response shape only, not `kind`**: both use `kind = "dashscope-multimodal"` and the same endpoint; but `qwen-audio` (via a DashScope-native client) uses an `{"audio": ...}` content block + a text instruction and reads `output.choices[0].message.content`, whereas `fun-asr-flash` (via OpenAI `/v1/audio/transcriptions`) uses an `input_audio` block + `parameters.format` and reads `output.text`. The shape difference is driven by the client endpoint, so no separate `kind` is needed. Console → Models → New model `fun-asr-flash` → upstream `dashscope-maas` / Upstream Model ID `fun-asr-flash-2026-06-15` / Protocol Override `dashscope` / kind `dashscope-multimodal`. Clients can then send a standard OpenAI `/v1/audio/transcriptions` multipart upload; the gateway builds the `fun-asr-flash` `input_audio` format (driven by the client endpoint) and translates back to `{text}`. **Audio ASR transcription (`kind` selects endpoint — async, DashScope native only)**: Audio ASR is async. Set `kind = "dashscope-audio-asr"` to select the endpoint path and `extra_config.dashscope.async_mode = true` so the executor injects `X-DashScope-Async: enable`. There is no OpenAI-compatible ASR protocol; clients send DashScope native format to `POST /v1/services/{*rest}` and poll `GET /v1/services/{model}/tasks/{task_id}`. Console → Models → New model `fun-asr` → upstream `dashscope-maas` / Upstream Model ID `fun-asr` / Protocol Override `dashscope` / kind `dashscope-audio-asr` / extra_config `{"dashscope":{"async_mode":true}}`. ### Client Endpoints & Protocol Translation Protoflux provides dedicated endpoints for DashScope clients, supporting both synchronous and asynchronous operations: | Client Request | Description | Translation Mode | |----------------|-------------|------------------| | `POST /v1/services/{*rest}` | Unified DashScope service endpoint (text, multimodal, image, video, embedding, rerank, audio ASR). Async mode is controlled by `extra_config.dashscope.async_mode` on the model config. | Direct (passthrough) | | `GET /v1/services/{model}/tasks/{task_id}` | Poll async task status. Proxies to DashScope task API. | Direct (passthrough) | | `POST /v1/chat/completions` | OpenAI format request routed to a DashScope model. | Translated | When calling `/v1/chat/completions` for a DashScope model, the gateway automatically translates the OpenAI Chat Completions payload to DashScope's native format and translates the response back to OpenAI format via the `OpenAIChatCompletionsToDashScope` translator. ### Async Mode For async-only model types (video generation, image synthesis, audio ASR), clients submit requests to the `POST /v1/services/{*rest}` unified endpoint. In addition to calling the correct endpoint, you must configure the model with `extra_config.dashscope.async_mode = true`. This tells the executor to handle the async specific HTTP headers (the gateway automatically adds `X-DashScope-Async: enable` to the upstream request). After submitting a task, the gateway returns `{task_id, task_status}` immediately. The client then polls `GET /v1/services/{model}/tasks/{task_id}` until the status is `SUCCEEDED` or `FAILED`. ## Access Key & Group Access Control Protoflux supports fine-grained model access control through **access keys** and **access key groups**: - **`access_keys`** — each entry defines a client API key, an optional human-readable `name`, an optional `groups` list (a key may belong to multiple groups), and an `enabled` flag. - **`access_key_groups`** — each group has a `name`, a `models` list, and a `load_balancers` list; the two dimensions are independent. An empty `models` list means no model access (deny-by-default); `models = ["*"]` allows all models. `load_balancers` governs load balancer access the same way (empty = deny all, `["*"]` = allow all, entries are canonical load balancer names). Allowing all models does **not** implicitly grant load balancer access. ### Authentication Resolution Authentication uses a **short-circuit chain** evaluated in order: 1. `Authorization: Bearer ` — extract key, attempt access key match 2. `x-api-key` — extract key, attempt access key match 3. `x-goog-api-key` — extract key, attempt access key match 4. Anonymous fallback — if none of the above matched a valid access key **and** an `anonymous = true`, `enabled = true` access key is configured, the request matches as the anonymous access key 5. If all of the above fail — return HTTP 401 **Key behavior**: a request carrying an **invalid** key falls through to the anonymous fallback. This means when an anonymous access key is enabled, requests with an invalid key and requests with no key at all are treated identically — both resolve as the anonymous access key. Requests with a **valid** key match the correct access key and never reach the anonymous fallback. ### Enabled and Anonymous Access Keys Each access key has an `enabled` field (default: `true`). When `enabled = false`, the access key's API key is rejected with HTTP 401 — the access key entry remains in config but is effectively deactivated. Console → Access Keys → New → Name `former-employee` / API Key `sk-deactivated-key` / Enabled off. Protoflux supports a single **anonymous access key** (`anonymous = true`) that matches requests without an `Authorization` header. This is useful for public-facing APIs or development. Only one anonymous access key is allowed per config. Console → Access Keys → New → Name `public` / Anonymous on / API Key (blank) / Groups `public-access`. When an access key belongs to a group, only models listed in that group's `models` array are accessible. Requests for other models receive HTTP 403. The `list_models` endpoint is also filtered accordingly. Access keys without a group have no model access. Example: - Console → Access Keys → New → Name `alice` / API Key `sk-client-key-1` / Groups `qwen-only`. - Console → Access Keys → New → Name `bob` / API Key `sk-client-key-2` / Groups `full-access`. - Console → Access Keys → Key Groups tab → New → Name `qwen-only` / Models `qwen3.6-plus`, `qwen3.6-plus-lb`. - Console → Access Keys → Key Groups tab → New → Name `full-access` / Models `*`. ## Supported Protocols | Config Key | Description | |-----------|-------------| | `openai` | OpenAI Chat Completions API | | `openai_response` | OpenAI Response API | | `openai_images` | OpenAI Images API (generations & edits) | | `openai_embeddings` | OpenAI Embeddings API | | `openai_audio` | OpenAI Audio API (transcriptions, translations, speech) | | `openai_rerank` | OpenAI Rerank API (/v1/rerank, /v2/rerank; Jina/Cohere/vLLM) | | `anthropic` | Anthropic Messages API | | `google` | Google Gemini API | | `aws_converse` | AWS Bedrock Converse API | | `dashscope` | Alibaba Cloud DashScope (text, multimodal, image, video) | ## Credential Sources All credentials are stored in the database and managed via the Console API. Credentials are treated as source of truth and exposed as read-only in the Console UI. ## Per-Model Base URL Override By default, the upstream's `base_url` is used for all requests routed through that upstream. You can override this at the model level by setting `base_url` on a model entry: Console → Models → New model `custom-model` → upstream `openai` / Upstream Model ID `qwen3.6-plus` / base_url `https://custom-proxy.example.com/v1`. When `base_url` is set on a model, requests for that model are sent to the specified URL instead of the upstream's `base_url`. Credentials (API keys) are still sourced from the upstream — only the endpoint changes. This is useful for routing specific models through dedicated proxies, vLLM instances, or custom endpoints while sharing the same upstream credential pool. ## Model Aliases Each model can define alternative names via the `aliases` field. Aliases resolve to the same model configuration and are registered in O(1) alongside the canonical name. Console → Models → New model `kimi-2.6-20250514` → upstream `anthropic` / Upstream Model ID `kimi-2.6-20250514` / Aliases `kimi-2.6-0711`, `kimi-2.6`. A client sending `model: "kimi-2.6-0711"` will be routed to the same upstream as `model: "kimi-2.6-20250514"`. This is useful for backwards compatibility when renaming models or accepting provider-specific naming conventions. ### How Aliases Work Internally Aliases are **only** used at the gateway entry point for route resolution and access control checks. After the route is resolved, all internal operations use the **canonical `name`** (from the model's `name` field): - **Credential filtering** (`can_serve_model()`) — uses canonical name - **Stats recording** (SQLite/PostgreSQL) — uses canonical name - **Tracing logs** — the `model` field in log output uses the canonical name - **Web console** — displays the canonical name The client alias is preserved only in `client_model` for debugging purposes (e.g. upstream request debug logging). This ensures consistency across all logging, monitoring, and stats data — searching logs by alias will not work. **Note**: Access key group ACLs match against the **canonical `name` only**. If a model's canonical name is `"kimi-2.6-20250514"` with alias `"kimi-2.6-0711"`, the group must list `"kimi-2.6-20250514"` — aliases are not resolved for access control. ### Hiding the Canonical Name When you want to **hide the real model name** from clients, set `hide_name = true`: Console → Models → New model `qwen3.6-plus` → upstream `openai` / Upstream Model ID `qwen-plus` / Aliases `kimi-k2-0711` / Hide Canonical Name on. With `hide_name = true`: - Clients **cannot** use `"qwen3.6-plus"` to access the model directly — requests will get `404 Model Not Found` - Clients **can only** use the alias `"kimi-k2-0711"` - **LoadBalancer entries can reference `"qwen3.6-plus"`** — hidden models can be used in load balancing - The homepage and `GET /v1/models` endpoint list only `"kimi-k2-0711"` (does not include hidden model's canonical name) - **All internal operations still use `"qwen3.6-plus"`** — statistics, logging, and credential filtering are unaffected This is useful for: - White-labeling: expose a model under a brand-specific name while keeping the real upstream model ID internal - Migration: expose a stable public alias while changing the underlying model name - Multi-tenant isolation: different tenants see different model names even when routing to the same upstream - LoadBalancer-only models: hide underlying model names, expose only through LB names to clients **No aliases required**: `hide_name = true` does not require `aliases` to be set. The model is invisible to external APIs but can still be referenced by LoadBalancer entries. ## Model Name Resolution Priority When a request is processed, the model identifier flows through several resolution stages. Understanding the priority order is important for debugging and configuration: ### 1. Client-facing Name (Entry Point) The `model` field from the client request body. This can be either the canonical `name` or any configured `alias`. Used **only** for: - Route resolution (`RouteResolver::resolve()`) - Access control checks (`check_user_model_access()`) ### 2. Canonical Model Name (Internal) The model's `name` from config. After route resolution, this becomes the authoritative model identifier for all internal operations: - Credential filtering (`can_serve_model()`) - Stats recording and monitoring - All internal logging ### 3. Upstream Model ID (Sent to Provider) The model identifier sent in the upstream request body. Resolved with the following priority: | Priority | Source | Description | |----------|--------|-------------| | 1 (highest) | `upstream_inference_profile` | Optional override (e.g. AWS Bedrock ARN). Takes precedence over everything. | | 2 | `upstream_model_id` | Explicit model ID override in a model entry. | | 3 (lowest) | `name` | Canonical model `name` — used as the default fallback. | Example: Console → Models → New model `glm-5` → upstream `aws-global` / Upstream Model ID `zai.glm-5` / upstream_inference_profile `arn:aws:.../abcd1234` *(field not in the Console UI field map — verify label)*. In this example: - Client sends `model: "glm-5"` (or an alias) - Gateway internally tracks `"glm-5"` (canonical) - Upstream receives `"arn:aws:.../abcd1234"` as the model ID - If inference profile were removed, upstream would receive `"zai.glm-5"` - If both were removed, upstream would receive `"glm-5"` For AWS Bedrock Converse (`aws_converse` protocol), `count_tokens` still uses `upstream_model_id` while `converse` / `converse_stream` use the inference profile ARN. For all other protocols, the inference profile is used as the model ID in every upstream request. ## Model Configuration Fields Each model entry supports the following fields: | Field | Type | Default | Description | |-------|------|---------|-------------| | `name` | string | (required) | Canonical model identifier (used internally for stats, logging, routing) | | `upstream` | string | (required) | Upstream name to route through | | `upstream_model_id` | string | falls back to `name` | Model ID sent in upstream request body | | `upstream_inference_profile` | string | none | Highest-priority override for upstream model ID (e.g. AWS Bedrock ARN) | | `base_url` | string | upstream's base_url | Override endpoint URL for this model | | `aliases` | string[] | `[]` | Alternative names clients can use | | `protocol` | string | upstream's protocol | Override the request/response format | | `enabled` | bool | `true` | Whether this model is active. Disabled models are excluded from routing and return 404. | | `hide_name` | bool | `false` | Exclude canonical `name` from client access and model listing; aliases or LoadBalancer work | | `direct_path` | bool | `false` | Use `base_url` as-is without appending any path | | `kind` | string | none | DashScope model type; selects the endpoint path by type (e.g. `dashscope-text-embedding`), no `direct_path` needed. See [DashScope API](#dashscope-api) | | `request_payload` | array | `[]` | Model-level payload rules (override upstream rules) | | `request_transform_before` | script | none | Request transform script run before protocol translation | | `request_transform_after` | script | none | Request transform script run after protocol translation | | `response_transform` | script | none | Script to transform response body | | `script_error_mode` | string | `log-and-continue` | How to handle script errors | | `count_token` | string | upstream's mode | Token counting mode override | ### Token Counting Modes The `count_token` field controls how the gateway estimates or counts tokens for rate limiting and the `/v1/messages/count_tokens` endpoint: | Mode | Description | |------|-------------| | `upstream-api` | Forward to the upstream's native token counting endpoint (Anthropic, Gemini, AWS Bedrock only) | | `tiktoken` | Local estimation using the `tiktoken-rs` crate (OpenAI BPE tokenizer) | When `count_token` is not set, the gateway inherits the upstream's default mode. If the upstream does not support native token counting, it falls back to `tiktoken`. **Local tiktoken estimation** applies model-family correction factors to compensate for vocabulary differences between OpenAI's tokenizer and other providers. For example, Qwen models use a correction factor of 1.15, meaning the raw tiktoken count is multiplied by 1.15 to produce the adjusted estimate. These factors are automatically refined at runtime using exponential moving average (EMA) calibration based on actual token usage reported by upstream APIs. See the [Architecture documentation](architecture.md#token-counting) for details on correction factors and the EMA calibration algorithm. ### Model Metadata System Protoflux uses a three-layer Model Metadata architecture to manage model capability metadata (tokenizer family, correction factors, capability flags, etc.): 1. **Static Catalog** (`meta.toml`): embedded at compile time, defines default metadata for model families 2. **Database Overrides** (`config_settings` table): managed via Console API, overrides specific fields from the static catalog at runtime 3. **In-Memory Registry** (`Arc>`): field-level overlay merge with hot-reload support The static catalog includes default configurations for the following model families: - **OpenAI GPT new-generation** (`gpt-5*`, `gpt-4.1*`, `o1`, `o3`, etc.): `tokenizer_family = "gpt"`, `tiktoken_encoding = "o200k_base"` - **OpenAI GPT legacy** (`gpt-4*`, `gpt-3.5*`): `tokenizer_family = "gpt"`, `tiktoken_encoding = "cl100k_base"` - **Anthropic** (`claude*`): `tokenizer_family = "claude"`, `token_correction_factor = 0.95` - **Qwen** (`qwen*`, `qwq*`): `tokenizer_family = "qwen"`, `token_correction_factor = 1.15`, `disable_thinking_by_default = true` - **DeepSeek** (`deepseek*`): `tokenizer_family = "deepseek"`, `token_correction_factor = 1.10` - **Google Gemini** (`gemini*`): `tokenizer_family = "gemini"` - **GLM / Kimi / MiniMax**: respective tokenizer families and correction factors **Overlay Merge Mechanism**: When multiple entries match the same model, they are merged by `priority` from low to high. Higher-priority entries' `Some` fields override lower-priority ones, while `None` fields preserve the lower layer. Example: ``` qwen-vl-max matches: - qwen* (priority=5): tokenizer_family=Qwen, factor=1.15, disable_thinking=true - qwen*-vl* (priority=10): disable_thinking=false (overrides) Final result: {Qwen, 1.15, disable_thinking=false} ``` **Protocol Fallback**: When a model doesn't match any real entry (only hits the `*` fallback), the tokenizer family is inferred from the request protocol: - `DashScope` → `Qwen` - `Anthropic` → `Claude` - `Google` → `Gemini` - Other → `Unknown` (uses default correction factor 1.10) This allows non-standard model names proxied through DashScope to automatically receive the correct tokenizer configuration without manually adding metadata entries. **Database Override Example**: Add tokenizer configuration for a custom model ```json PUT /console/api/settings/model-metadata [ { "patterns": ["my-custom-model*"], "priority": 50, "tokenizer_family": "gpt", "token_correction_factor": 1.05 } ] ``` See the [Console API documentation](console-api.md#model-metadata-management-db-override-layer) for endpoint details. ## Retry Policy When an upstream request fails with a retryable error (HTTP 5xx, connection timeout, read timeout), the gateway automatically retries with exponential backoff. Two independent limits control this behavior: | Field | Type | Default | Description | |-------|------|---------|-------------| | `max_upstream_retries` | u32 | 3 | Maximum number of retry attempts per request across all credentials | | `max_key_rotations` | u32 | 3 | Maximum number of distinct upstream keys to rotate through. `0` means unlimited | ### How it works 1. A request fails with a retryable error. 2. The gateway picks the next available credential (if the upstream has multiple keys) and waits for an exponentially increasing delay (`500ms × 2^attempt`, capped at 5 minutes). 3. It retries until either `max_upstream_retries` or `max_key_rotations` is exhausted. 4. If `max_key_rotations` is reached first, the gateway returns the last error to the client even if `max_upstream_retries` has not been hit. ### Configuration examples **Default behavior** (3 retries, unlimited key rotations): ```toml [retry] max_upstream_retries = 3 max_key_rotations = 0 ``` **Conservative** (only retry twice, try at most 2 different keys): ```toml [retry] max_upstream_retries = 2 max_key_rotations = 2 ``` **Disable retries entirely**: ```toml [retry] max_upstream_retries = 0 ``` ### Use cases - **High-availability upstreams with multiple keys**: Keep `max_key_rotations = 0` so every key gets a chance. A single bad key will not exhaust the retry budget because duplicates are deduplicated. - **Upstream with many keys but sensitive latency**: Cap `max_key_rotations` to avoid spending too much time rotating through a large key pool. - **Non-idempotent operations** (e.g., image generation): Consider setting `max_upstream_retries = 0` to avoid duplicate charges or side effects. ### Streaming caveat Retries on streaming responses are only attempted **before the first SSE chunk is sent**. Once the client starts receiving data, mid-stream retry is not supported because already-sent chunks cannot be rolled back. ## DNS Resolver Health (DoH/DoT rebuild) Each upstream's DoH/DoT resolver (configured via that upstream's `dns_servers`) is rebuilt when its `lookup_ip` crosses a sustained failure threshold — a flapping or rotated Anycast IP set is re-resolved to a fresh set, probed, and hot-swapped in place. See the "Upstream DNS Resolver" section of [Architecture](architecture.md) for the full design. ### How it works The resolver increments a failure counter on every `lookup_ip` failure (not masked by the stale-cache fallback) and resets it on every success. `ResolverHealthMonitor` watches the counter per resolver key and, past the threshold and outside cooldown, singleflights a `spawn_blocking` rebuild that re-resolves the DoH/DoT hostname, probes it, and overwrites the cached resolver in place (no MISS window; the old resolver keeps serving in-flight requests until replaced). A rebuild that fails validation keeps the old resolver and enters cooldown. ### Configuration fields ```toml [dns_health] rebuild_threshold = 5 # consecutive lookup_ip failures before rebuild rebuild_cooldown_secs = 120 # min seconds between rebuild attempts per resolver rebuild_timeout_secs = 30 # per-attempt rebuild budget (re-resolve + probe) ``` | Field | Type | Default | Description | |-------|------|---------|-------------| | `rebuild_threshold` | u32 | `5` | Consecutive `lookup_ip` failures before a resolver rebuild is triggered | | `rebuild_cooldown_secs` | u64 | `120` | Minimum seconds between rebuild attempts for the same resolver key. The DoH server hostname's own DNS TTL makes more frequent rebuilds pointless | | `rebuild_timeout_secs` | u64 | `30` | Per-attempt wall-clock budget for a rebuild (re-resolve + probe). On timeout the singleflight guard is released and cooldown entered so self-heal is not pinned by a hung system resolver | These fields are read at construction and cloned into the resolver health monitor; like most infrastructure parameters, **changes require a restart** (the value is not hot-reloaded into the live monitor). ## Configuration Sources Protoflux loads infrastructure configuration from the `--config` TOML file. `${VAR}` references in TOML values are expanded from the process environment at startup, with automatic type coercion to the target field type (see [Environment Variables](#environment-variables)). There is **no** prefixed env-var overlay — only `${VAR}` references that literally appear in the TOML file are expanded; environment variables that the file does not reference have no effect. Additionally, **business configuration** (access_keys, upstreams, models, access_key_groups) is loaded from the database (SQLite or PostgreSQL). In SQLite mode, a polling loop detects changes every 30 seconds for hot-reload. In PostgreSQL mode, multiple gateway instances share config changes via the same polling mechanism. ## Environment Variables ### `${VAR}` In-File Expansion Write `${VAR}` references directly in TOML string values. The gateway replaces them with environment variable values at startup. ```toml # server.docker.toml [server] port = "${PORT}" storage_mode = "${STORAGE_MODE}" ``` **Rules:** - Must use braces: `${VAR}` — bare `$VAR` is intentionally **not** expanded to avoid conflicts with bcrypt hashes (`$2y$12$...`) - Variable names are arbitrary — anything the TOML file references will be looked up - **Automatic type coercion**: when the entire value is a single `${VAR}` reference, the result is coerced to the target field type: - `""` (empty) → `null` → for `Option` fields like `secret_key`, `postgres_url` (disables the feature) - `"8080"` → integer → for `port`, `max_failures`, etc. - `"true"` → boolean → for `enabled`, `allow_remote`, etc. - `"50.0"` → float → for `max_request_size_mb`, etc. - anything else → string → for `host`, `storage_mode`, etc. This is the approach used by `server.docker.toml`, where every field references an environment variable. You can mount a custom TOML file that uses the same pattern for containerized deployments. ## Recommended Practices - Keep production secrets out of the repo. - Use separate upstreams for different operational policies. - Use structured `access_keys` and `access_key_groups` instead of legacy `api_keys` for any public or shared deployment. - Leave `web_console.allow_remote` set to `false` if the gateway is exposed to the internet, and access the console via SSH tunnel or reverse proxy with IP whitelist. ## Request-ID Correlation Every request that enters the gateway receives a unique `Protoflux-Request-ID` header in the format `-` (e.g. `20260424T083015Z-a1b2c3d4-...`). This ID is: - Injected into the request before handler execution - Mirrored into the response headers - Forwarded to upstream providers as an extra header - Embedded in all structured log entries via a `tracing` span Search your logs for `request_id="..."` to see the complete lifecycle of a single request. ## Body Logging Every request produces a **single unified `info`-level log event** containing the complete request lifecycle — client request, upstream request/response, and error details. Body capture happens at the outermost middleware layer, ensuring that even early failures (401 unauthorized, 429 rate limited) include the request body in the log. ```toml [logging] level = "info" # info is sufficient — body capture works at all levels max_body_size_mb = 25 # max MB per body (terminal + SSE/webconsole) persist_request_logs = true # persist to database log_retention_days = 7 # log retention (0 = disable auto-cleanup) # SSE streaming body disk spill (industrial-grade) stream_body_max_disk_mb = 256 # max disk usage for body segment files (MB) stream_body_batch_size_kb = 64 # writer batch size (KB) stream_body_linger_ms = 5 # max wait before flushing incomplete batch (ms) stream_body_channel_capacity_chunks = 2048 # mpsc channel capacity (chunk count, not bytes) ``` | Field | Description | |-------|-------------| | `max_body_size_mb` | Maximum megabytes per body capture across all logging — terminal + SSE/webconsole (default 25) | | `persist_request_logs` | Whether to persist request logs to the database (default true) | | `log_retention_days` | Days to retain logs in DB, 0 disables auto-cleanup (default 7) | | `stream_body_max_disk_mb` | Disk spill limit for SSE streaming bodies (default 256 MB), chunks dropped when exceeded | | `stream_body_batch_size_kb` | Writer task batch size (default 64 KB), flushed when full | | `stream_body_linger_ms` | Max wait for incomplete batch (default 5 ms) | | `stream_body_channel_capacity_chunks` | mpsc channel capacity in chunks (default 2048), chunks dropped when full (non-blocking) | The unified log event includes these fields for every request: - `request_headers` — client request headers (sensitive values redacted) - `request_body` — client request body (captured before any downstream middleware) - `upstream_url` — URL sent to the upstream provider - `upstream_request_headers` / `upstream_request_body` — translated request sent upstream - `upstream_response_status` / `upstream_response_headers` / `upstream_response_body` — raw upstream response - `error` — for 4xx/5xx responses, contains the response body text (e.g. `"Missing or invalid API key"`) **Web console logs** (SSE): When the console's Live log toggle is ON, ALL requests are broadcast to the web console — including 401/429 early failures, internal endpoints (`/v1/models`, `/healthz`), and console routes. Anonymous requests show `user: "anonymous"`. **Error detail extraction**: For non-streaming 4xx/5xx responses, the response body is consumed to extract the error message (e.g. `"Missing or invalid API key"`), then reconstructed so it is still sent to the client. Truncated to 512 bytes. For streaming responses, upstream body is logged after stream completion via a background task to avoid interrupting the stream. **Security warning**: Bodies may contain sensitive data (PII, secrets in prompts, generated content). Only enable when actively debugging, then disable. ## Prometheus Metrics Protoflux exposes Prometheus metrics at the `/metrics` endpoint for monitoring and alerting. This endpoint uses pull-based scraping, compatible with Prometheus server and other metrics collection systems. ### Authentication The `/metrics` endpoint requires Bearer token authentication and is **disabled by default** for security: | Configuration | Behavior | |---------------|----------| | `metrics_auth_token` not set or empty | Returns `403 Forbidden` with message "Metrics endpoint disabled" | | Request missing `Authorization: Bearer ` | Returns `401 Unauthorized` | | Request with incorrect token | Returns `401 Unauthorized` with `WWW-Authenticate: Bearer realm="metrics"` header | | Request with correct token | Returns metrics in Prometheus text exposition format | **Configuration example:** ```toml [server] metrics_auth_token = "your-secure-token-here" ``` **Scraping with Prometheus:** ```yaml # prometheus.yml scrape_configs: - job_name: 'protoflux' scrape_interval: 15s bearer_token: 'your-secure-token-here' static_configs: - targets: ['protoflux:7890'] ``` ### Available Metrics | Metric | Type | Labels | Description | |--------|------|--------|-------------| | `protoflux_requests_total` | Counter | `method`, `path`, `status` | Total HTTP requests processed | | `protoflux_request_duration_seconds` | Histogram | `method`, `path` | Request latency distribution | | `protoflux_requests_in_flight` | Gauge | `method` | Currently active requests | | `protoflux_upstream_errors_total` | Counter | `upstream`, `error_type` | Upstream provider failures | | `protoflux_tokens_total` | Counter | `direction`, `model` | Token usage (direction: prompt/completion) | | `protoflux_active_streams` | Gauge | — | Currently active SSE streams | | `protoflux_stream_semaphore_available` | Gauge | — | Available stream concurrency slots | **Example query (PromQL):** ```promql # Request rate by status code rate(protoflux_requests_total[5m]) # P95 latency histogram_quantile(0.95, rate(protoflux_request_duration_seconds_bucket[5m])) # Error rate sum(rate(protoflux_requests_total{status=~"5.."}[5m])) / sum(rate(protoflux_requests_total[5m])) ``` ## Health Checks Protoflux provides two health check endpoints for Kubernetes probes and load balancers: ### Liveness Probe (`/health`, `/healthz`) Always returns `200 OK`. Use this for Kubernetes `livenessProbe` to detect deadlocks or hung processes. ```yaml livenessProbe: httpGet: path: /healthz port: 7890 initialDelaySeconds: 10 periodSeconds: 10 ``` ### Readiness Probe (`/ready`) Returns `200 OK` when the gateway is ready to serve traffic, `503 Service Unavailable` when not ready. Checks (both instance-local): 1. **Drain mode off**: not in pre-stop drain 2. **Database health**: Storage backend (SQLite/PostgreSQL) is accessible A zero-upstream configuration deliberately does NOT fail readiness: `upstreams` is global business config shared across instances, and a transient empty state (config reload window, an operator clear) could otherwise trip every instance's readiness at once and drain the whole target group. An empty upstream resolves to `NoCredentials` → `503` at request time instead. Use this for Kubernetes `readinessProbe` to control traffic routing: ```yaml readinessProbe: httpGet: path: /ready port: 7890 initialDelaySeconds: 5 periodSeconds: 5 failureThreshold: 3 ``` **Response format:** ```json // 200 OK (ready: not draining and DB reachable) { "status": "ready", "checks": { "drain_mode": false, "database": true } } // 503 Service Unavailable (draining: pre-stop drain phase, DB probe skipped) { "status": "not_ready", "reason": "drain_mode", "message": "Pod is draining, removing from load balancer" } // 503 Service Unavailable (database unreachable) { "status": "not_ready", "checks": { "drain_mode": false, "database": false } } ``` ### Kubernetes Pre-Stop Drain When running in Kubernetes, configure `pre_stop_delay_secs` to ensure graceful shutdown without dropping in-flight requests: ```toml [server] pre_stop_delay_secs = 15 graceful_shutdown_timeout_secs = 30 ``` **Shutdown sequence:** 1. Kubernetes sends `SIGTERM` to the pod 2. Protoflux sets `/ready` to return `503` (drain mode) 3. Sleep for `pre_stop_delay_secs` (allows kube-proxy to remove pod from Service endpoints) 4. Stop accepting new connections 5. Wait up to `graceful_shutdown_timeout_secs` for in-flight requests to complete 6. Exit **Kubernetes deployment example:** ```yaml spec: terminationGracePeriodSeconds: 60 # pre_stop_delay + graceful_shutdown_timeout + buffer containers: - name: protoflux lifecycle: preStop: exec: command: ["/bin/sh", "-c", "sleep 1"] # Small delay to ensure SIGTERM is processed ``` ## Per-IP Rate Limiting In addition to per-user rate limiting (based on API key), Protoflux supports per-IP rate limiting as a defense against abuse and DDoS attacks. ### Configuration ```toml [server] ip_rate_limit_rpm = 120 # 120 requests per minute per IP trusted_proxies = "10.0.0.0/8" # Trust X-Forwarded-For from these CIDRs ``` ### How It Works - **Sliding window algorithm**: Each IP has a counter that resets after 60 seconds - **IP extraction**: - If `trusted_proxies` is configured and the request comes from a trusted proxy, the first IP in `X-Forwarded-For` is used - Otherwise, the direct connection IP is used - **Response**: When rate limited, returns `429 Too Many Requests` with `Retry-After` header - **Scope**: Applied before authentication, so it protects against brute-force attacks on API keys ### Use Cases | Scenario | Configuration | |----------|---------------| | Public API gateway | `ip_rate_limit_rpm = 60` to prevent abuse | | Behind Cloudflare/CDN | Set `trusted_proxies` to Cloudflare's IP ranges | | Internal service mesh | Disable (`ip_rate_limit_rpm` not set) or set high limit | | Multi-region deployment | Use consistent limits across all regions via Redis | **Note**: Per-IP rate limiting complements per-user rate limiting. Both can be enabled simultaneously. Per-user limits are more granular (per API key), while per-IP limits protect against key enumeration attacks. {% endraw %}