# Deployment

[English](deployment.md) | [简体中文](deployment.zh-CN.md)

## Local Binary

```bash
cargo build --workspace --release
./target/release/protoflux --config /path/to/server.toml serve
```

## Docker

### Build Image

The default build targets the host's native architecture:

```bash
docker build -t protoflux .
```

#### OpenTelemetry Build

To include OpenTelemetry distributed tracing support, build with the `otel` feature flag:

```bash
docker build --build-arg BUILD_FEATURES="--features otel" -t protoflux:otel .
```

This compiles the OTLP gRPC exporter (via tonic) into the binary. Without the feature flag, OTel-related config fields (`otel_endpoint`, `otel_service_name`) are ignored. The feature adds ~15MB to the binary size due to the gRPC/protobuf dependency chain.

> **Note**: The default Dockerfile `cargo build` command does not include `--features otel`. To enable it, modify the `RUN cargo build` line in the Dockerfile to: `RUN cargo build --release --bin protoflux --features otel`

#### Cross-Architecture Builds

Use `--platform` to build for a different architecture (requires [buildx](https://docs.docker.com/build/architecture/)):

```bash
# x86_64 / amd64
docker buildx build --platform linux/amd64 -t protoflux:amd64 --load .

# ARM64 (Apple Silicon, AWS Graviton, Raspberry Pi)
docker buildx build --platform linux/arm64 -t protoflux:arm64 --load .

# Multi-arch image (pushes to registry, cannot use --load)
docker buildx build --platform linux/amd64,linux/arm64 -t registry.example.com/protoflux:latest --push .
```

The build process embeds the console frontend assets (`console/`) into the binary. The runtime image contains no source code.

### Quick Start

The image ships with `server.docker.toml`, where every config field references an environment variable via `${ENV_VAR}` syntax. All variables have sensible defaults.

```bash
# Minimal start (all defaults)
docker run --rm -p 7890:7890 protoflux
```

> **Note**: The Dockerfile sets `ENV PORT=7890`, matching the code default. Map host port as needed (`-p 7890:7890`).

### PostgreSQL Storage (Recommended)

```bash
docker run --rm -p 7890:7890 \
  -e POSTGRES_URL="postgresql://user:password@pg-host:5432/protoflux" \
  -e CONSOLE_ENABLED=true \
  -e CONSOLE_PASSWORD="your-secret-password" \
  protoflux
```

### Docker Compose

```yaml
version: "3.8"

services:
  gateway:
    image: protoflux:latest
    ports:
      - "7890:7890"
    environment:
      POSTGRES_URL: "postgresql://postgres:password@db:5432/protoflux"
      CONSOLE_ENABLED: true
      CONSOLE_PASSWORD: "my-console-password"
      LOG_LEVEL: info
    depends_on:
      - db

  db:
    image: postgres:17-alpine
    environment:
      POSTGRES_DB: protoflux
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - "5432:5432"

volumes:
  pgdata:
```

### Custom Config File

For advanced configuration (custom CORS rules, upstream providers), mount a config file:

```bash
docker run --rm -p 7890:7890 \
  -v $(pwd)/server.docker.toml:/etc/protoflux/server.toml \
  -e POSTGRES_URL="postgresql://..." \
  protoflux
```

### Environment Variables

> **Note**: The variables below work because `server.docker.toml` uses `${VAR}` syntax for every field. If you mount a custom TOML with hardcoded values, reference the corresponding env vars via `${VAR}` in that file instead — the gateway only expands `${VAR}` references that appear in the TOML; there is no prefixed env-var overlay. See [Configuration Sources](configuration.md#configuration-sources) for details.

All config fields can be overridden via environment variables:

#### Server

| Variable | Default | Description |
|----------|---------|-------------|
| `HOST` | `0.0.0.0` | Listen address |
| `PORT` | `7890` | Listen port (code default and Dockerfile default are both `7890`) |
| `MAX_REQUEST_SIZE_MB` | `50` | Max request body size (MB) |
| `WORKER_THREADS` | `4` | Tokio worker thread count |
| `UPSTREAM_IDLE_CONNECTIONS` | `32` | Idle connection pool size per upstream host |
| `UPSTREAM_USER_AGENT` | `Protoflux` | Upstream request User-Agent |
| `STORAGE_MODE` | `postgresql` | Storage backend: `sqlite` or `postgresql` (code default is `sqlite`; Dockerfile overrides to `postgresql`) |
| `POSTGRES_URL` | *(empty)* | PostgreSQL connection URL |
| `SQLITE_PATH` | `/var/lib/protoflux/protoflux.sqlite` | SQLite file path |
| `REDIS_URL` | *(empty)* | Redis connection URL (distributed state sharing) |
| `REDIS_KEY_PREFIX` | `protoflux:` | Redis key prefix (distinguish multiple clusters sharing one Redis) |
| `SCRIPT_MAX_OPERATIONS` | `2000` | Max JavaScript operations |
| `METRICS_AUTH_TOKEN` | *(empty)* | Bearer token for `/metrics` endpoint (empty = endpoint disabled, returns 403) |
| `REQUEST_TIMEOUT_SECS` | *(empty)* | Global request timeout in seconds (default 120, SSE streams exempt) |
| `GRACEFUL_SHUTDOWN_TIMEOUT_SECS` | *(empty)* | Max seconds to wait for in-flight requests during shutdown (default 30) |
| `PRE_STOP_DELAY_SECS` | *(empty)* | Seconds to sleep after SIGTERM before stopping (for K8s endpoint drain, default 15) |
| `IP_RATE_LIMIT_RPM` | *(empty)* | Per-IP rate limit (requests/minute, sliding window) |
| `TRUSTED_PROXIES` | *(empty)* | Comma-separated trusted proxy CIDRs for X-Forwarded-For parsing |

#### Streaming (SSE)

Environment variable overrides for the `[streaming]` section fields (see [Configuration Reference](configuration.md#streaming-configuration) for field semantics):

| Variable | Default | Description |
|----------|---------|-------------|
| `STREAMING_KEEPALIVE_SECONDS` | `15` | SSE keep-alive heartbeat interval (seconds); gateway sends `:keep-alive` comments at this interval to prevent proxy/browser timeouts |
| `STREAMING_BOOTSTRAP_RETRIES` | `1` | Retries before the first byte (only retries if the upstream fails before the first SSE chunk; no retries once the first chunk is sent) |
| `STREAMING_LOG_TIMEOUT_SECS` | `3600` | SSE stream-log background task timeout (seconds); releases zombie tasks when a stream terminates abnormally |
| `STREAM_IDLE_TIMEOUT_SECS` | `600` | Upstream idle timeout (seconds): the gateway closes the stream if the upstream sends no data for N consecutive seconds. **Covers both the first-byte wait and mid-stream pauses** (not just the first token). Set to 0 to disable. Must be less than `upstream_read_timeout_secs` so the gateway-level timeout fires first (code default is `300`; Dockerfile overrides to `600`) |
| `MAX_STREAM_DURATION_SECS` | `3600` | Maximum total duration (seconds) for a single streaming response; the stream is closed unconditionally after this elapses. Set to 0 to disable |

#### Memory

| Variable | Default | Description |
|----------|---------|-------------|
| `MEMORY_SOFT_LIMIT_MB` | `0` | Memory cap in MB (= Critical hard-reject line, ADR-005). `0` = auto-detect (cgroup / ECS Container Metadata / `/proc/meminfo` / `sysctl`, limit − headroom). Set manually to override auto-detection |
| `MEMORY_QUEUE_TIMEOUT_SECS` | `300` | Max seconds a request waits in the memory-pressure queue before 429 |
| `SPILL_WRITE_CONCURRENCY` | `4` | Concurrent spill file write slots (spawn_blocking, each ~2.3MB) |
| `MEMORY_CONSUME_MAX_CONCURRENT` | `100` | Drain rate at Warning/Pressure/Reclaim (concurrent bodies read back from spill) |
| `MEMORY_CONSUME_CRITICAL_MAX_CONCURRENT` | `2` | Drain rate at Critical (only a few bodies read back when memory exhausted) |
| `MEMORY_HOLD_SECS` | `5` | Hysteresis hold (seconds) — gate stays active N secs after RSS drops below Warning, anti-oscillation |
| `MEMORY_RATE_ESCALATE_MB` | `40` | Per-tick MB growth threshold — early level escalation when single-tick RSS growth exceeds this |
| `MEMORY_ADMISSION_ENABLED` | `true` | Enable feed-forward admission (ADR-005): per-request decision before the body resides in memory using real-time RSS + in-flight body budget + Content-Length; over the admit line, spill to queue, closing the Normal-passthrough blind spot between watchdog ticks. `false` = legacy feedback-only |
| `MEMORY_ADMISSION_THRESHOLD_PCT` | `70` | Feed-forward admit line = cap × pct / 100 (default aligns with Warning 70%) |
| `MEMORY_ADMISSION_FRESH_READ_PCT` | `70` | Threshold to re-read RSS synchronously when close to the admit line (RSS snapshot up to ~1s stale) |
| `MEMORY_FORCE_SPILL_BODY_MB` | `0` | Force-spill threshold in MB (`0` = disabled) — bodies above this always spill, bounding one-shot consume cost |
| `MEMORY_ADMISSION_HANDLER_PERMIT` | `30` | Worst-case memory per handler (MB, default 30). `handler_permits_max = budget / (permit × 1MB)`. Covers body + response + reasoning + serde overhead. Must be > 0 |

#### Web Console

| Variable | Default | Description |
|----------|---------|-------------|
| `CONSOLE_ENABLED` | `false` | Enable web console |
| `CONSOLE_PASSWORD` | *(empty)* | Console login password (required when `enabled=true`) |
| `CONSOLE_SECRET_KEY` | *(empty)* | Bearer token for direct console API access (leave empty to disable) |
| `CONSOLE_ALLOW_REMOTE` | `true` | Allow non-localhost IP access |
| `CONSOLE_MAX_FAILURES` | `5` | Max failed login attempts before IP ban |
| `CONSOLE_BAN_DURATION` | `300` | IP ban duration in seconds |
| `CONSOLE_IP_BAN_ENABLED` | `true` | Enable IP ban |
| `CONSOLE_SESSION_AUTO_RENEW` | `true` | Session auto-renew |

#### Logging

| Variable | Default | Description |
|----------|---------|-------------|
| `LOG_LEVEL` | `info` | Log level: `trace`/`debug`/`info`/`warn`/`error` |
| `LOG_FORMAT` | `json` | Log format: `json`/`compact`/`pretty` |
| `LOG_QUEUE_DIR` | *(empty)* | Log disk queue directory |
| `LOG_MAX_BODY_SIZE_MB` | `25` | Max body capture size (MB) |
| `LOG_PERSIST_REQUEST_LOGS` | `true` | Write request logs to DB (set `false` to disable DB persistence, logs still stream to Console via SSE) |
| `LOG_RETENTION_DAYS` | `7` | Auto-delete logs older than N days (PostgreSQL: DROP expired partitions; SQLite: DELETE — VACUUM omitted to avoid full-DB lock, run manually to shrink file). `0` = disabled (manage externally) |
| `OTEL_ENDPOINT` | *(empty)* | OpenTelemetry OTLP gRPC endpoint (requires `--features otel` build) |
| `OTEL_SERVICE_NAME` | `protoflux` | Service name for OpenTelemetry traces |

#### Streaming

| Variable | Default | Description |
|----------|---------|-------------|
| `STREAMING_KEEPALIVE_SECONDS` | `15` | SSE keepalive interval (seconds) |
| `STREAMING_BOOTSTRAP_RETRIES` | `1` | Upstream retries before first token |
| `STREAMING_LOG_TIMEOUT_SECS` | `3600` | Stream log zombie-task timeout (seconds) |

#### Rate Limiting

| Variable | Default | Description |
|----------|---------|-------------|
| `RATE_LIMIT_ENABLED` | `false` | Enable per-user rate limiting |
| `RATE_LIMIT_RPM` | `120` | Max requests per minute per user |
| `RATE_LIMIT_WINDOW_SECS` | `60` | Sliding window size (seconds) |

#### Retry

| Variable | Default | Description |
|----------|---------|-------------|
| `MAX_UPSTREAM_RETRIES` | `3` | Max upstream retry attempts |
| `MAX_KEY_ROTATIONS` | `0` | Max key rotations (0=unlimited) |

#### Routing

| Variable | Default | Description |
|----------|---------|-------------|
| `LB_AFFINITY_TTL_SECS` | `300` | LB entry sticky-binding TTL (seconds) |
| `LOAD_WINDOW_SECS` | `60` | Load calculation sliding window (seconds) |
| `KEY_BINDING_TTL_SECS` | `1800` | Upstream multi-key sticky-binding idle TTL (seconds, 0=permanent) |

### Automatic Type Coercion

When a TOML field's value is entirely a single `${VAR}` reference, the environment variable value is automatically coerced to the appropriate type (see [Environment Variables](configuration.md#path-1-var-in-file-expansion) for full details):

- `""` (empty) → `null` → for `Option<T>` fields like `secret_key`, `postgres_url` (disables the feature)
- `"7890"` → integer → for `port`, `max_failures`, etc.
- `"true"` → boolean → for `enabled`, `allow_remote`, etc.
- `"50.0"` → float → for `max_request_size_mb`, etc.
- anything else → string → for `host`, `storage_mode`, etc.

```toml
# server.docker.toml example
port = "${PORT}"           # ENV PORT=9000 → integer 9000
enabled = "${CONSOLE_ENABLED}"  # ENV ...=true → boolean true
```

## Persistent Data

### Storage Backends

Protoflux supports two storage backends for request statistics and configuration persistence:

| Backend | Use case | Connection |
|---------|----------|------------|
| SQLite (code default) | Single-instance deployments | File path via `sqlite_path` |
| PostgreSQL (Docker default) | Multi-instance container deployments | Connection URL via `postgres_url` |

**SQLite** (single-instance):
```toml
[server]
storage_mode = "sqlite"
sqlite_path = "/var/lib/protoflux/protoflux.sqlite"
```

Docker persistence for SQLite:
```bash
docker run --rm -p 7890:7890 \
  -e STORAGE_MODE=sqlite \
  -v protoflux-data:/var/lib/protoflux \
  protoflux
```

**PostgreSQL** (multi-instance):
```toml
[server]
storage_mode = "postgresql"
postgres_url = "postgresql://user:password@pg-host:5432/protoflux?sslmode=require"
```

TLS is controlled by the `sslmode` query parameter:
- `sslmode=disable` — no TLS (default, local development)
- `sslmode=require` — TLS without certificate verification
- `sslmode=verify-full` — TLS with full certificate verification

When using PostgreSQL, tables are auto-created via migrations. Multiple gateway instances writing to the same database aggregate stats automatically via `ON CONFLICT DO UPDATE`. Config changes are synchronized across instances via 30-second polling with exponential backoff on connection failures.

### Redis Distributed State (Multi-Instance)

For deployments running multiple gateway instances, Redis provides shared runtime state across instances:

| Component | Shared State | Effect |
|-----------|-------------|--------|
| Session | Login sessions | Users stay logged in across instances |
| Rate Limit | Request counters | Users can't bypass limits by rotating between instances |
| Log Broadcast | Log stream | Console SSE shows request logs from all instances |
| IP Ban | IP bans | An IP banned on instance A is also banned on instance B |

**When `redis_url` is not set, all state is in-memory with zero overhead.**

```toml
[server]
storage_mode = "postgresql"
postgres_url = "postgresql://user:password@pg-host:5432/protoflux"
# Redis URL format: redis://[:password@]host[:port][/db_number]
# Default db 0; use /1, /2, etc. to select a different database
redis_url = "redis://redis-host:6379/1"
redis_key_prefix = "protoflux:"
```

Redis connection failure gracefully falls back to in-memory mode and does not prevent gateway startup.

**Multi-instance Docker Compose example:**

```yaml
version: "3.8"

services:
  gateway-a:
    image: protoflux:latest
    ports:
      - "7890:7890"
    environment:
      POSTGRES_URL: "postgresql://postgres:password@db:5432/protoflux"
      REDIS_URL: "redis://redis:6379"
      CONSOLE_ENABLED: true
      CONSOLE_PASSWORD: "my-console-password"
    depends_on:
      - db
      - redis

  gateway-b:
    image: protoflux:latest
    ports:
      - "7891:7890"
    environment:
      POSTGRES_URL: "postgresql://postgres:password@db:5432/protoflux"
      REDIS_URL: "redis://redis:6379"
      CONSOLE_ENABLED: true
      CONSOLE_PASSWORD: "my-console-password"
    depends_on:
      - db
      - redis

  db:
    image: postgres:17-alpine
    environment:
      POSTGRES_DB: protoflux
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
      - pgdata:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    volumes:
      - redisdata:/data

volumes:
  pgdata:
  redisdata:
```

Both instances share the same PostgreSQL (business config + stats) and Redis (runtime state). You can log in to the Console on instance A and see instance B's request logs in the SSE stream.

#### Redis Data Structures

| Component | Redis Type | Key Pattern | Notes |
|-----------|-----------|-------------|-------|
| Session | String (JSON) | `{prefix}session:{token}` | 24h TTL auto-expiry, revoke_all via Set index |
| Session Index | Set | `{prefix}session_index` | Used by revoke_all to iterate all tokens |
| Rate Limit | Hash | `{prefix}ratelimit:{key}` | `{count, window_start}`, atomic Lua script |
| IP Fail Count | Hash | `{prefix}ip_fail:{ip}` | `{count, last_failure}`, resets after ban_duration |
| IP Ban List | Sorted Set | `{prefix}banned_ips` | score=expiry timestamp, cleaned via ZREMRANGEBYSCORE |

#### Unified Broadcast Architecture (Redis Pub/Sub)

Protoflux uses a generic `EventBus<T>` trait (`protoflux_core::broadcast`) to decouple event producers from transport implementations. Both event types share this trait, but **use different transports**:

| Event Type | Channel | Purpose |
|---|---|---|
| `LogEvent` | `{prefix}logs` | Request logs → Console SSE log stream |
| `McpNotification` | `{prefix}mcp_notifications` | Tool list changes → MCP SSE clients |

The two event types differ significantly in transport characteristics: `McpNotification` is a low-volume tool-list-change signal carried over an in-memory `tokio::sync::broadcast` channel + Redis PUBLISH; `LogEvent` is a high-throughput request log carried over a **disk-backed `FileLogQueue`** (persisted for DB drain + local SSE). The diagram below depicts the MCP single-channel fan-out; `LogEvent` uses a **dual-queue** variant in Redis mode (see [LogEvent anti-loop mechanism](#logevent-anti-loop-mechanism-dual-queue) below).

Both backends follow the same **Redis Pub/Sub + fan-out pattern**:

```
Instance A                        Redis                       Instance B
───────────────────    ┌──────────────────┐    ───────────────────
event ──► publish() ──► PUBLISH channel   │    event ──► publish()
                       │                  │
                  ┌────► SUBSCRIBE  ◄─────┘
                  │    │ protoflux:*      │
            ┌─────┘    └──────────────────┘    ┌─────┐
            ▼                                  ▼     ▼
      pubsub_loop() (background task)    SSE client1 SSE client2
            │
            ▼
      local broadcast::channel (fan-out)
            │
            ▼
      Local SSE clients
```

- **publish()** — sends event JSON via `PUBLISH` to the Redis Pub/Sub channel
- **publish_local()** — sends only to local subscribers without Redis PUBLISH (anti-loop mechanism for MCP notifications)
- **pubsub_loop()** — each instance spawns a background task with a **dedicated Redis connection** that `SUBSCRIBE`s and forwards messages to a local `tokio::sync::broadcast`
- **subscribe()** — SSE clients subscribe to the local broadcast (zero Redis overhead). A single Redis SUB connection serves all local clients
- **auto-reconnect** — if the Redis connection drops, the background task retries every 2s in a loop
- **stale detection** — 60s message timeout detects silently dropped TCP connections
- **fail-open** — when no Console is open (`subscriber_count() == 0`), request body capture is skipped to reduce overhead

**MCP anti-loop mechanism**: When Instance A adds an MCP server, it publishes `notifications/tools/list_changed` to Redis. Instance B's `pubsub_loop` receives this and triggers `rebuild_indexes(local_only=true)`, which rebuilds the local tool cache and notifies local SSE clients via `publish_local()` — without re-publishing back to Redis.

**LogEvent anti-loop mechanism (dual-queue)**: Unlike MCP's `publish_local()` anti-loop, `LogEvent` cuts the Redis pubsub loop by **splitting two disk queues** (`RedisLogBroadcast`):

| Queue | Written by | Consumer | Role |
|---|---|---|---|
| `drain_queue` | **Only the local instance's `publish()` directly** (remote pubsub events never written here) | Local DB drain task | DB persistence — the **only path** this instance's events are written to the DB (per-instance ownership, no N× duplicate DB writes) |
| `live_queue` | Local `publish()` + remote events received via Redis pubsub | Local SSE subscribers | Immediate SSE (no Redis round-trip wait) |

- **Memory mode (single-instance)**: A single `FileLogQueue` shared by DB drain + SSE — no loop problem, no split needed.
- **Redis mode (multi-instance)**: Split into `drain_queue` + `live_queue`. Remote events received via Redis pubsub are written **only to `live_queue`**, **never to `drain_queue`** — otherwise every instance would drain all N instances' events, producing N-1 empty transactions, cursor lag, and disk-full.
- **Serialize-once reuse**: `publish()` serializes the `LogEvent` once and reuses the same bytes for `drain_queue` + `live_queue` + Redis `PUBLISH` (three destinations, one serialization). When broadcasting cross-instance the body is no longer inlined into the event — a `Ref` body is materialized into the Redis hot cache (compressed) before publish, and the event carries only a `GlobalRef` pointer + preview (KB-sized), so the serialized bytes are decoupled from body size (see the architecture doc «Cross-instance body: tiered storage»). `drain_queue` is written first (DB durability priority); StoreFull propagates as backpressure. `live_queue` / Redis PUBLISH failures only warn, not fail (the DB row is already durable).

> **Note**: Pub/Sub channels are not Redis keys and won't appear in `KEYS`.
> To debug, use `redis-cli SUBSCRIBE {prefix}logs` or `redis-cli SUBSCRIBE {prefix}mcp_notifications` to see messages in real-time.
> The Console Logs page must be open first to activate subscription; otherwise the gateway skips body capture.

#### Managed Background Task Inventory

The `R{N}` label on the instance card in the Console overview = `InstanceTaskSummary.running` (count of managed tasks in `TaskState::Running`). The total managed task count depends on the storage mode:

| Task | spawn site | Memory | Redis |
|---|---|---|---|
| `config_reload` (unified hot-reload loop) | lifecycle.rs | ✅ | ✅ |
| `housekeeping` | app_builder.rs | ✅ | ✅ |
| `price_refresh` (Tier-2 remote pricing fetch + refresh) | app_builder.rs | ✅ | ✅ |
| `mcp_background_sync` (periodic dirty-flag index rebuild safety net) | app_builder.rs | ✅ | ✅ |
| `request_log_subscriber`¹ | request_log_subscriber.rs | ✅ | ✅ |
| `stats_flush`² | stats/mod.rs | ✅ | ✅ |
| `log_queue_flush_cleanup` | provider.rs | ✅ | — |
| `redis_drain_queue_flush_cleanup` | provider.rs | — | ✅ |
| `redis_live_queue_flush_cleanup` | provider.rs | — | ✅ |
| `redis_log_broadcast` (log pubsub) | broadcast.rs | — | ✅ |
| `redis_mcp_broadcast` (MCP pubsub) | mcp_broadcast.rs | — | ✅ |
| `redis_config_broadcast` (config reload signal pubsub) | config_broadcast.rs | — | ✅ |
| `redis_instance_broadcast` (instance lifecycle pubsub) | instance_broadcast.rs | — | ✅ |
| `instance_health_reporter` | provider.rs | — | ✅ |
| `instance_event_probe` (consumes instance broadcast → live registry) | provider.rs | — | ✅ |
| **Total** | | **R7** | **R14** |

¹ Spawned only when `request_log_repo: Some` (DB persistence enabled)  ² Spawned only when `stats_backend: Some` (stats enabled)

R14 − R7 = 7 differential tasks, all required for Redis cross-instance coordination: the dual-queue flush pair (drain + live, net +1 over Memory's single queue) + log/MCP/config/instance Pub/Sub broadcasts + instance_health_reporter + instance_event_probe (consumes the instance broadcast). If Redis mode shows R7, a Redis task has Stopped/Panicked — check the `task_infos` of `/console/api/observability` to locate the non-Running task.

### Request Log Retention

The `request_logs` table stores per-request audit data including captured bodies
(up to 4 text columns × `max_body_size_mb`). Without periodic cleanup, the table
grows unbounded:

| Traffic  | Avg body | Daily growth | 30-day size |
|----------|----------|-------------|-------------|
| 100 rps  | 10 KB    | ~80 GB      | ~2.4 TB     |
| 10 rps   | 1 KB     | ~800 MB     | ~24 GB      |
| 1 rps    | 100 B    | ~8 MB       | ~240 MB     |

#### Data model

`LogEvent` is the single data structure that flows through the entire request
log pipeline: SSE broadcast to the Console UI → disk queue (`FileLogQueue`) →
DB persistence. The same `LogEvent` shape is used for real-time streaming,
disk-buffered durability, and database storage — no intermediate conversions.
Variable/low-frequency fields (headers, load balancer trace, rate limit info,
cache markers) are packed into a single `extensions` JSONB column, so adding
a new field requires no schema change.

#### PostgreSQL: Daily Partitioning

The `request_logs` table is partitioned by day (`PARTITION BY RANGE (occurred_at)`).
Each day's data lives in its own partition — retention is managed by `DROP`ping
expired partitions, not `DELETE`ing rows. **All partition boundaries are UTC.**

```
request_logs (parent)
├── request_logs_2026_06_22   (1 day)
├── request_logs_2026_06_23   (1 day)
├── ...
└── request_logs_2026_06_29   (1 day)
```

**Why partitioning instead of DELETE:**

| | DELETE | DROP partition |
|---|---|---|
| Lock | Row-level locks, blocks concurrent writes | DDL, no row locks |
| Speed | Minutes to hours for millions of rows | Instant |
| Dead tuples | Leaves dead tuples, needs VACUUM | None |
| Disk reclaim | Only after VACUUM | Immediate |

**Automatic partition creation and retention** (no cron or manual steps needed):

The application automatically manages the partition lifecycle via the drain loop:

- **Creation**: on startup and every hour thereafter, `ensure_partitions()`
  ensures today + the next 3 days' partitions exist. Uses
  `CREATE TABLE IF NOT EXISTS` — safe for concurrent multi-instance deployments.
  If it fails (e.g. transient DB outage), the retry interval shortens to 60
  seconds until success, then returns to the normal 1-hour cadence.

- **Retention**: on the same hourly tick, `drop_expired()` removes partitions
  older than `LOG_RETENTION_DAYS` (default 7). Uses `DROP TABLE IF
  EXISTS` — instant DDL, no row locks, no VACUUM needed. Set to `0` to disable
  automatic retention and manage expiry externally.

Both only run when `LOG_PERSIST_REQUEST_LOGS=true`.

**Manual partition creation** (reference — normally automatic):

```sql
-- Create a specific partition (idempotent)
CREATE TABLE IF NOT EXISTS request_logs_2026_06_30
    PARTITION OF request_logs
    FOR VALUES FROM ('2026-06-30') TO ('2026-07-01');
```

**Dropping expired partitions** (e.g. keep 7 days):

```sql
-- List all partitions
SELECT inhrelid::regclass AS partition_name
FROM pg_inherits
WHERE inhparent = 'request_logs'::regclass
ORDER BY 1;

-- Drop partitions older than 7 days
DROP TABLE IF EXISTS request_logs_2026_06_22;
```

**External cron** (only needed when `LOG_RETENTION_DAYS=0`):

```bash
#!/bin/bash
# /etc/cron.d/protoflux-partition-manager
# Daily at 00:05 UTC — drop expired partitions
# (only needed when automatic retention is disabled)

RETENTION_DAYS=7
DB_NAME=protoflux

CUTOFF=$(date -u -d "-${RETENTION_DAYS} days" +%Y-%m-%d)
psql -d "$DB_NAME" -t -c \
  "SELECT inhrelid::regclass FROM pg_inherits WHERE inhparent = 'request_logs'::regclass" |
while read -r partition; do
  part_date=$(echo "$partition" | grep -oP '\d{4}_\d{2}_\d{2}' | tr '_' '-')
  if [ -n "$part_date" ] && [[ "$part_date" < "$CUTOFF" ]]; then
    psql -d "$DB_NAME" -c "DROP TABLE IF EXISTS ${partition};"
    echo "Dropped expired partition: ${partition}"
  fi
done
```

#### SQLite

SQLite does not support table partitioning. For single-instance deployments,
the `request_logs` table is a regular table. Cleanup is a manual operation:

```bash
# Manual cleanup (run as needed)
sqlite3 /path/to/protoflux.sqlite \
    "DELETE FROM request_logs WHERE occurred_at < datetime('now', '-7 days');" \
    "VACUUM;"
```

For production deployments with significant traffic, PostgreSQL is recommended.

#### Retention period guidelines

| Use case | Recommended retention |
|----------|----------------------|
| Debugging / incident response | 3–7 days |
| Usage analytics / billing | 30–90 days |
| Compliance / audit trail | 1–7 years (body columns excluded) |

For long-term retention, export rows to cold storage (S3/GCS) before dropping
partitions, excluding body columns to reduce cost:

```sql
COPY (
    SELECT request_id, occurred_at, model, upstream, status_code,
           latency_ms, input_tokens, output_tokens, access_key
    FROM request_logs
    WHERE occurred_at < now() - interval '7 days'
) TO '/tmp/request_logs_archive.csv' WITH CSV HEADER;
```

## Reverse Proxy

Protoflux does **not** support native TLS. A reverse proxy is required for HTTPS:

- TLS termination (Nginx, Caddy, Traefik, or cloud LB)
- network restrictions for the console
- rate limiting and request size controls

## Kubernetes Deployment

Protoflux is designed for production Kubernetes deployments with full support for health probes, graceful shutdown, and observability.

### Deployment Manifest

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: protoflux
  labels:
    app: protoflux
spec:
  replicas: 2
  selector:
    matchLabels:
      app: protoflux
  template:
    metadata:
      labels:
        app: protoflux
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "7890"
        prometheus.io/path: "/metrics"
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: protoflux
          image: protoflux:latest
          ports:
            - name: http
              containerPort: 7890
          env:
            # Required
            - name: ENCRYPTION_KEY
              valueFrom:
                secretKeyRef:
                  name: protoflux-secrets
                  key: encryption-key
            - name: POSTGRES_URL
              valueFrom:
                secretKeyRef:
                  name: protoflux-secrets
                  key: postgres-url

            # Console
            - name: CONSOLE_ENABLED
              value: "true"
            - name: CONSOLE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: protoflux-secrets
                  key: console-password

            # Observability
            - name: METRICS_AUTH_TOKEN
              valueFrom:
                secretKeyRef:
                  name: protoflux-secrets
                  key: metrics-token
            - name: LOG_FORMAT
              value: "json"

            # Resilience
            - name: REQUEST_TIMEOUT_SECS
              value: "120"
            - name: GRACEFUL_SHUTDOWN_TIMEOUT_SECS
              value: "30"
            - name: PRE_STOP_DELAY_SECS
              value: "15"
            - name: IP_RATE_LIMIT_RPM
              value: "120"

          # Liveness probe: is the process alive?
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3

          # Readiness probe: can it serve traffic?
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3

          # Startup probe: give it time to initialize
          startupProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 12

          resources:
            requests:
              cpu: "500m"
              memory: "256Mi"
            limits:
              cpu: "2"
              memory: "1Gi"
```

### Service

```yaml
apiVersion: v1
kind: Service
metadata:
  name: protoflux
spec:
  selector:
    app: protoflux
  ports:
    - name: http
      port: 7890
      targetPort: http
  type: ClusterIP
```

### Health Probes

Protoflux provides two health endpoints:

| Endpoint | Purpose | Response |
|----------|---------|----------|
| `/healthz`, `/health` | Liveness — is the process alive? | Always `200 OK` |
| `/ready` | Readiness — can it serve traffic? | `200` when DB is reachable and the pod is not draining; `503` otherwise |

**Liveness probe** detects deadlocks or hung processes. Use `/healthz` with a generous `failureThreshold` — restarting a pod is expensive (connection drain, rescheduling).

**Readiness probe** controls whether the pod receives traffic. Use `/ready` — when it returns `503`, Kubernetes removes the pod from the Service endpoints automatically. The probe checks:
1. Drain mode is off (set during pre-stop drain)
2. Storage backend (SQLite/PostgreSQL) is reachable

A zero-upstream configuration deliberately does NOT fail readiness: `upstreams` is global business config shared across instances, and a transient empty state (config reload window, an operator clear) could otherwise trip every instance's readiness at once and drain the whole target group. An empty upstream resolves to `NoCredentials` → `503` at request time instead, where the error belongs.

### Graceful Shutdown & Pre-Stop Drain

When Kubernetes terminates a pod (scaling down, rolling update, node drain), the shutdown sequence is:

```
SIGTERM received
  → /ready returns 503 (drain mode activated)
  → sleep pre_stop_delay_secs (15s default)
     ↳ kube-proxy removes pod from Service endpoints
     ↳ in-flight requests continue processing
  → stop accepting new connections
  → wait graceful_shutdown_timeout_secs (30s default)
     ↳ in-flight requests complete naturally
  → exit
```

**Why the pre-stop delay?** Kubernetes sends `SIGTERM` and updates iptables/endpoints concurrently. There's a brief window where new requests may still be routed to the terminating pod. The `pre_stop_delay_secs` (default 15s) ensures:
1. The pod returns `503` on `/ready` immediately
2. Kubernetes has time to propagate the endpoint removal
3. In-flight requests finish before the server stops

Set `terminationGracePeriodSeconds` ≥ `pre_stop_delay_secs` + `graceful_shutdown_timeout_secs` + buffer (recommended: 60s).

### Prometheus Monitoring

Add Prometheus scraping via pod annotations or ServiceMonitor:

**Pod annotations** (used above):
```yaml
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "7890"
  prometheus.io/path: "/metrics"
```

**ServiceMonitor** (Prometheus Operator):
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: protoflux
spec:
  selector:
    matchLabels:
      app: protoflux
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
      bearerTokenSecret:
        name: protoflux-secrets
        key: metrics-token
```

> **Security**: Set `METRICS_AUTH_TOKEN` to require Bearer token authentication on `/metrics`. Without it, the endpoint returns `403 Forbidden`.

## Operational Baseline

- enable `api_keys` for the data plane
- set `web_console.allow_remote = false` when exposing the gateway to the internet
- do not expose `/console` publicly without network controls