gatellm

Protoflux

Enterprise-grade AI Gateway built in Rust — protocol translation, credential management, and operational visibility in a single binary.

English · 简体中文

Configuration · Scripting Guide · Console API · MCP Gateway · Deployment

---

✨ Features

Feature	Description
🔐 User-level access control	Per-user API keys with group-based model ACLs, anonymous user support
🔄 Credential rotation & failover	Multiple upstream keys with automatic rotation and retry
📊 Operational visibility	Request logging, persistent stats, real-time SSE log stream, Web Console
📈 Prometheus metrics	/metrics endpoint with Bearer auth, request counters, latency histograms, circuit breaker state
🔭 OpenTelemetry tracing	OTLP gRPC export (feature-gated), W3C Trace Context propagation, 42 #[instrument] spans
📋 Audit logging	Config change audit trail (who/what/when) in SQLite/PostgreSQL, fire-and-forget writes
🌐 Protocol translation	OpenAI ↔ Anthropic ↔ Gemini ↔ AWS Bedrock ↔ DashScope ↔ Passthrough, transparent bridging
📝 Script transforms	Custom request/response manipulation via JavaScript scripts (Scripting Guide)
🚀 Deployment-friendly	TOML config + env vars, Docker-ready, single binary (zero external deps for single-instance; PostgreSQL/Redis optional for multi-instance)
🛡️ Security hardened	bcrypt password hashing, IP ban protection, API key masking, security headers, per-IP rate limiting
⚡ High performance	Rust + Tokio async runtime, no GC pauses
⚖️ Model-level load balancer	Weighted round-robin, sticky sessions, failure-based degradation, 429 rate-limit failover across upstreams
🗄️ Distributed state	Redis-backed sessions, rate limiting, log broadcast, and IP ban for multi-instance deployments
☸️ Kubernetes-ready	Liveness/readiness probes, graceful shutdown with pre-stop drain, configurable timeouts
🛠️ MCP Gateway	Built-in MCP (Model Context Protocol) proxy, aggregating external tool servers with ACL, context budgeting, and BM25 tool search

🌍 Supported Protocols

Protocol	Executor	Status
OpenAI Chat Completions	openai	✅ Production
OpenAI Response	openai_response	✅ Production
OpenAI Images	openai_images	✅ Production
OpenAI Embeddings	openai_embeddings	✅ Production
OpenAI Audio	openai_audio	✅ Production
Anthropic Messages	anthropic	✅ Production
Gemini Generative Language	google	✅ Production
AWS Bedrock Converse	aws_converse	✅ Production
AWS Bedrock InvokeModel	aws_invoke	✅ Production
Alibaba DashScope	dashscope	✅ Production
Passthrough (proxy)	passthrough	✅ Production

Endpoints outside this scope return 501 Not Implemented instead of placeholder data.

[!NOTE] Protoflux uses API Key authentication exclusively. OAuth is not supported — subscription services like OpenAI and Anthropic typically prohibit using accounts for API access, violating vendor terms.

🚀 Quick Start

Prerequisites

• Rust 1.80+ (rustup default stable)
• Your LLM API keys

1. Copy config

cp server.example.toml server.toml
# Edit server.toml for infrastructure settings (server, logging, cors, etc.)
# Business config (users, upstreams, models) is managed via Console after startup

2. Validate config

cargo run -- --config server.toml config-validate

3. Start the server

cargo run -- --config server.toml serve

4. Health check

cargo run -- health-check --url http://localhost:7890

💡 Config files support ${ENV_VAR} expansion. Bare $VAR is intentionally not expanded to avoid conflicts with bcrypt hashes.

🐳 Docker Quick Start

Protoflux supports two storage modes. SQLite for single-instance deployments (default) and PostgreSQL for multi-instance container deployments with shared state.

SQLite Mode (single instance)

Config and stats are stored in local files. Business config is persisted in the SQLite database. Ideal for development, single-node deployments, or when you don’t need multi-instance coordination.

docker pull ghcr.io/taichulab/protoflux:latest

docker run -d \
  --name protoflux \
  -p 7890:7890 \
  -v $(pwd)/server.toml:/etc/protoflux/server.toml:ro \
  -v protoflux-data:/var/lib/protoflux \
  ghcr.io/taichulab/protoflux:latest

SQLite mode requires mounting a data volume (/var/lib/protoflux) to persist the stats and config database across restarts. Configure endpoints and view stats via the Web Console at http://localhost:7890/console.

PostgreSQL Mode (multi-instance)

Config, stats, and business data are stored in PostgreSQL. Multiple gateway instances can share the same database for horizontal scaling and centralized config management via Console.

# Start PostgreSQL
docker run -d \
  --name protoflux-pg \
  -e POSTGRES_USER=protoflux \
  -e POSTGRES_PASSWORD=changeme \
  -e POSTGRES_DB=protoflux \
  -p 5432:5432 \
  postgres:17-alpine

# Start Protoflux (connects to PG)
docker run -d \
  --name protoflux \
  -p 7890:7890 \
  -e POSTGRES_URL="postgresql://protoflux:changeme@host.docker.internal:5432/protoflux" \
  -v $(pwd)/server.toml:/etc/protoflux/server.toml:ro \
  ghcr.io/taichulab/protoflux:latest

With the config TOML:

[server]
storage_mode = "postgresql"
postgres_url = "${POSTGRES_URL}"

PostgreSQL auto-creates stats tables via refinery migrations. Business config (users, upstreams, models) is managed through the Console and persisted in the database. Both SQLite and PostgreSQL modes support Console-based endpoint configuration and statistics viewing.

🛠️ For Developers

Build

# Debug build (fast compilation)
cargo build

# Release build (optimized binary, LTO enabled)
cargo build --release

# Build a specific crate
cargo build -p protoflux-core
cargo build -p protoflux-server

Test

Tests use inline #[cfg(test)] blocks by default. Extract to crates/<crate>/tests/ for large or integration tests.

# Run all workspace tests
cargo test --workspace

# Run tests for a specific crate
cargo test -p protoflux-core
cargo test -p protoflux-server

# Run a specific test
cargo test -p protoflux-core resolved_route_user_agent

Code Quality

# Format check
cargo fmt --all --check

# Config validation (always run before committing)
cargo run -- --config server.example.toml config-validate

Architecture

Protoflux is organized as a Cargo workspace with five crates:

Crate	Responsibility
protoflux-core	Domain models, config parsing, route resolution, auth, protocol translators, provider executors
protoflux-server	Axum HTTP server, middleware (auth, client auth, rate limit, access log, stats), console API, plugin runtime
protoflux-plugin	Plugin SPI trait definitions — the extension interface for custom auth, routes, middleware, UI overlays
protoflux-license	License verification, memory cap enforcement, license API (commercial builds)
protoflux-bin	Binary entry point — CLI parsing, config loading, startup lifecycle via AppBuilder

Request flow: Client → Auth middleware → Route resolver → Request transform → Protocol translator → Provider executor → Response transform → Protocol translator → Client

Key patterns:

• Protocol translation: Client and upstream can use different protocols (e.g., Anthropic → OpenAI). Translators convert payloads bidirectionally.
• Credential rotation: Each upstream holds multiple API keys; the gateway rotates them automatically and retries on failure.
• Scripting: Request/response bodies can be manipulated via JavaScript scripts for custom routing or payload manipulation.
• Plugin SPI: Extend the gateway with custom authentication, routes, middleware, console UI, and more via the Plugin Development Guide.

Console Frontend

The Web Console is a TypeScript single-page app in console/ using React, TailwindCSS, and DaisyUI. It is built with Vite, and the compiled output is embedded into the binary via rust-embed.

# Console is served at the same port as the gateway
# http://localhost:7890/console

Environment Variables

Variable	Description
LOG	Log level filter (e.g., info, debug, warn)
RUST_LOG	Fallback log filter (tracing-subscriber)
RUST_BACKTRACE	Set to 1 for full backtraces on panics

Config files support ${ENV_VAR} syntax for injecting secrets at runtime without modifying the TOML.

Contributing

1. Create a feature branch from main
2. Write tests inline (#[cfg(test)]) or in crates/<crate>/tests/ for integration tests
3. Run cargo fmt --all --check && cargo test --workspace before committing
4. Validate config: cargo run -- --config server.example.toml config-validate

📖 Documentation

• Architecture · 架构文档
• Configuration · 配置文档
• Plugin Development · 插件开发指南
• Deployment · 部署文档
• Console API · Console API 文档
• MCP Gateway · MCP 网关
• Testing · 测试文档
• ADR · Architecture Decision Records

Documentation language convention:

• Top-level docs & user-manual — authored in Chinese as the primary version; English and other languages are translations. The Chinese version is authoritative; translations must stay in sync. When adding or editing docs, write the Chinese version first, then translate — translations must not contain content that the Chinese version lacks.
• ADR (docs/adr/) — Chinese only. No English translation.
• Tasks (docs/tasks/) — Chinese only. Records business-related task milestones (migration scripts, release checklists, etc.) as historical snapshots. These files reflect the state at the time of writing and must not be used as architecture or business reference — consult the authoritative docs (architecture.zh-CN.md, configuration.zh-CN.md, etc.) for current design.

📁 Repository Layout

protoflux/
├── crates/
│   ├── protoflux-core/        # domain model, config, routing, auth, translators, executors, pipeline, script
│   ├── protoflux-server/     # Axum server, middleware, console API, stats, storage, plugin runtime
│   ├── protoflux-plugin/      # Plugin SPI trait definitions and context types
│   ├── protoflux-license/     # License verification, memory cap, license API
│   └── protoflux-bin/         # Binary entry point (CLI, config loading, AppBuilder)
├── e2e/                       # end-to-end integration tests
├── docs/                      # long-form documentation
│   ├── adr/                   # Architecture Decision Records (Chinese only)
│   └── tasks/                 # business task milestones — historical snapshots (Chinese only)
├── console/                   # Web Console UI (TypeScript + React + Vite)
├── server.example.toml        # infrastructure configuration example
└── server.docker.toml         # Docker deployment configuration with env vars

📄 License

AGPL-3.0-or-later