Local inference & self-hosted

Ollama, vLLM, llama.cpp server, LM Studio, SGLang, TGI — these local/self-hosted inference services are not client SDKs; they are upstreams. They run models and expose a (usually OpenAI-compatible) HTTP endpoint; the gateway configures that endpoint as an upstream, and your client side keeps using any SDK from Vendor SDKs or any framework, unchanged.

client (any SDK) ──> gateway (openai-compatible upstream) ──> self-hosted service (Ollama/vLLM/...)
      base_url → gateway            upstream → http://<host>:11434/v1

Generic wiring

1. Start the local service (one line each, below).
2. Console → Upstream services → New, protocol openai, base URL = the local service address, the rest as needed.
3. Create the model, configure the key group, issue an access key — identical to Connect an OpenAI upstream.
4. Client uses the OpenAI SDK with base_url → gateway and model → gateway model name.

Ollama

ollama serve                      # default: http://localhost:11434
ollama pull qwen3:8b

Gotcha: Ollama's native /api/chat is not OpenAI-compatible. When configuring the upstream in the gateway, the base URL must use the OpenAI-compatible path http://localhost:11434/v1 (with /v1), not http://localhost:11434.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:7890/v1", api_key="<your access key>")
resp = client.chat.completions.create(
    model="qwen3-8b",   # the model name configured in the gateway; upstream model ID is Ollama's qwen3:8b
    messages=[{"role": "user", "content": "Introduce yourself in one sentence"}],
)
print(resp.choices[0].message.content)

vLLM

vllm serve Qwen/Qwen3-8B --port 8000   # OpenAI-compatible endpoint at http://localhost:8000/v1

vLLM is natively OpenAI-compatible (/v1/chat/completions); upstream base URL = http://localhost:8000/v1. Its rerank / embedding endpoints (/v1/rerank, /v1/embeddings) map to the gateway's openai_rerank / openai_embeddings protocols — create a separate openai_rerank upstream + model for a vLLM rerank model.

llama.cpp server

llama-server -m model.gguf --port 8080   # OpenAI-compatible endpoint at http://localhost:8080/v1

Same as above; upstream base URL = http://localhost:8080/v1.

SGLang / TGI / LM Studio

• SGLang: python -m sglang.launch_server ... --port 30000, OpenAI-compatible endpoint at /v1.
• TGI (Hugging Face): text-generation-launcher, compatible endpoint at /v1 (some versions only /generate — use the passthrough protocol as a fallback).
• LM Studio: start the server in the GUI and enable "OpenAI-compatible", default http://localhost:1234/v1.

Wiring is identical: protocol openai, base URL = its /v1 endpoint. If a service only exposes a non-OpenAI-compatible native endpoint (a few TGI / private implementations), configure passthrough protocol to forward as-is, with the client going through /v3/{model}/{*rest} or /v1/services/* — see Endpoints.

Why run local models through the gateway

• One entry point: local and cloud models share one base_url; the client switches with zero code changes.
• Reuse capabilities: load balancing, access control, billing, observability all apply to local upstreams too.
• Offline / data compliance: models never leave the intranet; the gateway only routes and governs.

Next

• Full steps to configure an openai-compatible upstream: see Connect an OpenAI upstream.
• Load balancing and failover across local/cloud upstreams: see Multi-upstream load balancing.