> Raw Markdown twin (generated at build time from the source Markdown). Rendered page: https://docs.gatellm.io/en/sdk/local-inference · Doc index: https://docs.gatellm.io/en/llms.txt


# Local inference & self-hosted

Ollama, vLLM, llama.cpp server, LM Studio, SGLang, TGI — these **local/self-hosted inference services are not client SDKs; they are upstreams**. They run models and expose a (usually OpenAI-compatible) HTTP endpoint; the gateway configures that endpoint as an upstream, and your client side keeps using any SDK from [Vendor SDKs](/en/sdk/vendor-sdks.md) or any framework, unchanged.

```
client (any SDK) ──> gateway (openai-compatible upstream) ──> self-hosted service (Ollama/vLLM/...)
      base_url → gateway            upstream → http://<host>:11434/v1
```

## Generic wiring {#generic}

1. Start the local service (one line each, below).
2. Console → **Upstream services** → New, protocol `openai`, base URL = the local service address, the rest as needed.
3. Create the model, configure the key group, issue an access key — identical to [Connect an OpenAI upstream](/en/quickstart/openai-compatible.md).
4. Client uses the OpenAI SDK with `base_url` → gateway and `model` → gateway model name.

## Ollama {#ollama}

```bash
ollama serve                      # default: http://localhost:11434
ollama pull qwen3:8b
```

> **Gotcha: Ollama's native `/api/chat` is not OpenAI-compatible.** When configuring the upstream in the gateway, the base URL must use the OpenAI-compatible path `http://localhost:11434/v1` (with `/v1`), not `http://localhost:11434`.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:7890/v1", api_key="<your access key>")
resp = client.chat.completions.create(
    model="qwen3-8b",   # the model name configured in the gateway; upstream model ID is Ollama's qwen3:8b
    messages=[{"role": "user", "content": "Introduce yourself in one sentence"}],
)
print(resp.choices[0].message.content)
```

## vLLM {#vllm}

```bash
vllm serve Qwen/Qwen3-8B --port 8000   # OpenAI-compatible endpoint at http://localhost:8000/v1
```

vLLM is natively OpenAI-compatible (`/v1/chat/completions`); upstream base URL = `http://localhost:8000/v1`. Its rerank / embedding endpoints (`/v1/rerank`, `/v1/embeddings`) map to the gateway's `openai_rerank` / `openai_embeddings` protocols — create a separate `openai_rerank` upstream + model for a vLLM rerank model.

## llama.cpp server {#llamacpp}

```bash
llama-server -m model.gguf --port 8080   # OpenAI-compatible endpoint at http://localhost:8080/v1
```

Same as above; upstream base URL = `http://localhost:8080/v1`.

## SGLang / TGI / LM Studio {#others}

- **SGLang**: `python -m sglang.launch_server ... --port 30000`, OpenAI-compatible endpoint at `/v1`.
- **TGI (Hugging Face)**: `text-generation-launcher`, compatible endpoint at `/v1` (some versions only `/generate` — use the `passthrough` protocol as a fallback).
- **LM Studio**: start the server in the GUI and enable "OpenAI-compatible", default `http://localhost:1234/v1`.

Wiring is identical: protocol `openai`, base URL = its `/v1` endpoint. If a service only exposes a **non-OpenAI-compatible** native endpoint (a few TGI / private implementations), configure `passthrough` protocol to forward as-is, with the client going through `/v3/{model}/{*rest}` or `/v1/services/*` — see [Endpoints](/en/reference/endpoints.md).

## Why run local models through the gateway {#why}

- **One entry point**: local and cloud models share one base_url; the client switches with zero code changes.
- **Reuse capabilities**: load balancing, access control, billing, observability all apply to local upstreams too.
- **Offline / data compliance**: models never leave the intranet; the gateway only routes and governs.

## Next {#next}

- Full steps to configure an `openai`-compatible upstream: see [Connect an OpenAI upstream](/en/quickstart/openai-compatible.md).
- Load balancing and failover across local/cloud upstreams: see [Multi-upstream load balancing](/en/quickstart/multi-upstream-load-balancing.md).