> Raw Markdown twin (generated at build time from the source Markdown). Rendered page: https://docs.gatellm.io/en/sdk/orchestration-frameworks · Doc index: https://docs.gatellm.io/en/llms.txt


# Orchestration frameworks

Orchestration frameworks wrap another abstraction layer on top of the official SDKs. Their common thread through the gateway: **point the framework's underlying chat model / LLM at the gateway's OpenAI-compatible endpoint**. Each framework has a specific "how to change base_url / declare the model" entry point; this section gives a minimal example and the per-framework gotcha.

## LangChain (Python / TypeScript) {#langchain}

```bash
pip install langchain-openai langchain-anthropic     # Python
npm i @langchain/openai @langchain/anthropic         # Node
```

::: code-group

```python [Python]
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:7890/v1",
    api_key="<your access key>",
    model="gpt-5.4-mini",
)
print(llm.invoke("Introduce yourself in one sentence").content)
```

```ts [TypeScript]
import { ChatOpenAI } from "@langchain/openai";

const llm = new ChatOpenAI({
  configuration: { baseURL: "http://localhost:7890/v1" },
  apiKey: "<your access key>",
  model: "gpt-5.4-mini",
});

const resp = await llm.invoke("Introduce yourself in one sentence");
console.log(resp.content);
```

:::

To call Claude, use `ChatAnthropic` (with `anthropic_api_url` at the root), or use `init_chat_model` to auto-select the provider by `model` name:

```python
from langchain.chat_models import init_chat_model

llm = init_chat_model(
    "gpt-5.4-mini",                       # gateway model name
    model_provider="openai",
    base_url="http://localhost:7890/v1",  # passed through to the OpenAI provider
    api_key="<your access key>",
)
```

## LangGraph {#langgraph}

LangGraph's "model" is just a LangChain chat model — pass the `llm` above into your graph. The gateway is **invisible to it** (LangGraph only orchestrates the graph; network requests still go through the chat model's base_url):

```python
from langgraph.graph import StateGraph, MessagesState

graph = StateGraph(MessagesState)
# ... define nodes that call the llm above ...
```

## LlamaIndex (Python / TypeScript) {#llamaindex}

```bash
pip install llama-index-llms-openai-like        # Python
npm i @llamaindex/openai                         # Node
```

> **Gotcha: the `OpenAI` class validates the model name against an allowlist.** LlamaIndex's `OpenAI` class only accepts families it recognizes (`gpt-*` etc.) and rejects custom gateway names. Use `OpenAILike` (Python) / declare the model info explicitly to bypass the allowlist:

```python
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="gpt-5.4-mini",
    api_base="http://localhost:7890/v1",
    api_key="<your access key>",
    is_chat_model=True,                 # declare it's a chat model
    is_function_calling_model=True,     # supports function calling
)
print(llm.complete("Introduce yourself in one sentence").text)
```

## CrewAI (Python) {#crewai}

> **Gotcha: CrewAI is driven by LiteLLM under the hood.** So base_url isn't a framework parameter — it's the environment variable `OPENAI_API_BASE`, and `model` needs the `openai/` prefix:

```bash
pip install crewai
```

```python
import os
os.environ["OPENAI_API_BASE"] = "http://localhost:7890/v1"
os.environ["OPENAI_API_KEY"] = "<your access key>"

from crewai import Agent, Task, Crew

agent = Agent(
    role="Assistant",
    goal="Answer user questions",
    backstory="A friendly assistant",
    llm="openai/gpt-5.4-mini",   # LiteLLM prefix + gateway model name
)
```

## AutoGen / AG2 (Python) {#autogen}

> **Gotcha: unknown model names require explicit `model_info`.** The classic `ConversableAgent` API errors on model names not in its default list — declare them explicitly.

> **Note**: AutoGen has split into two lines — Microsoft's `autogen-agentchat` (`autogen_agentchat`, i.e. the [Microsoft Agent Framework](/en/sdk/agent-sdks.md#microsoft-agent) Python line) and the community fork **AG2** (continuing the classic `ConversableAgent` API). The example below follows the classic API; the `from autogen import ConversableAgent` import path varies by version (AG2 1.x moved to `from ag2 import …`). This section is written per the official docs and not run in this environment:

```python
from autogen import ConversableAgent

config_list = [{
    "model": "gpt-5.4-mini",
    "base_url": "http://localhost:7890/v1",
    "api_key": "<your access key>",
}]

agent = ConversableAgent(
    "assistant",
    llm_config={
        "config_list": config_list,
        "model_info": {   # explicit declaration, bypasses the model-name list check
            "gpt-5.4-mini": {"function_calling": True, "json_output": False},
        },
    },
)
```

## DSPy (Python) {#dspy}

```bash
pip install dspy
```

```python
import dspy

lm = dspy.LM("openai/gpt-5.4-mini", api_base="http://localhost:7890/v1", api_key="<your access key>")
dspy.configure(lm=lm)

qa = dspy.Predict("question -> answer")
print(qa(question="Introduce yourself in one sentence"))
```

> This section is written per the official docs and not run in this environment; the `LM` constructor's `api_base`/`api_key` kwargs vary with the DSPy version.

## Haystack (Python) {#haystack}

```bash
pip install haystack-ai
```

```python
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

generator = OpenAIChatGenerator(
    model="gpt-5.4-mini",
    api_base_url="http://localhost:7890/v1",
    api_key="<your access key>",
)
resp = generator.run([ChatMessage.from_user("Introduce yourself in one sentence")])
print(resp["replies"][0].content)
```

> This section is written per the official docs and not run in this environment.

## Llama Stack {#llama-stack}

Llama Stack is Meta's Llama toolchain; its remote-inference provider can point at an OpenAI-compatible endpoint. Point its `OPENAI_BASE_URL` / `OPENAI_API_KEY` at the gateway to reach self-hosted or cloud Llama models; the wiring varies with its provider config version — see the [Llama Stack docs](https://llama-stack.readthedocs.io/) (this section is not run in this environment).

## Next {#next}

- Agent-specific frameworks (OpenAI Agents / Claude Agent / Pydantic AI…): see [Agent SDKs](/en/sdk/agent-sdks.md).
- Underlying model capabilities (image/embeddings/rerank): see [Vendor SDKs](/en/sdk/vendor-sdks.md).