Orchestration frameworks

Orchestration frameworks wrap another abstraction layer on top of the official SDKs. Their common thread through the gateway: point the framework's underlying chat model / LLM at the gateway's OpenAI-compatible endpoint. Each framework has a specific "how to change base_url / declare the model" entry point; this section gives a minimal example and the per-framework gotcha.

LangChain (Python / TypeScript)

pip install langchain-openai langchain-anthropic     # Python
npm i @langchain/openai @langchain/anthropic         # Node

Python

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:7890/v1",
    api_key="<your access key>",
    model="gpt-5.4-mini",
)
print(llm.invoke("Introduce yourself in one sentence").content)

TypeScript

import { ChatOpenAI } from "@langchain/openai";

const llm = new ChatOpenAI({
  configuration: { baseURL: "http://localhost:7890/v1" },
  apiKey: "<your access key>",
  model: "gpt-5.4-mini",
});

const resp = await llm.invoke("Introduce yourself in one sentence");
console.log(resp.content);

To call Claude, use ChatAnthropic (with anthropic_api_url at the root), or use init_chat_model to auto-select the provider by model name:

from langchain.chat_models import init_chat_model

llm = init_chat_model(
    "gpt-5.4-mini",                       # gateway model name
    model_provider="openai",
    base_url="http://localhost:7890/v1",  # passed through to the OpenAI provider
    api_key="<your access key>",
)

LangGraph

LangGraph's "model" is just a LangChain chat model — pass the llm above into your graph. The gateway is invisible to it (LangGraph only orchestrates the graph; network requests still go through the chat model's base_url):

from langgraph.graph import StateGraph, MessagesState

graph = StateGraph(MessagesState)
# ... define nodes that call the llm above ...

LlamaIndex (Python / TypeScript)

pip install llama-index-llms-openai-like        # Python
npm i @llamaindex/openai                         # Node

Gotcha: the OpenAI class validates the model name against an allowlist. LlamaIndex's OpenAI class only accepts families it recognizes (gpt-* etc.) and rejects custom gateway names. Use OpenAILike (Python) / declare the model info explicitly to bypass the allowlist:

from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="gpt-5.4-mini",
    api_base="http://localhost:7890/v1",
    api_key="<your access key>",
    is_chat_model=True,                 # declare it's a chat model
    is_function_calling_model=True,     # supports function calling
)
print(llm.complete("Introduce yourself in one sentence").text)

CrewAI (Python)

Gotcha: CrewAI is driven by LiteLLM under the hood. So base_url isn't a framework parameter — it's the environment variable OPENAI_API_BASE, and model needs the openai/ prefix:

pip install crewai

import os
os.environ["OPENAI_API_BASE"] = "http://localhost:7890/v1"
os.environ["OPENAI_API_KEY"] = "<your access key>"

from crewai import Agent, Task, Crew

agent = Agent(
    role="Assistant",
    goal="Answer user questions",
    backstory="A friendly assistant",
    llm="openai/gpt-5.4-mini",   # LiteLLM prefix + gateway model name
)

AutoGen / AG2 (Python)

Gotcha: unknown model names require explicit model_info. The classic ConversableAgent API errors on model names not in its default list — declare them explicitly.

Note: AutoGen has split into two lines — Microsoft's autogen-agentchat (autogen_agentchat, i.e. the Microsoft Agent Framework Python line) and the community fork AG2 (continuing the classic ConversableAgent API). The example below follows the classic API; the from autogen import ConversableAgent import path varies by version (AG2 1.x moved to from ag2 import …). This section is written per the official docs and not run in this environment:

from autogen import ConversableAgent

config_list = [{
    "model": "gpt-5.4-mini",
    "base_url": "http://localhost:7890/v1",
    "api_key": "<your access key>",
}]

agent = ConversableAgent(
    "assistant",
    llm_config={
        "config_list": config_list,
        "model_info": {   # explicit declaration, bypasses the model-name list check
            "gpt-5.4-mini": {"function_calling": True, "json_output": False},
        },
    },
)

DSPy (Python)

pip install dspy

import dspy

lm = dspy.LM("openai/gpt-5.4-mini", api_base="http://localhost:7890/v1", api_key="<your access key>")
dspy.configure(lm=lm)

qa = dspy.Predict("question -> answer")
print(qa(question="Introduce yourself in one sentence"))

This section is written per the official docs and not run in this environment; the LM constructor's api_base/api_key kwargs vary with the DSPy version.

Haystack (Python)

pip install haystack-ai

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

generator = OpenAIChatGenerator(
    model="gpt-5.4-mini",
    api_base_url="http://localhost:7890/v1",
    api_key="<your access key>",
)
resp = generator.run([ChatMessage.from_user("Introduce yourself in one sentence")])
print(resp["replies"][0].content)

This section is written per the official docs and not run in this environment.

Llama Stack

Llama Stack is Meta's Llama toolchain; its remote-inference provider can point at an OpenAI-compatible endpoint. Point its OPENAI_BASE_URL / OPENAI_API_KEY at the gateway to reach self-hosted or cloud Llama models; the wiring varies with its provider config version — see the Llama Stack docs (this section is not run in this environment).

Next

• Agent-specific frameworks (OpenAI Agents / Claude Agent / Pydantic AI…): see Agent SDKs.
• Underlying model capabilities (image/embeddings/rerank): see Vendor SDKs.