Multi-model code review with GitHub Actions

Single-model code review has blind spots; both false positives and false negatives are hard to eliminate. Have 2–3 models independently review the same PR in parallel, then merge and dedupe and annotate confidence — the accuracy is clearly higher than a single model. GitHub's claude-code-action only speaks the Anthropic protocol — relying on the gateway's protocol interop, non-Anthropic models like qwen3.8-max and kimi-k3 can also enter the review queue; a single gateway access key uniformly carries all review traffic, with centralized credentials and measurable cost.

See Enhance model capability scenario three for the scenario background and "why go through the gateway".

The experience after setup: every time a PR is opened or updated, the review job runs automatically and models post review comments under the PR (findings + a machine-parseable TODO fix list); multi-model mode additionally carries confidence annotations. Developers can also reply @claude in the comment section to trigger a context-aware re-review.

How it works:

PR opened/updated (or an @claude comment)
    → GitHub Actions triggers the review job
    → calls the review models through the gateway (ANTHROPIC_BASE_URL points to the gateway, which does protocol translation)
    → multiple models review in parallel independently → merge and dedupe, annotate consensus → post one PR comment

Ready-made component: the open-source gatellm-io/gatellm-code-review wraps anthropics/claude-code-action, with a PR line-count gate, historical-comment minimization, short-result retry, and @claude follow-up trigger built in. Set it up in 5 steps below.

Prerequisites

• The gateway is running and you can log into the console (if not deployed yet, follow Docker single node first)
• ENCRYPTION_KEY is set (needed both to save upstreams and to issue keys; see Docker single node → Prerequisites)
• The models participating in review are configured with upstreams on the gateway
• Admin permissions on the target GitHub repository (needed to configure Secrets / Variables)

Step 1: configure the review models on the gateway

Console → Upstreams & Models, confirm that the review models (e.g. qwen3.8-max, kimi-k3) are created. If there are many large PRs, prefer the 1M-context-window variants with the [1m] suffix. Note that [1m] is just part of the model name (a naming convention), not a syntax requirement of the review flow — names with or without the suffix are both valid model names, subject to your actual gateway configuration; so both qwen3.8-max and qwen3.8-max[1m] in the examples below are valid.

The model name is used in two places later: single-mode's CODE_REVIEW_MODEL, multi-mode's CODE_REVIEW_MODELS / summary_model. They must match the gateway's model names character-for-character, otherwise the review request hits a 404 model_not_found.

Step 2: issue a dedicated access key for CI

Console → Access Keys:

1. Create a new key group, with models checking only the review models (least privilege; see Multi-tenant isolation for the mechanism).
2. Issue an access key attached to that group and write it down — it goes into a GitHub Secret.
3. Set an RPM / concurrency cap on this key (RATE_LIMIT_*): multi-model review makes parallel requests, and rate limiting avoids crowding out production traffic.

The actual console UI (the screenshot is the English UI; the Chinese UI corresponds to "Access control / Access keys / Key groups / Add access key"):

What the group checks is the model's full name (with upstream prefix) — the same list as the console model list.

The benefit of a dedicated key: review cost is naturally separated from business traffic; on a problem, revoking the key instantly cuts review traffic.

Step 3: configure credentials on the GitHub side

Repository Settings → Secrets and variables → Actions (for multi-repo sharing you can put them at the organization level):

Type	Name	Value
Secret	CODE_REVIEW_API_KEY	The gateway access key issued in step 2
Variable	CODE_REVIEW_BASE_URL	The gateway root URL, e.g. http://your-gateway:7890 (without /v1, Claude Code appends it itself)
Variable	CODE_REVIEW_MODEL	The review model name for single mode, e.g. qwen3.8-max
Variable	CODE_REVIEW_MODELS	Comma-separated list for multi mode, e.g. qwen3.8-max[1m],kimi-k3[1m]
Variable	CODE_REVIEW_SUMMARY_MODEL	The summary model for multi mode (optional; blank takes the first in the list)

Step 4: add the workflow file

Two integration options, pick one per your needs.

Option A: single-model review (Composite Action, quick start)

Create .github/workflows/claude-code-review.yml. The following is ready to use after changing three places: branches, runs-on, and the model variable:

name: Claude Code Review

on:
  pull_request:
    types: [opened, synchronize, reopened]
    branches: [main]          # ← only review PRs targeting this branch; change to your default branch
  issue_comment:
    types: [created]

jobs:
  claude-review:
    # Two triggers: PR opened/updated; or someone @claude asks in a PR comment
    if: >-
      github.event_name == 'pull_request' ||
      (github.event_name == 'issue_comment' &&
       github.event.issue.pull_request != null &&
       github.event.comment.user.type != 'Bot' &&
       contains(github.event.comment.body, '@claude'))
    runs-on: ubuntu-latest    # ← can switch to a self-hosted runner label, e.g. [code-review, arm64]
    permissions:
      contents: read
      pull-requests: write
      issues: write
      actions: read
    # A new push to the same PR cancels the in-progress old review, keeping only the latest round
    concurrency:
      group: claude-review-${{ github.event.pull_request.number || github.event.issue.number }}-${{ github.event_name }}
      cancel-in-progress: true
    steps:
      - name: Checkout repository
        uses: actions/checkout@v7
        with:
          fetch-depth: 1

      - name: Run Claude Auto Review
        uses: gatellm-io/gatellm-code-review@v1
        with:
          # Credentials: point at your gateway
          anthropic_api_key: ${{ secrets.CODE_REVIEW_API_KEY }}
          anthropic_base_url: ${{ vars.CODE_REVIEW_BASE_URL }}

          # Business parameters (all have defaults, override as needed)
          model: ${{ vars.CODE_REVIEW_MODEL }}   # must be forwarded explicitly: a composite action cannot read vars
          # max_lines: 10000                     # skip when the PR changed-lines exceed the cap; -1 = unlimited
          # review_language: "Simplified Chinese"  # review comment language, default English
          # prompt: |                            # override the default review prompt
          #   custom...

Key points:

• The review model can equally be a non-Anthropic model like qwen3.8-max or kimi-k3 — claude-code-action speaks the Anthropic protocol, and the gateway handles the translation.
• When the runner has no Claude CLI preinstalled, the action auto-installs it; a self-hosted runner with it preinstalled reuses it directly.
• A comment shorter than 100 characters is judged a review failure and automatically retried once; no extra handling needed.

Option B: multi-model consensus review (Reusable Workflow, recommended)

Only multiple models give consensus confidence. Create .github/workflows/pr-review.yml, invoking the three-stage workflow as a whole (setup → review matrix → summarize):

name: PR Review
on:
  pull_request:
    types: [opened, synchronize, reopened]
  issue_comment:
    types: [created]

jobs:
  claude-review:
    if: >-
      github.event_name == 'pull_request' ||
      (github.event_name == 'issue_comment' &&
       github.event.issue.pull_request != null &&
       github.event.comment.user.type != 'Bot' &&
       contains(github.event.comment.body, '@claude'))
    uses: gatellm-io/gatellm-code-review/.github/workflows/claude-auto-review.yml@v1
    secrets: inherit   # pass CODE_REVIEW_API_KEY etc. through to the child workflow
    with:
      runs_on: "ubuntu-latest"          # or a self-hosted runner label
      models: "qwen3.8-max[1m],kimi-k3[1m]"   # 2–3; more than 3 are truncated
      summary_model: "qwen3.8-max"
      review_language: "Simplified Chinese"    # review comments in Chinese

Key points:

• models with ≥2 enables multi-model mode; only 1 (or blank falling through to model) automatically falls back to single-model direct review.
• permissions / concurrency are both built into the child workflow; the caller doesn't need to worry about them.
• To change the model combination uniformly across repos, swap models for ${{ vars.CODE_REVIEW_MODELS }} for centralized management.

Step 5: verification

1. Open a test PR (change any one line); a review job should appear in Actions and start running.
2. After the job completes, a review comment appears on the PR page: in multi-model mode findings carry [Consensus N/M] / [Single model] annotations.
3. Console → Logs, you can see review traffic passing through the gateway (User-Agent contains gatellm-claude-code-review); → Statistics to view review usage per model.
4. Reply @claude focus on concurrency safety in the PR comment section, which should trigger one re-review carrying that request.

When it doesn't work:

• Job reports 401 → CODE_REVIEW_API_KEY is wrong or the key is disabled.
• Reports 404 model_not_found → the model variable doesn't match the gateway model name; go back to step 1 to check.
• Can't reach the gateway / timeout → GitHub-hosted runners can only reach public addresses; an intranet gateway needs a self-hosted runner or network access.
• Job succeeds but no comment → check the execution log in the Actions artifacts.

What the output looks like

• Findings carry confidence (multi-model): [Consensus N/M] means N models independently reached the same conclusion (M is the number of participating models), [Single model] means a single-model-only finding; the trust priority is clear at a glance.
• The comment structure is fixed: Findings (sorted by severity, with file:line) → ## TODO Fix List (machine-parseable, graded [P0]–[P3], can be handed directly to a coding AI to claim and fix) → Requires manual attention (items needing human sign-off); multi-model mode keeps each model's raw review in a collapsed block at the bottom.
• Guardrails: a PR whose changed-lines exceed max_lines (default 10000) is auto-skipped, and a manual re-run can force it; before each review round, historical Claude comments are auto-collapsed to OUTDATED, so the PR doesn't accumulate noise.

See the repository's README for the full inputs, permission requirements, and behavior details.

FAQ

Q: Reviewing every PR — will comments flood the timeline? No. Before each review round, the action auto-collapses historical Claude comments on that PR to OUTDATED, leaving only the latest round's expanded comment in the PR timeline; extra-large PRs (exceeding max_lines) are skipped by default.

Q: How is review cost calculated? Each review model is billed separately and aggregated uniformly by the gateway, appearing in statistics and billing as each review model's usage. See Configure model pricing and view monthly billing.

Next: Enhance model capability for the full scenario picture; Claude Code via the gateway for the basic Claude Code integration; Multi-tenant isolation for the least-privilege key group design.