ypollak2/llm-router

Why This Exists

AI coding assistants route every task — simple questions, complex architecture — to the same expensive model. You pay full price for work that a cheaper model handles equally well.

llm-router sits between your AI tool and the LLM providers. It classifies each task by complexity, picks the cheapest capable model, and falls back through a provider chain on failure. You don't change your workflow. The router handles model selection automatically.

Use this if:

• You use Claude Code, Codex CLI, Gemini CLI, or Pi and want to reduce spend
• You want automatic fallback when a provider is down or rate-limited
• You want local Ollama models tried first (free) before paid APIs
• You want visibility into token spend across providers

Don't use this if:

• You always want the best possible model regardless of cost
• You don't use MCP-compatible tools
• You need guaranteed latency (routing adds classification overhead)

Quick Start

1. Install

pip install llm-routing
llm-router install

Package name: llm-routing on PyPI. CLI command: llm-router.

2. Add providers (optional)

export OPENAI_API_KEY="sk-..."      # GPT-4o, o3
export GEMINI_API_KEY="AIza..."     # Gemini Flash/Pro (free tier available)
export OLLAMA_BASE_URL="http://localhost:11434"  # Local models (free)

Works with zero API keys on Claude Code Pro/Max subscriptions — routing uses MCP tools that call external models only when beneficial.

3. Verify

llm-router install --check   # Preview what will be installed
llm-router health            # Check provider connectivity

In Claude Code, ask a simple question. The session-end summary shows routing decisions and savings.

How It Works

User prompt
    │
    ▼
┌──────────────────────┐
│ Complexity Classifier │  ← Heuristic (free, instant) or Ollama/Flash ($0.0001)
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  Free-First Router   │  ← Tries cheapest model first, walks up the chain
│                      │
│  Ollama (free)       │
│  → Codex (prepaid)   │
│  → Gemini Flash      │
│  → GPT-4o / Claude   │
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  Guards (parallel)   │  ← Circuit breaker, budget pressure, quality check
└──────────┬───────────┘
           │
           ▼
      Response + cost logged to local SQLite

Routing examples

Classification is free (regex heuristics catch ~70% of tasks) or near-free (local Ollama / Gemini Flash for ambiguous cases).

Host Support

Full = hooks intercept prompts and route automatically. No workflow change needed. Manual = MCP tools are available; you invoke them explicitly (e.g., call llm_query).

llm-router install                    # Claude Code (default)
llm-router install --host codex       # Codex CLI
llm-router install --host gemini-cli  # Gemini CLI
llm-router install --host vscode      # VS Code
llm-router install --host cursor      # Cursor

See docs/HOST_SUPPORT_MATRIX.md for full details on each host.

What You Can Do

Providers

Routing chains are built from your configured providers. You only need one.

See docs/PROVIDERS.md for setup instructions and model recommendations.

Routing Policies

Control how aggressively the router offloads to cheap models.

export LLM_ROUTER_POLICY=aggressive     # Or: balanced, conservative
export LLM_ROUTER_ENFORCE=smart          # smart | hard | soft | off
export LLM_ROUTER_PROFILE=balanced       # budget | balanced | premium

LLM_ROUTER_ENFORCE controls how strictly the auto-route hook blocks direct model use:

• smart — route when confident, pass through when uncertain
• hard — always route, block unrouted tool calls
• soft — suggest routing, never block
• off — disable hook enforcement

MCP Tools (60)

llm-router exposes 60 MCP tools organized by function:

Slim mode (LLM_ROUTER_SLIM=routing or core) reduces registered tools to save context tokens in constrained environments.

Full Tool Reference

Savings: How It Works

Savings are calculated by comparing actual spend against a baseline of routing every task to Claude Sonnet/Opus.

Methodology:

1. Each routed task logs: model used, tokens consumed, estimated cost
2. A baseline cost is computed as if the same tokens were processed by the most expensive model in the chain
3. Savings = (baseline - actual) / baseline

Assumptions and limitations:

• Baseline assumes you would have used Opus/Sonnet for everything (worst case)
• Token estimates use len(text) / 4 approximation, not exact tokenizer counts
• Cost data comes from LiteLLM's pricing tables (may lag provider price changes)
• Savings vary significantly by workload — code-heavy sessions route more to cheap models
• The router itself adds small overhead (classification costs ~$0.0001 per ambiguous task)

Observed range: 35–80% savings depending on policy and task mix. The "87%" figure in some docs represents a single-user peak over a specific development period, not a guaranteed outcome.

Trust, Privacy, and Local-First Design

llm-router runs entirely on your machine. There is no hosted proxy, no telemetry, no account required.

What we do:

• Scrub API keys from structured logs
• Detect hook deadlocks before installation
• Store all data locally in ~/.llm-router/
• Respect provider rate limits and TOS

What you should know:

• Prompts are sent to whichever provider the router selects — review your provider's privacy policy
• Usage logs (SQLite) are not encrypted at rest — use full-disk encryption if needed
• The router cannot prevent model jailbreaks or prompt injection at the provider level

See SECURITY.md for responsible disclosure policy and docs/SECURITY_DESIGN.md for the full threat model.

Configuration

Minimal setup — only configure what you have:

# Provider keys (set any combination)
export OPENAI_API_KEY="sk-proj-..."
export GEMINI_API_KEY="AIza..."
export OLLAMA_BASE_URL="http://localhost:11434"
export OLLAMA_BUDGET_MODELS="gemma4:latest,qwen3.5:latest"

# Routing behavior
export LLM_ROUTER_PROFILE="balanced"       # budget | balanced | premium
export LLM_ROUTER_POLICY="balanced"        # aggressive | balanced | conservative
export LLM_ROUTER_ENFORCE="smart"          # smart | hard | soft | off

For teams or environments where .env is restricted:

# User-level config (no project .env needed)
mkdir -p ~/.llm-router && chmod 700 ~/.llm-router
cat > ~/.llm-router/config.yaml << 'EOF'
openai_api_key: "sk-proj-..."
gemini_api_key: "AIza..."
ollama_base_url: "http://localhost:11434"
llm_router_profile: "balanced"
EOF
chmod 600 ~/.llm-router/config.yaml

Documentation

Contributing

Contributions welcome. See CONTRIBUTING.md for full guidelines.

git clone https://github.com/ypollak2/llm-router.git
cd llm-router
uv sync --extra dev
uv run pytest tests/ -q         # Run tests (1700+)
uv run ruff check src/ tests/   # Lint

Package Names