🚀 Swarm — Standalone Model Gateway or Multi-Agent Planner in One Runtime 🦀

Swarm is a production-grade, Rust-native AI runtime that lets you run either as an ultra-fast, resilient Model Gateway, an autonomous Multi-Agent Planner DAG, or both in a unified server — giving your team complete architectural freedom without managing separate stacks.

---

Table of Contents

• Why Swarm?
• Deployment Patterns: One Codebase, Two Deploy Patterns
• Enterprise Resilience & Production Hardening
• Context-Aware Intelligence (Memory, Facts & Identity)
• Quick Onboarding Scenario
  • Step 1 — Put Swarm in Front of Your Models
  • Step 2 — Verify Health & Readiness
  • Step 3 — Execute Chat & Stateful Responses
  • Step 4 — Add Agent Orchestration When You Need It
• Key Capabilities
• Project Direction
• Documentation
• Repository Structure
• Contributing
• License

---

Why Swarm?

Building AI products typically forces engineering teams to manage two separate, disconnected stacks:

1. An LLM Gateway for multi-provider routing, timeouts, rate limiting, and failover.
2. An Agent Framework for DAG planning, tool execution, session memory, and multi-agent coordination.

Operating separate systems introduces protocol impedance, duplicated authentication, fragmented observability, and unpredictable latency.

Swarm solves this by unifying both worlds around a single Tokio-based async runtime and typed protocol abstractions.

               Client Application / API Consumer
                               │
                               ▼
 ┌───────────────────────────────────────────────────────────┐
 │                       S W A R M                           │
 ├─────────────────────────────┬─────────────────────────────┤
 │   Standalone Gateway        │   Full Agent Runtime        │
 │   • /v1/chat/completions    │   • Planner & Executor DAGs │
 │   • /v1/responses (State)   │   • Domain Specialist Agents│
 │   • Provider Circuit Breaker│   • Native MCP Tool Host    │
 │   • Concurrency Throttling  │   • Shared Working Memory   │
 │   • Health Probes           │   • LLM-as-a-Judge Eval     │
 └──────────────┬──────────────┴──────────────┬──────────────┘
                │                             │
         ┌──────┴──────────────┐       ┌──────┴──────────────┐
         ▼                     ▼       ▼                     ▼
    Cloud Models          Local Models   External Tools   Enterprise State
    OpenAI, Gemini, Groq  Ollama, vLLM   MCP SSE/HTTP     Discovery & Memory

Key Customer Value Proposition

Start as a high-performance, resilient model gateway. Scale into full agent workflows without ever replacing your underlying AI infrastructure.

Direct LLM Call
  → Resilient Model Gateway (12MB binary, sub-10ms startup)
    → Stateful Turn Chaining (/v1/responses)
      → MCP Tool Integration (Streaming SSE)
        → Autonomous Multi-Agent DAGs
          → Automated LLM Evaluation & Guardrails

---

Deployment Patterns: One Codebase, Two Deploy Patterns

Different workloads demand different operational footprints. Swarm provides two first-class deployment targets from the same codebase:

Pattern A: Standalone Gateway (swarm_gateway)

• Target Workloads: Kubernetes sidecar proxy, edge deployment, microservice LLM egress, high-throughput model router.
• Footprint: ~12 MB release binary, cold starts under 10ms, minimal RSS memory.
• Isolation: Strips away agent orchestration, MCP tool engines, and planner DAGs. If downstream agent tools fail, your gateway stays 100% online.
• Capabilities: Stateless /v1/chat/completions (OpenAI spec compliant), stateful /v1/responses with multi-tenant session isolation, full tool forwarding & preservation across streaming and non-streaming, multi-provider routing with automatic fallback retries, concurrency control, timeouts, circuit breaking, optional inbound bearer auth, and secure default binding to 127.0.0.1:8080.

Pattern B: Unified Swarm Server (swarm_server)

• Target Workloads: All-in-one AI platform, enterprise agent workspaces, complete developer environments.
• Footprint: Single unified binary containing both the gateway routes and the full multi-agent orchestration engine.
• Capabilities: Everything in Pattern A plus dynamic DAG planning, executor workers, domain specialist agents, native in-process NATS transport (nats feature), native MCP server hosting, discovery service, and evaluation.

---

Enterprise Resilience & Production Hardening

When deploying LLMs to production, upstream provider latency spikes, outages, and connection hangs can compromise your application. Swarm includes built-in, zero-dependency resilience features with structured error mapping:

Resilience Feature	Customer Benefit	HTTP Status On Trip
Concurrency Limiting	Caps inflight requests to prevent thread starvation and out-of-memory crashes under peak traffic surges.	429 Too Many Requests
Execution Deadlines	Enforces per-request timeouts so hung provider connections or slow generation never lock client sockets.	408 Request Timeout / 504 Gateway Timeout
Provider Circuit Breakers	Automatically detects consecutive provider failures (e.g. OpenAI or Groq outage), failing fast and allowing automatic cooldown recovery.	503 Service Unavailable
Multi-Provider Fallback Retries	Automatically retries requests across fallback providers in MultiModelGatewayBackend before failing.	Transparent recovery / 502 Bad Gateway
Upstream Error Mapping	Translates upstream rate limits and gateway failures into clean, structured HTTP responses.	429 Too Many Requests / 502 Bad Gateway
Inbound Auth Middleware	Optional SharedAuthenticator enforcing bearer token validation on API endpoints while keeping /health open for probes.	401 Unauthorized
Cloud-Native Health Checks	Built-in public /health and /v1/health endpoints returning provider readiness for Kubernetes probes and load balancers.	200 OK / 503 Unhealthy

Resilience Configuration Example

Add this simple section to your gateway_config.toml to guard your production endpoints:

[server]
# Secure default bind address (use 0.0.0.0:8080 for external container exposure)
bind_address = "127.0.0.1:8080"

[resilience]
# Concurrency limit for inflight requests (protects Tokio worker threads)
max_concurrent_requests = 512

# Global request timeout in seconds (guards against hung upstream connections)
request_timeout_seconds = 60

# Circuit breaker: trip provider offline if N consecutive errors occur
circuit_breaker_failure_threshold = 5

# Cooldown window: wait N seconds before testing provider recovery
circuit_breaker_reset_seconds = 30

---

Context-Aware Intelligence (Memory, Facts & Identity)

Production agents cannot rely on stateless, single-turn interactions. Swarm introduces a Four-Tier Context Architecture that enriches user prompts with conversational history, persistent factual knowledge, structured agent identity, and procedural tool definitions before reaching the LLM:

┌─────────────────────────────────────────────────────────────┐
│                   Context Assembly Pipeline                 │
└──────────────────────────────┬──────────────────────────────┘
                               │
       ┌───────────────────────┼───────────────────────┐
       ▼                       ▼                       ▼
┌──────────────┐       ┌──────────────┐       ┌──────────────┐
│  Tier 1:     │       │  Tier 2:     │       │  Tier 3:     │
│ Conversational│      │  Persistent  │       │  Agent       │
│ Memory       │       │  Facts       │       │  Identity    │
│ (Sliding     │       │ (Categorized │       │ (Role, Goals,│
│  Window)     │       │  Fact Store) │       │  Capabilities│
└──────┬───────┘       └──────┬───────┘       └──────┬───────┘
       │                      │                      │
       └──────────────────────┼──────────────────────┘
                              ▼
           ┌─────────────────────────────────────┐
           │        ContextProvider Trait        │
           │  provide_context(&ContextRequest)   │
           └──────────────────┬──────────────────┘
                              ▼
           ┌─────────────────────────────────────┐
           │   [Base System Prompt]              │
           │   [Identity & Capability Directives]│
           │   [Relevant Known Facts]            │
           │   [Sliding-Window Conversation]     │
           │   [Current User Query]              │
           └──────────────────┬──────────────────┘
                              ▼
                     LLM Execution Loop

Four-Tier Context Architecture

Tier	Component	Description	Default Behavior
Tier 1: Conversational Memory	HistoryContextProvider	Retrieves previous turns from MemoryService using a sliding window.	Configurable agent_mcp_history_length (default: 10 when enabled)
Tier 2: Persistent Facts	SemanticMemoryContextProvider	Matches query text against structured facts via /facts and /facts/recall endpoints.	Opt-in via agent_mcp_enable_memory_recall = true
Tier 3: Agent Identity	IdentityContextProvider	Injects structured role, objectives, constraints, and capabilities into the system prompt.	Opt-in via agent_mcp_enable_identity_context = true
Tier 4: Procedural Skills	MCP Tool Engine	Dynamically discovers and invokes external tools, APIs, and workflows.	Active via MCP runtime

100% Optional & Backward Compatible

All context features are strictly opt-in:

• Zero Alterations to Existing Agents: When disabled or unconfigured, agents run in their standard stateless mode with zero performance overhead.
• Runs Without Services: Agents function identically whether or not agent_memory_service or agent_discovery_service are running in the cluster.

Enabling Context-Aware Features

Configure context awareness in your agent's MCP runtime TOML configuration:

# --- Context-Aware Intelligence Settings ---

# Number of past conversation turns to retrieve as sliding window (e.g., 10)
agent_mcp_history_length = 10

# Enable semantic fact recall from MemoryService matching user input
agent_mcp_enable_memory_recall = true

# Enable structured identity injection (role, objectives, constraints, capabilities)
agent_mcp_enable_identity_context = true

---

Quick Onboarding Scenario

Step 1 — Put Swarm in Front of Your Models

Launch the standalone gateway in seconds:

cd swarm
./kickstart/gateway_kickstart/01_launch_gateway.sh

Step 2 — Verify Health & Readiness

Confirm the gateway and configured providers are healthy:

curl http://localhost:8080/health

{
  "status": "healthy",
  "version": "0.1.0",
  "providers_configured": 4
}

Step 3 — Execute Chat & Stateful Responses

Standard OpenAI-Compatible Stateless Chat (/v1/chat/completions)

/v1/chat/completions is strictly stateless (no unintended session history side effects) and supports upstream tool forwarding:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-20b",
    "messages": [
      {"role": "user", "content": "Explain Swarm in one sentence."}
    ]
  }'

Tool Calling Example (Forwarded to upstream Gemini / Groq / OpenAI with preserved tool calls):

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-20b",
    "messages": [
      {"role": "user", "content": "What is the weather in Boston?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {"type": "string"}
            },
            "required": ["location"]
          }
        }
      }
    ]
  }'

Stateful Open Responses (/v1/responses — Multi-Turn Isolated Session Memory)

Conversational turns are isolated by session key, and assistant responses are automatically appended to session history:

# Turn 1: Initial query (returns session ID "resp_123...")
curl -X POST http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-20b",
    "input": "My favorite city is Boston."
  }'

# Turn 2: Follow-up query linked with previous_response_id
curl -X POST http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-20b",
    "previous_response_id": "resp_123...",
    "input": "What is its most famous monument?"
  }'

(Note: When inbound auth is enabled via with_authenticator, pass -H "Authorization: Bearer <token>".)

Swarm routes seamlessly across cloud and local providers:

Provider	Type	Typical Use Case
Groq	Cloud	Ultra-low latency Llama 3.3 / Qwen generation
Google Gemini	Cloud	Multimodal and large context-window reasoning
OpenAI	Cloud	Industry-standard GPT-4o models
Ollama / vLLM / llama.cpp	Local / Self-Hosted	Private, on-premise, zero-data-egress inference

Step 4 — Add Agent Orchestration When You Need It

When your use case evolves from single LLM calls to multi-step reasoning, launch the full orchestration stack:

./kickstart/multi_agent_orchestration_kickstart/01_launch_all.sh

Test a coordinated agent task:

./kickstart/multi_agent_orchestration_kickstart/02_test_weather_query.sh \
  "What is the current weather in Boston?"

The request automatically coordinates across:

User → Planner → Execution DAG → Executor → Domain Agent → MCP Tool → Evaluation → Response

---

Key Capabilities

🌐 Model Gateway

• OpenAI-compatible stateless /v1/chat/completions (OpenAI spec compliant; zero accidental session store writes).
• Stateful /v1/responses with strict multi-tenant session isolation and automatic assistant response history persistence.
• Tool Forwarding & Call Preservation: Full forwarding of tools and tool_choice (GatewayTurnOptions) to upstream providers (Gemini, Groq, OpenAI), retaining tool call structures across streaming SSE and non-streaming responses.
• Multi-provider intelligent routing with automatic fallback retry loops in MultiModelGatewayBackend.
• Pluggable inbound auth middleware (SharedAuthenticator) enforcing bearer token security across /v1/chat/completions and /v1/responses while maintaining open /health probes.
• SSE streaming with token-level output.
• Secure default localhost binding: 127.0.0.1:8080.
• Available as a standalone, zero-orchestration 12MB binary (swarm_gateway).

🛡️ Enterprise Resilience

• Concurrency Throttling: Semaphore-backed inflight request bounding (429 Too Many Requests).
• Request Timeouts & Deadlines: Configurable per-request deadlines (408 Request Timeout / 504 Gateway Timeout).
• Circuit Breaker: Automatic provider outage isolation with half-open recovery probes (503 Service Unavailable).
• Fallback Retries & Error Mapping: Automatic fallback provider retries with structured status mappings (429 for rate limits, 502 for upstream failures, 503 on tripped breakers).
• Health Probes: Public liveness and readiness /health endpoints for Kubernetes probes and load balancers.

🤖 Agent Runtime & Transports

• Turn Concurrency: Fully lock-free turn processing (removed Mutex serialization bottleneck in AgentHandler), allowing parallel turns across sessions.
• Multi-Tenant Session Isolation: Pluggable SessionKeyResolver strategies (A2aSessionKeyResolver eliminates cross-tenant leakage; TenantThreadResolver supports composite {tenant}_{thread} keys).
• Session Limits & Background TTL: Bounded FIFO session stores with automatic background eviction (SessionStore::with_limits(max_items, ttl)).
• Native In-Process NATS Transport: High-throughput messaging without HTTP loopback overhead under the nats feature, featuring subject/header credential validation via CredentialExtractor.
• Dynamic Workflow Planning: DAG execution graphs with strict Planner / Executor separation for safety and auditability.
• Unified RequestContext: Normalized extraction of tenant IDs, thread IDs, and JWT credentials across metadata aliases (agent_models::RequestContext).
• Ergonomic Builders: AgentHandlerBuilder and SecureAgentServerBuilder.
• Production Safety: Strict #![deny(clippy::unwrap_used, clippy::expect_used)] compile-time assurance across the core runtime.

🧠 Context-Aware Intelligence

• Pluggable Context Engine: Extensible ContextProvider pipeline assembling conversation history, persistent facts, and agent identity before LLM execution.
• Sliding-Window Compaction: Request-scoped conversation memory retrieval (HistoryContextProvider) with configurable turn limits.
• Semantic Fact Recall: Substring and category fact indexing (SemanticMemoryContextProvider) connecting agents to shared memory.
• Structured Identity Injection: Automatic prompt structuring for agent role, objectives, constraints, and capabilities (IdentityContextProvider).
• Fully Opt-In: Operates smoothly without discovery or memory services; zero breaking changes for existing stateless deployments.

🔌 Native MCP (Model Context Protocol)

• Full MCP client and server protocol support.
• Standard SSE and Streamable HTTP transports.
• Dynamic tool discovery and streaming execution.

🔍 Built-in Evaluation

• Automated LLM-as-a-Judge quality assessment.
• Historical audit scoring, critique generation, and regression tracking.
• Pre-deployment quality gates for agent behaviors and prompts.

⚡ Rust & Tokio Performance

• Sub-10ms cold start times.
• Zero-mutex turn concurrency across sessions.
• Extremely low memory footprint with zero garbage-collection pauses.
• Predictable P99 latencies under heavy enterprise concurrency.

---

Project Direction

Swarm is evolving from an independent Gateway + Framework into an Enterprise AI Control Plane:

[ Gateway + Agent Framework ]  ──►  [ Unified AI Runtime & Control Plane ]

Key roadmap initiatives:

• Unified End-to-End Tracing: Distributed OpenTelemetry traces spanning prompt, gateway routing, DAG nodes, and MCP tool execution.
• Cost & Latency Routing: Dynamic model selection based on real-time price, latency SLAs, and provider health.
• Durable Workflow Execution: Checkpoint and resume long-running agent workflows across process restarts.
• Human-in-the-Loop Policies: Interactive approval steps embedded directly into execution graphs.
• Unified swarm serve CLI: Single command local development and deployment.

See docs/ROADMAP.md for full roadmap details.

---

Documentation

• Architecture & Design
• Deployment & Operations
• Evaluation & Continuous Improvement
• Design Principles
• Roadmap

---

Repository Structure

swarm/
├── kickstart/
│   ├── gateway_kickstart/                     # Standalone gateway scripts & configs
│   │   ├── 01_launch_gateway.sh
│   │   ├── 02_test_chat_completions.sh
│   │   ├── 03_test_open_responses.sh
│   │   ├── 04_terminate_gateway.sh
│   │   └── config_files/gateway_config.toml  # Resilience & provider settings
│   └── multi_agent_orchestration_kickstart/   # Full agent cluster kickstarts
├── agent_factory/                             # Unified swarm_server binary & builder
├── basic_agent/                               # Base agent scaffolding & traits
├── mcp_runtime/                               # MCP execution engine & context providers
├── planner_agent/                             # DAG planning engine
├── executor_agent/                            # Dynamic task execution worker
├── workflow_management/                       # State and workflow transitions
├── resource_invoker/                          # Tool invocation & MCP bridge
└── docs/                                      # Comprehensive architectural guides

---

Contributing

Contributions, issues, architecture discussions, and real-world feedback are warmly welcomed!

• Report bugs or submit feature requests via GitHub Issues.
• Discuss design ideas and deployment patterns in Discussions.
• If Swarm accelerates your AI stack, consider giving the repository a ⭐!

---

License

Distributed under the Apache License 2.0.

---

Swarm — Resilient Model Gateway or Multi-Agent Planner in One Runtime.