About Smg
Smg is an MCP server published by smg-project in the RAG category: engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.
The repository has 613 stars and 190 forks, with the last commit 3 hours ago. Recent commit activity is the strongest signal that reported issues will get attention.
Install
git clone https://github.com/smg-project/smgThis server has no ConduID identity, so agent calls to it are not receipted. Pin the version you install and review the source before granting it credentials.
Ask AI
Ask AI about Smg
Powered by Claude · Grounded in docs
Security checks
Static checks derived from the repository 3 hours ago. Heuristics on the source, not an audit.
- ✓README presentThe repository ships documentation.
- ✓License declaredUsable under a stated license.
- ✓Tests presentA test suite exists in the repository.
- ✓Dependencies pinnedA lockfile pins dependency versions.
- !No dynamic code executionSource scan found eval or shell execution. Review how tool inputs reach it before granting credentials.
1 check has not been run yet.
Releases
README
Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. SMG centralizes worker lifecycle management, balances traffic across self-hosted engines and cloud providers, and gives you enterprise-grade control over multi-tenancy, chat-history storage, MCP tooling, and observability — behind one unified endpoint.
Why SMG?
| 🚀 Maximize GPU Utilization | Cache-aware routing tracks each worker's KV-cache state in radix trees to reuse prefixes across SGLang, vLLM, TensorRT-LLM, TokenSpeed, and MLX — with load modeling that accounts for queued token work and KV pressure. |
| 🔌 One API, Any Backend | Route to self-hosted engines over HTTP or gRPC, or to OpenAI, Anthropic, Gemini, and xAI — plus any OpenAI-compatible endpoint — through a single unified gateway. |
| ⚡ Built for Speed | Native Rust with streaming gRPC pipelines, cached tokenization with zero-copy cache hits, prefill/decode disaggregation (including a separate encode stage for vision), and DP-aware routing for data-parallel engines. |
| 🔒 Enterprise Control | Priority admission scheduling with preemption and per-tenant controls, API-key auth with OIDC on the control plane, WebAssembly plugins for custom logic, and chat history that never leaves your infrastructure. |
| 📊 Full Observability | 90+ Prometheus metrics, OpenTelemetry tracing with W3C trace context propagated into the engines over both HTTP and gRPC, and structured JSON logs with request correlation. |
API Coverage: OpenAI Chat Completions, Completions, Embeddings, Rerank, and Classify; Responses and Conversations APIs for agents; Anthropic Messages; Gemini Interactions; Realtime over WebSocket and WebRTC; audio transcription; tokenize/detokenize; and MCP tool execution with approval policies in the Responses and Messages APIs.
Quick Start
Install — pick your preferred method:
# Docker
docker pull lightseekorg/smg:latest
# Kubernetes (Helm)
helm install smg oci://ghcr.io/smg-project/charts/smg
# Python
pip install smg
# Rust (needs protoc)
cargo install smg
Run — point SMG at your inference workers:
# Single worker
smg launch --worker-urls http://localhost:8000
# Multiple workers with cache-aware routing
smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware
# With high availability mesh
smg launch --worker-urls http://gpu1:8000 --enable-mesh \
--mesh-advertise-host 10.0.0.1 --mesh-peer-urls 10.0.0.2:39527
Use — send requests to the gateway:
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama3", "messages": [{"role": "user", "content": "Hello!"}]}'
That's it. SMG is now load-balancing requests across your workers.
Supported Backends
| Self-Hosted Engines | vLLM · SGLang · TokenSpeed · TensorRT-LLM · MLX (Apple Silicon) · any OpenAI-compatible server (e.g. Ollama) |
| Cloud Providers | OpenAI · Anthropic · Google Gemini · xAI · OCI Generative AI · AWS Bedrock · Azure OpenAI · any OpenAI-compatible provider (Groq, Together, …) |
Features
| Feature | Description |
|---|---|
| 10 Routing Policies | cache_aware, least_load, power_of_two, consistent_hashing, prefix_hash, bucket, round_robin, random, manual, passthrough |
| gRPC Pipeline | Native streaming gRPC to the engines with prefill/decode and encode disaggregation and DP-aware routing |
| Kubernetes Discovery | Native pod watchers with label selectors, per-role prefill/decode/encode selectors, and router peer discovery |
| Model Parsers | 21 tool-call parsers and 16 reasoning parsers with automatic model detection — DeepSeek, Qwen, Kimi, GLM, Llama, Mistral, Command, Nemotron, and more |
| MCP Integration | Tool discovery and execution over stdio, SSE, and streamable HTTP, with approval policies and audit logging |
| High Availability | Mesh networking with SWIM gossip and CRDT-replicated state for multi-node deployments |
| Chat History | Pluggable storage with schema migrations: PostgreSQL, Oracle, Redis, or in-memory |
| WASM Plugins | Extend request and response handling with custom WebAssembly middleware |
| Resilience | Circuit breakers, retries with backoff and jitter, rate limiting, and priority admission scheduling |
Documentation
Full documentation lives at lightseek.org/smg.
| Getting Started | Installation and first steps |
| Architecture | How SMG works |
| Configuration | CLI reference and options |
| API Reference | OpenAI-compatible endpoints |
| Kubernetes Setup | In-cluster discovery and production setup |
Contributing
We welcome contributions! See the Contributing Guide for details.
README fetched from the repository 3 hours ago. The original is authoritative.