llama-mcp-server

Local LLM inference MCP server powered by llama.cpp.

Standalone binary that exposes a configurable chat tool over MCP (Model Context Protocol) via stdio or HTTP transport using JSON-RPC 2.0.

Features

• Configurable tool name — default chat, customizable via --tool_name
• Flexible input — accepts plain text (text) or multi-turn conversation (messages array)
• Per-request parameters — temperature, max_tokens, top_p, top_k, prompt_suffix overridable per call
• Chat template — uses the model's built-in GGUF chat template for prompt formatting
• Thinking block stripping — automatically removes <think>...</think> blocks from output
• Optional CUDA — GPU acceleration via llama.cpp CUDA backend
• HTTP transport — MCP Streamable HTTP with Bearer token authentication and session management
• Dual transport — stdio (default) or HTTP mode via --transport flag

Architecture

Single crate, four source modules:

CLI Arguments

llama-mcp-server --model <PATH> [OPTIONS]

Options:
  --model <PATH>            Path to GGUF model file [required]
  --n_gpu_layers <N>        GPU layers to offload (0 = CPU, 99 = all) [default: 99]
  --n_ctx <N>               Context window size in tokens [default: 4096]
  --n_batch <N>             Batch size for prompt processing [default: 2048]
  --n_threads <N>           Number of CPU threads [default: 4]
  --tool_name <NAME>        MCP tool name [default: chat]
  --max_tokens <N>          Default max tokens to generate [default: 2048]
  --temperature <F>         Default temperature (0.0 = greedy) [default: 0.7]
  --top_p <F>               Default top-p (nucleus sampling) [default: 0.9]
  --top_k <N>               Default top-k sampling [default: 40]
  --repeat_penalty <F>      Repeat penalty [default: 1.1]
  --seed <N>                Random seed [default: 42]
  --prompt_suffix <TEXT>    Suffix appended to system prompt (e.g. "/think")
  --transport <MODE>        Transport mode: stdio or http [default: stdio]
  --host <HOST>             Host to bind HTTP server [default: 127.0.0.1]
  --port <PORT>             Port for HTTP server [default: 8080]
  --auth <TOKEN>            Bearer token for HTTP authentication (optional)
  --version                 Print version and exit

Build

Prerequisites

• Rust toolchain (stable)
• CMake (llama.cpp build dependency)
• libclang (bindgen dependency)

Native build

cargo build --release

CUDA build

CUDA requires glibc and NVIDIA CUDA toolkit (nvcc, cudart, cuBLAS):

cargo build --release --features cuda

With Nix (requires unfree packages):

nix build .#cuda

Runtime Dependencies

• GGUF model file — any GGUF-format model compatible with llama.cpp (e.g. from Hugging Face)

Rust Dependencies

MCP Protocol

The server supports two transport modes:

• stdio (default) — communicates over stdin/stdout, one JSON object per line
• HTTP — MCP Streamable HTTP on POST /mcp and DELETE /mcp

HTTP Transport

Start the server in HTTP mode:

llama-mcp-server --model model.gguf --transport http --port 8080 --auth secret123

Authentication: when --auth is set, all requests must include Authorization: Bearer <token>. Without --auth, authentication is disabled.

Sessions: the initialize request returns an Mcp-Session-Id header. All subsequent requests must include this header. Sessions are terminated via DELETE /mcp.

Example session:

# Initialize (get session ID)
curl -s -D- -X POST http://127.0.0.1:8080/mcp \
  -H "Authorization: Bearer secret123" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}'

# Use the Mcp-Session-Id from the response headers for subsequent requests
curl -s -X POST http://127.0.0.1:8080/mcp \
  -H "Authorization: Bearer secret123" \
  -H "Content-Type: application/json" \
  -H "Mcp-Session-Id: <session-id>" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}'

# Terminate session
curl -s -X DELETE http://127.0.0.1:8080/mcp \
  -H "Authorization: Bearer secret123" \
  -H "Mcp-Session-Id: <session-id>"

Stdio Transport

The server communicates over stdin/stdout using JSON-RPC 2.0, one JSON object per line.

Initialize

Request:

{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}

Response:

{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2024-11-05","capabilities":{"tools":{}},"serverInfo":{"name":"llama-mcp-server","version":"<version>"}}}

List tools

Request:

{"jsonrpc":"2.0","id":2,"method":"tools/list"}

Chat (simple text)

Request:

{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"chat","arguments":{"text":"Hello, how are you?"}}}

Chat (multi-turn)

Request:

{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"chat","arguments":{"messages":[{"role":"system","content":"You are helpful."},{"role":"user","content":"Hi"}],"temperature":0.5}}}

Response:

{"jsonrpc":"2.0","id":4,"result":{"content":[{"type":"text","text":"Hello! How can I help you today?"}]}}

CI/CD

GitHub Actions workflows:

• CI (ci.yml) — runs cargo fmt, cargo clippy, cargo test on every push/PR to main/develop
• Release (release.yml) — builds binaries for 6 targets on tag push (v*), uploads as release assets

Release targets:

Release process:

1. Create a git tag: git tag vX.Y.Z && git push --tags
2. CI builds binaries for all 6 targets (Linux/macOS via nix, Windows/CUDA via cargo)
3. Create a GitHub release from the tag — CI attaches build artifacts automatically

To update cargoHash in flake.nix after changing dependencies:

./scripts/update-cargo-hash.sh

Usage

Claude Desktop (stdio)

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "llama": {
      "command": "/path/to/llama-mcp-server",
      "args": ["--model", "/path/to/model.gguf"]
    }
  }
}

HTTP mode

llama-mcp-server --model /path/to/model.gguf --transport http --port 8080 --auth mytoken

Connect any HTTP-capable MCP client to http://127.0.0.1:8080/mcp. All requests require Authorization: Bearer mytoken and Content-Type: application/json. See HTTP Transport for protocol details.

Any MCP client (stdio)

The server reads JSON-RPC requests from stdin and writes responses to stdout. Logs go to stderr. Connect any MCP-compatible client using stdio transport.