Kubeflow MCP Server

Proposal: KEP-936 · ROADMAP · SECURITY · CONTRIBUTING

Overview

The Kubeflow MCP Server exposes Kubeflow Training operations as Model Context Protocol tools, enabling AI agents (Claude, Cursor, Claude Code, or any custom agents etc.) to plan, submit, monitor, and manage training jobs through natural language — without users needing to learn Kubernetes or the Kubeflow SDK directly.

Benefits

• Agent-Native: Tools auto-discovered via MCP — no manual API wiring
• Guided Workflow: Phase ordering with next-step hints (Plan → Discover → Train → Monitor)
• Preview-Before-Submit: Every mutating operation requires explicit confirmation
• Security-First: Persona gating, namespace enforcement, input validation, bearer/JWT auth
• Multi-Platform: Auto-detects OpenShift, EKS, GKE with platform-specific guidance
• Token-Efficient: Progressive/semantic modes compress 23 tools into 2-3 meta-tools
• Extensible: Plugin architecture for additional Kubeflow clients (TODO: optimizer, hub)

Demo

Get Started

Install from PyPI

pip install kubeflow-mcp

Install from source (development)

git clone https://github.com/kubeflow/mcp-server.git
cd mcp-server
make install-dev

Prerequisites

Before starting the Kubeflow MCP Server, make sure the following requirements are available:

• Python 3.10, 3.11, or 3.12
• A running Kubernetes cluster
• kubectl installed and available in PATH
• A kubeconfig configured for the target cluster
• Kubeflow Trainer installed in the cluster
• Kubeflow Trainer CRDs and at least one ClusterTrainingRuntime
• Kubernetes permissions for the namespace you train in, plus cluster-scoped read access for planning and runtime discovery (see RBAC Configuration)

Verify the connection and your permissions in the target namespace. Each can-i command should print yes:

kubectl config current-context
kubectl auth can-i create trainjobs.trainer.kubeflow.org -n <namespace>
kubectl auth can-i list pods -n <namespace>
kubectl auth can-i get pods --subresource=log -n <namespace>

The checks below need cluster-scoped read access, so they can return Forbidden for a namespace-scoped user even when Kubeflow Trainer is installed correctly. Without that access, the tools that read these resources (get_cluster_resources, check_compatibility, list_runtimes) fail; tools that work on jobs in your namespace are unaffected.

kubectl get nodes
kubectl get crd trainjobs.trainer.kubeflow.org
kubectl get clustertrainingruntimes

Run the server

kubeflow-mcp serve

Run with Docker

Pre-built multi-arch images are published to GHCR on every release:

docker run --rm -p 8000:8000 \
  -e KUBEFLOW_MCP_AUTH_TOKEN=my-secret-token \
  ghcr.io/kubeflow/mcp-server:latest

The server listens on http://localhost:8000/mcp.

Container and Kubernetes probes are available without MCP authentication:

GET /health  # liveness: the server process is accepting HTTP requests
GET /ready   # readiness: configured clients imported and packaged resources loaded

/ready returns 200 only when both clients_ready and resources_ready are true. It does not contact Kubernetes or other APIs, so it is not a live dependency check. A missing packaged resource Markdown file keeps /ready at 503 even though /health and registered tools remain available; check the server logs and package contents rather than cluster dependencies.

Environment variables

MCP client config (HTTP transport)

{
  "mcpServers": {
    "kubeflow": {
      "url": "http://localhost:8000/mcp",
      "headers": { "Authorization": "Bearer my-secret-token" }
    }
  }
}

For in-cluster deployments, replace localhost:8000 with the Kubernetes Service address and mount KUBEFLOW_MCP_AUTH_TOKEN from a Secret.

Note: DNS rebinding protection allows only loopback Host/Origin headers by default. When exposing the server through a Service or Ingress, set KUBEFLOW_MCP_ALLOWED_HOSTS (e.g. KUBEFLOW_MCP_ALLOWED_HOSTS=kubeflow-mcp.kubeflow.svc:*,mcp.example.com) or requests will be rejected with HTTP 421.

Example: Fine-tune a model via AI agent

Once connected, your AI agent can run a complete training workflow through natural language:

User: "Fine-tune gemma-2b on the alpaca dataset"

Agent calls: check_compatibility()        → ✅ K8s 1.29, Trainer CRD installed
Agent calls: get_cluster_resources()      → 4x A100 GPUs available
Agent calls: estimate_resources("google/gemma-2b") → needs ~16GB GPU, 1x A100
Agent calls: list_runtimes()              → torchtune-llama, torchtune-gemma, ...
Agent calls: fine_tune(                   → preview config (confirmed=False)
    model="hf://google/gemma-2b",
    dataset="hf://tatsu-lab/alpaca",
    runtime="torchtune-gemma-2b"
)
Agent calls: fine_tune(..., confirmed=True) → TrainJob "train-gemma-abc" created
Agent calls: get_training_logs("train-gemma-abc") → training progress...

All mutating tools require confirmed=True and return a preview before making changes.

MCP Client Config

Add to .cursor/mcp.json (or use the .mcp.json at the repo root for local dev):

{
  "mcpServers": {
    "kubeflow": {
      "command": "uv",
      "args": ["run", "kubeflow-mcp", "serve"]
    }
  }
}

claude mcp add kubeflow -- kubeflow-mcp serve

Tools

23 tools organized by workflow phase:

Requirements

CLI Reference

kubeflow-mcp serve

# Modules: trainer, optimizer (stub), hub (stub)
# Persona: readonly | data-scientist | ml-engineer | platform-admin
# Mode: full | progressive | semantic
# Instruction tier: full | compact | minimal
# Transport: stdio | http | sse (deprecated)
# Auth token: bearer token for HTTP auth (dev/staging)
# OTel endpoint: optional OTLP HTTP endpoint for tracing
# Log level: DEBUG | INFO | WARNING | ERROR
# Log format: console | json (auto-detected if omitted)
# No banner: suppress the FastMCP startup banner
kubeflow-mcp serve \
  --clients trainer \
  --persona ml-engineer \
  --mode full \
  --instruction-tier full \
  --transport stdio \
  --auth-token SECRET \
  --otel-endpoint URL \
  --log-level INFO \
  --log-format console \
  --no-banner

--mode progressive exposes 3 meta-tools (~85 tokens) for hierarchical discovery. --mode semantic exposes 2 meta-tools (~69 tokens) using embedding search. Both reduce token consumption significantly for agent workflows.

--persona controls which tools are exposed. Each persona includes the tools of the one above it:

When using --transport http, configure auth to secure the endpoint:

# Simple API key (dev/staging)
kubeflow-mcp serve --transport http --auth-token my-secret-token

# Or via env var
export KUBEFLOW_MCP_AUTH_TOKEN=my-secret-token
kubeflow-mcp serve --transport http

# JWT verification (production)
export KUBEFLOW_MCP_JWKS_URI=https://auth.example.com/.well-known/jwks.json
export KUBEFLOW_MCP_JWT_ISSUER=https://auth.example.com
export KUBEFLOW_MCP_JWT_AUDIENCE=kubeflow-mcp
kubeflow-mcp serve --transport http

Without auth configured, the server logs a warning that the HTTP endpoint is open.

Observability

OpenTelemetry tracing is optional and can be enabled without changing tool code.

• Install optional dependencies from source: uv sync --group otel
• Enable tracing with CLI flag or env var:

kubeflow-mcp serve --otel-endpoint http://localhost:4318
# or
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
kubeflow-mcp serve

Each tool invocation emits a span with attributes: tool.name, tool.args_preview, tool.success, tool.duration_ms, kubeflow.persona, and correlation_id.

Development

make install-dev                  # setup environment
make verify                       # lint + format check
make test-python                  # run tests
make inspector                    # launch MCP Inspector (stdio)
make inspector TRANSPORT=http     # Inspector + Streamable HTTP (start server separately)
make inspector TRANSPORT=sse      # Inspector + SSE (deprecated; start server separately)

Community

• Slack: Join #kubeflow-ml-experience on CNCF Slack
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly call
• GitHub: Issues and contributions at kubeflow/mcp-server

Documentation

• CONTRIBUTING: Development workflow and PR guidelines
• ROADMAP: Project roadmap
• SECURITY: Vulnerability reporting; see ARCHITECTURE.md#security-model for threat model, RBAC, and hardening
• Client conventions: Checklist for adding client modules
• KEP-936: Design proposal

License

Apache License 2.0 — see LICENSE.