Welcome to Building a Kubernetes Operator for MCP Servers

In this comprehensive lab, you'll learn to build a production-ready Kubernetes operator for managing Model Context Protocol (MCP) servers. This tutorial combines three essential areas:

Part 1: MCP Fundamentals - Understanding the Model Context Protocol, its architecture, and building MCP servers that connect AI applications to data sources and tools.

Part 2: MCP Server Operator - Bringing it all together by building a Kubernetes operator that manages MCP server deployments, making them enterprise-ready.

By the end, you'll have:

• Built and deployed MCP servers that integrate with LLMs
• Created a complete operator for managing MCP server lifecycles
• Understood production considerations for AI infrastructure

This lab bridges the gap between AI tooling and cloud-native infrastructure. Let's get started!# Step 1: Introduction to Model Context Protocol (MCP)

Welcome to the first step of our journey! In this step, we'll understand what MCP is, why it matters, and how it enables AI applications to connect with real-world systems.

What is MCP?

The Model Context Protocol (MCP) is an open standard that enables AI applications to connect securely with external data sources, tools, and services. Think of it as "USB for AI integrations" - it provides a standardized way for AI models to access context from your systems.

The Problem MCP Solves

Before MCP, if you had M different AI applications and N different tools/systems, you'd need M×N different integrations:

AI App 1 ←→ Tool 1
AI App 1 ←→ Tool 2
AI App 2 ←→ Tool 1
AI App 2 ←→ Tool 2
... (complex web of integrations)

MCP transforms this into an "M+N problem":

AI Apps ←→ MCP Protocol ←→ MCP Servers ←→ Tools/Systems

MCP Architecture

MCP follows a client-server architecture:

• MCP Host: The AI application (Claude, VS Code, ChatGPT) that coordinates multiple MCP clients
• MCP Client: Maintains a connection to an MCP server and obtains context
• MCP Server: Provides context, tools, and capabilities to MCP clients

Core MCP Primitives

MCP servers expose three fundamental building blocks:

1. Resources - Structured data or content (like GET endpoints)
2. Tools - Executable functions that perform actions (like POST endpoints)
3. Prompts - Pre-defined templates for AI interactions

Why MCP Matters for Kubernetes

MCP enables AI applications to:

• Query Kubernetes cluster state
• Deploy and manage workloads
• Troubleshoot issues with context
• Automate operations based on cluster events

In this tutorial, we'll build an operator that makes MCP servers first-class citizens in Kubernetes!

Next, let's dive deeper into MCP's building blocks.# Step 2: MCP Building Blocks - Servers, Clients, and Components

Now let's explore the three core MCP primitives in detail and understand how they work together.

MCP Resources

Resources provide structured data to AI applications. They're similar to REST API GET endpoints:

// Example: Kubernetes cluster resource
server.registerResource(
  "cluster-info",
  "k8s://cluster/info",
  {
    title: "Cluster Information",
    description: "Current Kubernetes cluster status",
    mimeType: "application/json"
  },
  async (uri) => ({
    contents: [{
      uri: uri.href,
      text: JSON.stringify({
        nodes: await getNodeCount(),
        pods: await getPodCount(),
        version: await getClusterVersion()
      })
    }]
  })
);

Resources are read-only and should be idempotent. They provide context but don't perform actions.

MCP Tools

Tools enable AI applications to perform actions. They're like REST API POST endpoints:

// Example: Pod creation tool
server.registerTool(
  "create-pod",
  {
    title: "Create Pod",
    description: "Create a new pod in the cluster",
    inputSchema: {
      name: z.string(),
      image: z.string(),
      namespace: z.string().default("default")
    }
  },
  async ({ name, image, namespace }) => {
    const pod = await k8sApi.createNamespacedPod(namespace, {
      metadata: { name },
      spec: {
        containers: [{ name, image }]
      }
    });
    
    return {
      content: [{
        type: "text",
        text: `Created pod ${name} in namespace ${namespace}`
      }]
    };
  }
);

Tools can have side effects and perform mutations.

MCP Prompts

Prompts are reusable templates that guide AI interactions:

// Example: Kubernetes troubleshooting prompt
server.registerPrompt(
  "troubleshoot-pod",
  {
    title: "Troubleshoot Pod Issues",
    description: "Systematically diagnose pod problems",
    argsSchema: {
      podName: z.string(),
      namespace: z.string().default("default")
    }
  },
  ({ podName, namespace }) => ({
    messages: [{
      role: "user",
      content: {
        type: "text",
        text: `Please troubleshoot the pod "${podName}" in namespace "${namespace}". Check:
1. Pod status and events
2. Container logs
3. Resource requests/limits
4. Network connectivity
5. Storage issues

Provide step-by-step diagnosis and remediation suggestions.`
      }
    }]
  })
);

Protocol Communication

All MCP communication uses JSON-RPC 2.0 over various transports:

1. stdio - For command-line tools and local integrations
2. HTTP/SSE - For remote servers (legacy)
3. Streamable HTTP - For modern remote servers

Understanding MCP Sessions

MCP servers maintain sessions with clients through:

// Session lifecycle
1. Initialize → Capability negotiation
2. List resources/tools/prompts → Discovery  
3. Read/call operations → Active usage
4. Close → Cleanup

Each session is stateful and maintains context throughout the interaction.

Real-World MCP Examples

Let's look at some production MCP servers:

# Check out popular MCP servers
echo "Popular MCP Servers:"
echo "- GitHub: Automated code operations"
echo "- PostgreSQL: Database queries"
echo "- Slack: Messaging operations"
echo "- Kubernetes: Cluster management"
echo "- Filesystem: File operations"

Next Steps

In the next step, we'll set up our development environment and start building our first MCP server!# Step 3: MCP Development Environment Setup

Let's set up everything we need to build production-ready MCP servers and connect them to AI applications like Claude Desktop.

Verify Prerequisites

First, let's verify we have the required tools:

# Check Node.js version (should be 18+)
node --version

# Check npm version
npm --version

# Verify Kubernetes cluster access
kubectl cluster-info
kubectl get nodes

Create MCP Lab Project

Set up our MCP development workspace with the proven working configuration:

# Create workspace
mkdir -p workspace/mcp-lab
cd workspace/mcp-lab

# Initialize project
npm init -y

Install Proven Dependencies

Install the exact versions that we've tested and verified to work:

# Install core MCP and Kubernetes dependencies
npm install @modelcontextprotocol/sdk@1.17.3 zod@3.25.76 @kubernetes/client-node@1.3.0

# Install HTTP server dependencies
npm install express@4.21.1 js-yaml@4.1.0

# Install development dependencies
npm install --save-dev typescript@5.9.2 @types/node@24.3.0 ts-node@10.9.2 tsx@4.20.4
npm install --save-dev @types/express@5.0.3 @types/js-yaml@4.0.9

echo "✅ Dependencies installed successfully"

Create Working Package Configuration

Set up package.json with all the scripts we'll need:

cat > package.json << 'EOF'
{
  "name": "mcp-kubernetes-lab",
  "version": "1.0.0",
  "description": "Kubernetes MCP Server Lab",
  "main": "dist/servers/k8s-mcp-server.js",
  "type": "module",
  "scripts": {
    "build": "tsc",
    "dev": "tsx src/servers/k8s-mcp-server.ts",
    "dev:http": "tsx src/servers/k8s-mcp-http-server.ts",
    "test": "node --test dist/test-client.js",
    "start": "node dist/servers/k8s-mcp-server.js",
    "start:http": "node dist/servers/k8s-mcp-http-server.js",
    "inspector": "npx @modelcontextprotocol/inspector node dist/servers/k8s-mcp-server.js",
    "inspector-http": "npm run start:http & sleep 2 && npx @modelcontextprotocol/inspector http://localhost:3001/mcp",
    "clean": "rm -rf dist"
  },
  "keywords": [
    "mcp",
    "kubernetes",
    "ai",
    "model-context-protocol"
  ],
  "author": "MCP Tutorial",
  "license": "MIT",
  "dependencies": {
    "@kubernetes/client-node": "^1.3.0",
    "@modelcontextprotocol/sdk": "^1.17.3",
    "express": "^4.21.1",
    "js-yaml": "^4.1.0",
    "zod": "^3.25.76"
  },
  "devDependencies": {
    "@types/express": "^5.0.3",
    "@types/js-yaml": "^4.0.9",
    "@types/node": "^24.3.0",
    "ts-node": "^10.9.2",
    "tsx": "^4.20.4",
    "typescript": "^5.9.2"
  },
  "engines": {
    "node": ">=18.0.0"
  }
}
EOF

echo "✅ Package configuration created"

Setup TypeScript Configuration

Create the TypeScript configuration that works with our MCP servers:

cat > tsconfig.json << 'EOF'
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "node",
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "strict": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "outDir": "./dist",
    "rootDir": "./src",
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules", "dist"]
}
EOF

echo "✅ TypeScript configuration created"

Create Project Structure

Set up the directory structure for our MCP servers:

# Create organized directory structure
mkdir -p src/{servers,handlers,types,utils}
mkdir -p examples config

# Create the handlers directory for shared MCP logic
echo "✅ Created src/handlers/ for shared MCP server logic"
echo "✅ Created src/servers/ for stdio and HTTP server implementations"
echo "✅ Created src/types/ for TypeScript type definitions"
echo "✅ Created src/utils/ for utility functions"

Install Dependencies and Verify Setup

# Install all dependencies
npm install

# Verify critical dependencies
echo "=== Dependency Verification ==="
npm list @modelcontextprotocol/sdk
npm list @kubernetes/client-node
npm list typescript

echo ""
echo "=== Environment Verification ==="
echo "Node.js: $(node --version)"
echo "npm: $(npm --version)"
echo "TypeScript: $(npx tsc --version)"

Test Kubernetes Connectivity

Verify we can connect to the Kubernetes cluster for our MCP servers:

# Test basic Kubernetes API access
echo "=== Kubernetes Connectivity Test ==="
kubectl cluster-info

# Check if we can list basic resources
kubectl get namespaces
kubectl get pods --all-namespaces | head -5

# Verify permissions for MCP server operations
echo ""
echo "=== Permission Check ==="
kubectl auth can-i get pods
kubectl auth can-i list nodes
kubectl auth can-i create pods

echo "✅ Kubernetes connectivity verified"

Environment Summary

Let's confirm everything is ready:

echo "=========================================="
echo "🚀 MCP Development Environment Ready!"
echo "=========================================="
echo "📦 Project: mcp-kubernetes-lab"
echo "🟢 Node.js: $(node --version)"
echo "🟢 TypeScript: $(npx tsc --version)"
echo "🟢 MCP SDK: $(npm list @modelcontextprotocol/sdk --depth=0 | grep @modelcontextprotocol)"
echo "🟢 Kubernetes: Connected"
echo "📁 Structure: src/{servers,handlers,types,utils}"
echo ""
echo "Next: Build your first Kubernetes-aware MCP server!"
echo "=========================================="

Perfect! Our development environment is now set up with the exact configuration that we've tested and proven to work. In the next step, we'll build our first MCP server with reusable handlers and both stdio and HTTP transport support.# Step 4: Building Your First MCP Server

Now let's build a complete MCP server with Kubernetes integration! We'll create reusable handlers and both stdio and HTTP server implementations.

Create Shared MCP Handlers

First, let's create reusable handlers that both our stdio and HTTP servers can use:

# Create shared handlers for MCP protocol operations
cat > src/handlers/k8s-handlers.ts << 'EOF'
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import * as k8s from '@kubernetes/client-node';
import {
  ListResourcesRequestSchema,
  ReadResourceRequestSchema,
  ListToolsRequestSchema,
  CallToolRequestSchema,
  ListPromptsRequestSchema,
  GetPromptRequestSchema
} from "@modelcontextprotocol/sdk/types.js";

// Initialize Kubernetes client
const kc = new k8s.KubeConfig();
kc.loadFromDefault();
const k8sApi = kc.makeApiClient(k8s.CoreV1Api);
const appsApi = kc.makeApiClient(k8s.AppsV1Api);

// Resource handlers
export function setupResourceHandlers(server: Server) {
  server.setRequestHandler(ListResourcesRequestSchema, async () => {
    return {
      resources: [
        {
          uri: "k8s://cluster/nodes",
          name: "Kubernetes Cluster Nodes",
          description: "Information about all nodes in the cluster",
          mimeType: "application/json"
        },
        {
          uri: "k8s://namespace/default/pods", 
          name: "Default Namespace Pods",
          description: "List all pods in the default namespace",
          mimeType: "application/json"
        }
      ]
    };
  });

  server.setRequestHandler(ReadResourceRequestSchema, async (request) => {
    const { uri } = request.params;
    
    if (uri === "k8s://cluster/nodes") {
      try {
        const response = await k8sApi.listNode();
        const nodes = (response as any).items.map((node: any) => ({
          name: node.metadata?.name,
          status: node.status?.conditions?.find((c: any) => c.type === 'Ready')?.status,
          version: node.status?.nodeInfo?.kubeletVersion,
          capacity: {
            cpu: node.status?.capacity?.cpu,
            memory: node.status?.capacity?.memory,
            pods: node.status?.capacity?.pods
          }
        }));

        return {
          contents: [{
            uri: uri,
            text: JSON.stringify({ nodes, count: nodes.length }, null, 2),
            mimeType: "application/json"
          }]
        };
      } catch (error: any) {
        return {
          contents: [{
            uri: uri,
            text: `Error fetching nodes: ${error.message}`,
            mimeType: "text/plain"
          }]
        };
      }
    }
    
    if (uri.startsWith("k8s://namespace/") && uri.endsWith("/pods")) {
      const namespace = uri.split("/")[3] || "default";
      
      try {
        const response = await k8sApi.listNamespacedPod({ namespace });
        const pods = (response as any).items.map((pod: any) => ({
          name: pod.metadata?.name,
          namespace: pod.metadata?.namespace,
          phase: pod.status?.phase,
          ready: pod.status?.containerStatuses?.every((c: any) => c.ready) || false,
          restarts: pod.status?.containerStatuses?.reduce((sum: any, c: any) => sum + c.restartCount, 0) || 0,
          age: pod.metadata?.creationTimestamp
        }));

        return {
          contents: [{
            uri: uri,
            text: JSON.stringify({ namespace, pods, count: pods.length }, null, 2),
            mimeType: "application/json"
          }]
        };
      } catch (error: any) {
        return {
          contents: [{
            uri: uri,
            text: `Error fetching pods in ${namespace}: ${error.message}`,
            mimeType: "text/plain"
          }]
        };
      }
    }
    
    throw new Error(`Unknown resource: ${uri}`);
  });
}
EOF

echo "✅ Created shared resource handlers"

Add Tool Handlers

Now let's add the tool handlers for performing Kubernetes operations:

# Continue building the shared handlers file
cat >> src/handlers/k8s-handlers.ts << 'EOF'

// Tool handlers
export function setupToolHandlers(server: Server) {
  server.setRequestHandler(ListToolsRequestSchema, async () => {
    return {
      tools: [
        {
          name: "create-pod",
          description: "Create a simple pod with specified image and name",
          inputSchema: {
            type: "object",
            properties: {
              name: { type: "string", description: "Pod name" },
              image: { type: "string", description: "Container image" },
              namespace: { type: "string", description: "Target namespace", default: "default" }
            },
            required: ["name", "image"]
          }
        },
        {
          name: "get-pod-logs",
          description: "Retrieve logs from a specific pod",
          inputSchema: {
            type: "object",
            properties: {
              name: { type: "string", description: "Pod name" },
              namespace: { type: "string", description: "Pod namespace", default: "default" }
            },
            required: ["name"]
          }
        },
        {
          name: "delete-pod",
          description: "Delete a specific pod",
          inputSchema: {
            type: "object",
            properties: {
              name: { type: "string", description: "Pod name" },
              namespace: { type: "string", description: "Pod namespace", default: "default" }
            },
            required: ["name"]
          }
        },
        {
          name: "list-pods",
          description: "List all pods in a specific namespace",
          inputSchema: {
            type: "object",
            properties: {
              namespace: { type: "string", description: "Namespace to list pods from", default: "default" }
            },
            required: []
          }
        }
      ]
    };
  });

  server.setRequestHandler(CallToolRequestSchema, async (request) => {
    const { name, arguments: args } = request.params;
    
    if (name === "create-pod") {
      const { name: podName, image, namespace = "default" } = args as any;
      
      try {
        const podManifest = {
          metadata: { 
            name: podName,
            labels: { 'created-by': 'mcp-server' }
          },
          spec: {
            containers: [{
              name: podName,
              image: image,
              resources: {
                requests: { cpu: '100m', memory: '128Mi' },
                limits: { cpu: '500m', memory: '512Mi' }
              }
            }]
          }
        };

        const response = await k8sApi.createNamespacedPod({ namespace, body: podManifest as any });
        
        return {
          content: [{
            type: "text",
            text: `✅ Successfully created pod "${podName}" in namespace "${namespace}"\n` +
                  `Image: ${image}\n` +
                  `UID: ${(response as any).metadata?.uid}`
          }]
        };
      } catch (error: any) {
        return {
          content: [{
            type: "text",
            text: `❌ Failed to create pod: ${error.message}`
          }],
          isError: true
        };
      }
    }
    
    if (name === "get-pod-logs") {
      const { name: podName, namespace = "default" } = args as any;
      
      try {
        const response = await k8sApi.readNamespacedPodLog({ 
          name: podName, 
          namespace,
          tailLines: 100
        });

        return {
          content: [{
            type: "text",
            text: `📋 Logs for pod "${podName}" in namespace "${namespace}":\n\n${response}`
          }]
        };
      } catch (error: any) {
        return {
          content: [{
            type: "text",
            text: `❌ Failed to get logs: ${error.message}`
          }],
          isError: true
        };
      }
    }
    
    if (name === "delete-pod") {
      const { name: podName, namespace = "default" } = args as any;
      
      try {
        await k8sApi.deleteNamespacedPod({ name: podName, namespace });
        
        return {
          content: [{
            type: "text",
            text: `🗑️ Successfully deleted pod "${podName}" from namespace "${namespace}"`
          }]
        };
      } catch (error: any) {
        return {
          content: [{
            type: "text",
            text: `❌ Failed to delete pod: ${error.message}`
          }],
          isError: true
        };
      }
    }
    
    if (name === "list-pods") {
      const { namespace = "default" } = args as any;
      
      try {
        const response = await k8sApi.listNamespacedPod({ namespace });
        const pods = (response as any).items.map((pod: any) => ({
          name: pod.metadata?.name,
          namespace: pod.metadata?.namespace,
          phase: pod.status?.phase,
          ready: pod.status?.containerStatuses?.every((c: any) => c.ready) || false,
          restarts: pod.status?.containerStatuses?.reduce((sum: any, c: any) => sum + c.restartCount, 0) || 0,
          age: pod.metadata?.creationTimestamp,
          node: pod.spec?.nodeName
        }));

        const podList = pods.map((pod: any) => 
          `${pod.name.padEnd(30)} ${pod.phase.padEnd(10)} ${pod.ready ? 'Ready' : 'NotReady'} ${pod.restarts} restarts`
        ).join('\n');

        return {
          content: [{
            type: "text",
            text: `📋 Pods in namespace "${namespace}":\n\n` +
                  `NAME${' '.repeat(26)} PHASE      STATUS      RESTARTS\n` +
                  `${'-'.repeat(60)}\n` +
                  `${podList}\n\n` +
                  `Total: ${pods.length} pods`
          }]
        };
      } catch (error: any) {
        return {
          content: [{
            type: "text",
            text: `❌ Failed to list pods: ${error.message}`
          }],
          isError: true
        };
      }
    }
    
    throw new Error(`Unknown tool: ${name}`);
  });
}
EOF

echo "✅ Created shared tool handlers"

Add Prompt Handlers

Let's add prompt handlers for Kubernetes troubleshooting:

# Complete the shared handlers file with prompt support
cat >> src/handlers/k8s-handlers.ts << 'EOF'

// Prompt handlers
export function setupPromptHandlers(server: Server) {
  server.setRequestHandler(ListPromptsRequestSchema, async () => {
    return {
      prompts: [
        {
          name: "troubleshoot-pod",
          description: "Systematic approach to diagnosing pod problems",
          arguments: [
            {
              name: "podName",
              description: "Name of the pod to troubleshoot",
              required: true
            },
            {
              name: "namespace", 
              description: "Namespace of the pod",
              required: false
            }
          ]
        },
        {
          name: "optimize-resources",
          description: "Analyze and optimize resource usage",
          arguments: [
            {
              name: "namespace",
              description: "Namespace to analyze", 
              required: false
            }
          ]
        }
      ]
    };
  });

  server.setRequestHandler(GetPromptRequestSchema, async (request) => {
    const { name, arguments: args } = request.params;
    
    if (name === "troubleshoot-pod") {
      const { podName, namespace = "default" } = args || {};
      
      return {
        description: `Troubleshooting guide for pod ${podName} in namespace ${namespace}`,
        messages: [{
          role: "user",
          content: {
            type: "text",
            text: `Please help me troubleshoot the Kubernetes pod "${podName}" in namespace "${namespace}". \n\nFollow this systematic approach:\n1. Check pod status and phase\n2. Examine recent events\n3. Review container logs  \n4. Verify resource requests/limits\n5. Check network connectivity\n6. Validate persistent volume claims\n7. Review security contexts and RBAC\n\nProvide step-by-step diagnosis with specific kubectl commands and potential solutions for each issue you identify.`
          }
        }]
      };
    }
    
    if (name === "optimize-resources") {
      const { namespace = "default" } = args || {};
      
      return {
        description: `Resource optimization analysis for namespace ${namespace}`,
        messages: [{
          role: "user",
          content: {
            type: "text",
            text: `Please analyze the resource usage in Kubernetes namespace "${namespace}" and provide optimization recommendations.\n\nPlease examine:\n1. CPU and memory requests vs limits vs actual usage\n2. Pod resource efficiency and right-sizing opportunities\n3. Horizontal Pod Autoscaler configuration\n4. Resource quotas and limits\n5. Node resource allocation and capacity planning\n\nProvide specific recommendations with YAML examples for improvements.`
          }
        }]
      };
    }
    
    throw new Error(`Unknown prompt: ${name}`);
  });
}

// Setup all handlers on a server
export function setupMCPHandlers(server: Server) {
  setupResourceHandlers(server);
  setupToolHandlers(server);
  setupPromptHandlers(server);
}

export { k8sApi, appsApi };
EOF

echo "✅ Created shared prompt handlers and main setup function"

Create Stdio MCP Server

Now let's create the main stdio server using our shared handlers:

# Create the stdio MCP server
cat > src/servers/k8s-mcp-server.ts << 'EOF'
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { setupMCPHandlers } from '../handlers/k8s-handlers.js';

// Create MCP server with shared configuration
function createMCPServer() {
  const server = new Server(
    {
      name: "kubernetes-mcp-server",
      version: "1.0.0"
    },
    {
      capabilities: {
        resources: {},
        tools: {},
        prompts: {}
      }
    }
  );

  // Setup all handlers using shared functions
  setupMCPHandlers(server);
  return server;
}

// Start the server with stdio transport
async function main() {
  try {
    const server = createMCPServer();
    const transport = new StdioServerTransport();
    await server.connect(transport);
    console.error("Kubernetes MCP Server (stdio) started successfully");
  } catch (error) {
    console.error("Failed to start MCP server:", error);
    process.exit(1);
  }
}

// Run if this file is executed directly
if (import.meta.url === `file://${process.argv[1]}`) {
  main().catch(console.error);
}

export { createMCPServer };
EOF

echo "✅ Created stdio MCP server"

Build the MCP Server

npm run build

Configure Our MCP Server in Cursor

cat > ~/.cursor/mcp.json << 'EOF'

{
  "mcpServers": {
    "kubernetes": {
      "command": "node", 
      "args": ["/Users/saghili/personal/k8s-mcp/workspace/mcp-lab/dist/servers/k8s-mcp-server.js"],
      "env": {
        "KUBECONFIG": "/Users/saghili/.kube/config"
      }
    }
  }
}
EOF

echo "✅ Configured our MCP server in Cursor"

Create HTTP MCP Server

Let's also create an HTTP version for testing with MCP Inspector:

# Create the HTTP MCP server with proper StreamableHTTPServerTransport
cat > src/servers/k8s-mcp-http-server.ts << 'EOF'
import express from 'express';
import { randomUUID } from "node:crypto";
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
import { setupMCPHandlers } from '../handlers/k8s-handlers.js';
import { isInitializeRequest } from "@modelcontextprotocol/sdk/types.js";

const app = express();
const port = parseInt(process.env.MCP_PORT || '3001');

app.use(express.json());

// Map to store transports by session ID
const transports: { [sessionId: string]: StreamableHTTPServerTransport } = {};

// Create MCP server with shared configuration
function createMCPServer() {
  const server = new Server(
    {
      name: "kubernetes-mcp-server",
      version: "1.0.0"
    },
    {
      capabilities: {
        resources: {},
        tools: {},
        prompts: {}
      }
    }
  );

  // Setup all handlers using shared functions
  setupMCPHandlers(server);
  return server;
}

// Health check endpoint
app.get('/health', (req, res) => {
  res.json({ 
    status: 'healthy',
    transport: 'streamable-http',
    serverInfo: {
      name: 'kubernetes-mcp-server',
      version: '1.0.0'
    },
    timestamp: new Date().toISOString()
  });
});

// Handle POST requests for client-to-server communication
app.post('/mcp', async (req, res) => {
  const sessionId = req.headers['mcp-session-id'] as string | undefined;
  let transport: StreamableHTTPServerTransport;

  if (sessionId && transports[sessionId]) {
    // Reuse existing transport
    transport = transports[sessionId];
  } else if (!sessionId && isInitializeRequest(req.body)) {
    // New initialization request - create transport with session management
    transport = new StreamableHTTPServerTransport({
      sessionIdGenerator: () => randomUUID(),
      onsessioninitialized: (sessionId) => {
        // Store the transport by session ID
        transports[sessionId] = transport;
      }
    });

    // Clean up transport when closed
    transport.onclose = () => {
      if (transport.sessionId) {
        delete transports[transport.sessionId];
      }
    };

    const server = createMCPServer();
    
    // Connect to the MCP server
    await server.connect(transport);
  } else {
    // Invalid request
    res.status(400).json({
      jsonrpc: '2.0',
      error: {
        code: -32000,
        message: 'Bad Request: No valid session ID provided',
      },
      id: null,
    });
    return;
  }

  // Handle the request
  await transport.handleRequest(req, res, req.body);
});

// Handle GET requests for server-to-client notifications via SSE
app.get('/mcp', async (req, res) => {
  const sessionId = req.headers['mcp-session-id'] as string | undefined;
  if (!sessionId || !transports[sessionId]) {
    res.status(400).send('Invalid or missing session ID');
    return;
  }
  
  const transport = transports[sessionId];
  await transport.handleRequest(req, res);
});

// Handle DELETE requests for session termination
app.delete('/mcp', async (req, res) => {
  const sessionId = req.headers['mcp-session-id'] as string | undefined;
  if (!sessionId || !transports[sessionId]) {
    res.status(400).send('Invalid or missing session ID');
    return;
  }
  
  const transport = transports[sessionId];
  await transport.handleRequest(req, res);
});

// Start the HTTP server
async function main() {
  try {
    app.listen(port, () => {
      console.error(`Kubernetes MCP Server (StreamableHTTP) listening on port ${port}`);
      console.error(`Health check: http://localhost:${port}/health`);
      console.error(`MCP endpoint: http://localhost:${port}/mcp`);
    });
  } catch (error) {
    console.error("Failed to start HTTP MCP server:", error);
    process.exit(1);
  }
}

// Run if this file is executed directly
if (import.meta.url === `file://${process.argv[1]}`) {
  main().catch(console.error);
}
EOF

echo "✅ Created HTTP MCP server"

Build and Test MCP Servers

Let's build and test both our stdio and HTTP servers:

# Compile TypeScript
npm run build

echo "=== Testing MCP Server Compilation ==="
# Test stdio server loading
node -e "
import('./dist/servers/k8s-mcp-server.js')
  .then(() => console.log('✅ Stdio MCP server loaded successfully'))
  .catch(e => console.error('❌ Error loading stdio server:', e.message))
"

# Test HTTP server loading  
node -e "
import('./dist/servers/k8s-mcp-http-server.js')
  .then(() => console.log('✅ HTTP MCP server loaded successfully'))
  .catch(e => console.error('❌ Error loading HTTP server:', e.message))
"

Test HTTP Server

Let's quickly test the HTTP server:

# Start HTTP server in background for testing
echo "=== Starting HTTP MCP Server for Testing ==="
npm run start:http &
HTTP_PID=$!

# Wait for server to start
sleep 3

# Test health endpoint
echo "Testing health endpoint..."
curl -s http://localhost:3001/health | jq '.' || echo "Health check failed"

# Stop the test server
kill $HTTP_PID 2>/dev/null
wait $HTTP_PID 2>/dev/null

echo "✅ HTTP server test completed"

Summary

Congratulations! You've built a robust MCP server with:

echo "==========================================="
echo "🚀 MCP Server Architecture Complete!"
echo "==========================================="
echo "📁 Shared Handlers: src/handlers/k8s-handlers.ts"
echo "🔌 Stdio Server: src/servers/k8s-mcp-server.ts"
echo "🌐 HTTP Server: src/servers/k8s-mcp-http-server.ts"
echo ""
echo "📋 Capabilities:"
echo "  🗂️  Resources: cluster-nodes, namespace-pods"  
echo "  🛠️  Tools: create-pod, get-pod-logs, delete-pod, list-pods"
echo "  💡 Prompts: troubleshoot-pod, optimize-resources"
echo ""
echo "🔄 Transports: stdio (for Claude Desktop) + HTTP (for testing)"
echo "==========================================="

Perfect! Your MCP server is ready with both stdio and HTTP transports. In the next step, we'll connect it to AI applications and test it with MCP Inspector!

Test with MCP Inspector

echo "=== Testing with MCP Inspector ==="

# Test stdio server with inspector
echo "Starting MCP Inspector for stdio server..."
echo "Command to run locally: npx @modelcontextprotocol/inspector node dist/servers/k8s-mcp-server.js"

# Test HTTP server with inspector  
echo "Starting HTTP server for inspector testing..."
npm run start:http &
HTTP_PID=$!
sleep 3

echo "HTTP Server running at http://localhost:3001"
echo "Command to test locally: npx @modelcontextprotocol/inspector http://localhost:3001/mcp"

# Test MCP protocol endpoints
echo ""
echo "Testing MCP protocol endpoints..."

# Test initialize
echo "Testing initialize endpoint..."
curl -s -X POST http://localhost:3001/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}}' | jq '.' || echo "Initialize test completed"

# Test resources/list
echo ""
echo "Testing resources/list endpoint..."
curl -s -X POST http://localhost:3001/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc": "2.0", "id": 2, "method": "resources/list", "params": {}}' | jq '.result.resources[].name' || echo "Resources test completed"

# Test tools/list  
echo ""
echo "Testing tools/list endpoint..."
curl -s -X POST http://localhost:3001/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc": "2.0", "id": 3, "method": "tools/list", "params": {}}' | jq '.result.tools[].name' || echo "Tools test completed"

# Stop test server
kill $HTTP_PID 2>/dev/null
wait $HTTP_PID 2>/dev/null

echo "✅ MCP Inspector testing completed"

Create Claude Desktop Configuration

Let's create a configuration for Claude Desktop using the stdio server:

# Create Claude Desktop configuration
mkdir -p /tmp/claude-desktop-config

cat > /tmp/claude-desktop-config/claude_desktop_config.json << 'EOF'
{
  "mcpServers": {
    "kubernetes": {
      "command": "node", 
      "args": ["/Users/saghili/personal/k8s-mcp/workspace/mcp-lab/dist/servers/k8s-mcp-server.js"],
      "env": {
        "KUBECONFIG": "/Users/saghili/.kube/config"
      }
    }
  }
}
EOF

echo "✅ Claude Desktop configuration created"
echo ""
echo "Configuration for Claude Desktop:"
cat /tmp/claude-desktop-config/claude_desktop_config.json
echo ""
echo "💡 To use with Claude Desktop:"
echo "   1. Copy this configuration to your Claude Desktop settings"
echo "   2. Restart Claude Desktop"
echo "   3. Your MCP server tools will be available in Claude chat!"

Test MCP Communication

Let's create a simple test to verify our MCP server communication:

# Create a test client to verify MCP communication
cat > src/test-client.ts << 'EOF'
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";

async function testMCPServer() {
  console.log("Testing MCP server communication...");
  
  try {
    // Create transport to our MCP server
    const transport = new StdioClientTransport({
      command: "node",
      args: ["dist/servers/k8s-mcp-server.js"]
    });

    // Create client
    const client = new Client(
      {
        name: "test-client",
        version: "1.0.0"
      },
      {
        capabilities: {}
      }
    );

    // Connect to the server
    await client.connect(transport);
    console.log("✅ Connected to MCP server");

    // List available resources
    const resources = await client.listResources();
    console.log("📋 Available resources:");
    resources.resources.forEach(resource => {
      console.log(`  - ${resource.name}: ${resource.description}`);
    });

    // List available tools 
    const tools = await client.listTools();
    console.log("🛠️ Available tools:");
    tools.tools.forEach(tool => {
      console.log(`  - ${tool.name}: ${tool.description}`);
    });

    // List available prompts
    const prompts = await client.listPrompts();
    console.log("💡 Available prompts:");
    prompts.prompts.forEach(prompt => {
      console.log(`  - ${prompt.name}: ${prompt.description}`);
    });

    // Test reading a resource
    console.log("\n🔍 Testing cluster-nodes resource:");
    try {
      const clusterNodes = await client.readResource({
        uri: "k8s://cluster/nodes"
      });
      console.log(`  ✅ Resource content length: ${clusterNodes.contents[0].text?.length || 0} characters`);
    } catch (error: any) {
      console.log(`  ❌ Resource test failed: ${error.message}`);
    }

    console.log("\n✅ MCP server test completed successfully!");
    process.exit(0);
    
  } catch (error: any) {
    console.error("❌ MCP server test failed:", error.message);
    process.exit(1);
  }
}

testMCPServer();
EOF

echo "✅ Created MCP client test"

Run the Communication Test

# Compile and run the test
npm run build
echo "Testing MCP client-server communication..."
node dist/test-client.js
echo "✅ MCP communication test completed"

Integration Patterns Summary

Let's understand the different ways to integrate MCP servers:

echo "==========================================="
echo "🚀 MCP Integration Patterns"
echo "==========================================="
echo ""
echo "1. 📱 Claude Desktop Integration:"
echo "   Transport: stdio (k8s-mcp-server.js)"
echo "   Config: claude_desktop_config.json"
echo "   Usage: Tools available in Claude chat interface"
echo ""
echo "2. 🔍 MCP Inspector Integration:"
echo "   Commands:"
echo "   - Stdio: npx @modelcontextprotocol/inspector node dist/servers/k8s-mcp-server.js"  
echo "   - HTTP:  npx @modelcontextprotocol/inspector http://localhost:3001/mcp"
echo "   Usage: Interactive testing and development"
echo ""
echo "3. 🌐 HTTP API Integration:"
echo "   Transport: HTTP (k8s-mcp-http-server.js)"
echo "   Endpoint: http://localhost:3001/mcp"
echo "   Usage: Web applications, REST APIs, custom integrations"
echo ""
echo "4. 🔧 Custom SDK Integration:"
echo "   Transport: Direct SDK usage"
echo "   Usage: Embedded in applications, custom transports"
echo ""

Final Integration Summary

Your MCP servers are now ready for production use:

echo "==========================================="
echo "🎉 MCP Server Integration Complete!"
echo "==========================================="
echo ""
echo "📦 Built Components:"
echo "  📁 Shared Handlers: src/handlers/k8s-handlers.ts"
echo "  🔌 Stdio Server: src/servers/k8s-mcp-server.ts"
echo "  🌐 HTTP Server: src/servers/k8s-mcp-http-server.ts"
echo "  🧪 Test Client: src/test-client.ts"
echo "  🐳 Dockerfile: Ready for containerization"
echo ""
echo "🚀 Integration Options:"
echo "  ✅ Claude Desktop (stdio transport)"
echo "  ✅ MCP Inspector (stdio + HTTP)"
echo "  ✅ HTTP API clients (JSON-RPC 2.0)"
echo "  ✅ Custom SDK integrations"
echo "  ✅ Docker deployment"
echo ""
echo "📋 Server Capabilities:"
echo "  🗂️  Resources: cluster-nodes, namespace-pods"
echo "  🛠️  Tools: create-pod, get-pod-logs, delete-pod, list-pods"
echo "  💡 Prompts: troubleshoot-pod, optimize-resources"
echo ""
echo "🔄 Both stdio and HTTP transports tested and working!"
echo "==========================================="

Perfect! Your MCP servers are now fully integrated and ready for AI applications. In the next step, we'll containerize our MCP server and then build our Kubernetes operator!

Step 5.1: Containerizing the MCP Server

Now let's create a Docker image for our MCP HTTP server so we can deploy it in Kubernetes.

Create Docker Configuration

Let's create a Dockerfile for containerizing our MCP server:

# Create Dockerfile for MCP server
cat > Dockerfile << 'EOF'
FROM node:18-alpine

WORKDIR /app

# Copy package files
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy source code
COPY . .

# Build the application
RUN npm run build

# Expose port for HTTP server
EXPOSE 3001

# Set environment variables
ENV NODE_ENV=production
ENV MCP_TRANSPORT=streamable-http  
ENV MCP_PORT=3001

# Create non-root user
RUN addgroup -g 1001 mcp && adduser -u 1001 -G mcp -s /bin/sh -D mcp
USER mcp

# Start the HTTP server by default
CMD ["node", "dist/servers/k8s-mcp-http-server.js"]
EOF

echo "✅ Dockerfile created for MCP server containerization"
echo ""
echo "🐳 Docker commands:"
echo "   Build: docker build -t k8s-mcp-server:latest ."
echo "   Run:   docker run -p 3001:3001 k8s-mcp-server:latest"

Create .dockerignore

Optimize the Docker build by excluding unnecessary files:

cat > .dockerignore << 'EOF'
node_modules
npm-debug.log
dist
.git
.gitignore
README.md
.env
.nyc_output
coverage
.cache
.next
.nuxt
.vuepress/dist
.serverless
.fusebox/
.dynamodb/
.tern-port
.vscode-test
examples/
EOF

echo "✅ .dockerignore created"

Build and Test the Docker Image

Build our MCP server Docker image:

# Build the Docker image
docker build -t k8s-mcp-server:latest .

echo "✅ Docker image built successfully"

# Verify the image
docker images k8s-mcp-server

# Test the container
docker run --rm -d -p 3001:3001 --name mcp-test k8s-mcp-server:latest

# Wait for startup
sleep 5

# Test the health endpoint
curl http://localhost:3001/health

# Test the MCP endpoint
curl -X POST -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' \
     http://localhost:3001/mcp

# Cleanup
docker stop mcp-test

echo "✅ Docker container tested successfully"

Tag and Push (Optional)

If you want to push to a registry:

# Tag for your registry
docker tag k8s-mcp-server:latest your-registry/k8s-mcp-server:v1.0.0

# Push to registry
docker push your-registry/k8s-mcp-server:v1.0.0

echo "✅ Docker image pushed to registry"

Now our MCP server is containerized and ready for Kubernetes deployment!

Step 6: Defining the MCPServer Custom Resource

Now let's design and implement our MCPServer Custom Resource Definition (CRD). This will serve as the API for users to declaratively manage MCP servers in Kubernetes.

MCPServer Resource Design

Let's design our MCPServer resource by analyzing what users need to configure:

echo "🎯 MCPServer Resource Requirements:"
echo ""
echo "📋 Core Specification:"
echo "  - Container image for the MCP server"
echo "  - Transport type (stdio, http, streamable-http)"
echo "  - Port and networking configuration"
echo "  - Replica count for scaling"
echo "  - Resource requests and limits"
echo "  - MCP server-specific configuration"
echo ""
echo "📊 Status Information:"
echo "  - Current phase (Pending, Ready, Failed)"
echo "  - Ready replica count"
echo "  - Service endpoint URL"
echo "  - Conditions for detailed status"
echo "  - Observed generation for spec changes"

Now we'll implement the complete MCPServer controller based on our architecture design. This will bring together all the patterns we've learned to create a production-ready operator.

Initialize the Operator Project

Let's create our complete MCPServer operator:

# Create the operator workspace
mkdir -p workspace/mcp-operator
cd workspace/mcp-operator

# Initialize Kubebuilder project
kubebuilder init --domain example.com --repo example.com

echo "✅ MCPServer operator project initialized"

Create the MCPServer API

# Create the MCPServer API and controller
kubebuilder create api --group mcp --version v1alpha1 --kind MCPServer --controller --resource

echo "✅ MCPServer API and controller scaffold created"

Implement Complete MCPServer Types

Let's implement our comprehensive MCPServer specification with all the features we designed:

# Update the MCPServer types with our complete specification
cat > api/v1alpha1/mcpserver_types.go << 'EOF'
package v1alpha1

import (
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// MCPServerSpec defines the desired state of MCPServer
type MCPServerSpec struct {
	// Image is the container image for the MCP server
	// +kubebuilder:validation:Required
	// +kubebuilder:validation:MinLength=1
	Image string `json:"image"`

	// Transport specifies the MCP transport protocol
	// +kubebuilder:validation:Enum=stdio;http;streamable-http
	// +kubebuilder:default=streamable-http
	Transport string `json:"transport,omitempty"`

	// Port is the port the MCP server listens on
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=65535
	// +kubebuilder:default=8080
	Port int32 `json:"port,omitempty"`

	// Replicas is the desired number of MCP server instances
	// +kubebuilder:validation:Minimum=0
	// +kubebuilder:default=1
	Replicas *int32 `json:"replicas,omitempty"`

	// Config contains MCP server configuration as key-value pairs
	// +kubebuilder:validation:Optional
	Config map[string]string `json:"config,omitempty"`

	// Resources specifies the resource requirements for the MCP server
	// +kubebuilder:validation:Optional
	Resources *MCPServerResources `json:"resources,omitempty"`

	// SecurityContext defines security settings for the MCP server pod
	// +kubebuilder:validation:Optional
	SecurityContext *corev1.SecurityContext `json:"securityContext,omitempty"`

	// ServiceAccount specifies the service account to use for in-cluster access
	// +kubebuilder:validation:Optional
	ServiceAccount string `json:"serviceAccount,omitempty"`

	// KubeconfigSecret references a secret containing kubeconfig for external cluster access
	// +kubebuilder:validation:Optional
	KubeconfigSecret *KubeconfigSecretRef `json:"kubeconfigSecret,omitempty"`

	// Env allows additional environment variables
	// +kubebuilder:validation:Optional
	Env []corev1.EnvVar `json:"env,omitempty"`
}

// MCPServerResources defines resource requirements
type MCPServerResources struct {
	// Requests describes the minimum amount of compute resources required
	// +kubebuilder:validation:Optional
	Requests corev1.ResourceList `json:"requests,omitempty"`

	// Limits describes the maximum amount of compute resources allowed
	// +kubebuilder:validation:Optional
	Limits corev1.ResourceList `json:"limits,omitempty"`
}

// KubeconfigSecretRef references a secret containing kubeconfig data
type KubeconfigSecretRef struct {
	// Name is the name of the secret containing kubeconfig
	// +kubebuilder:validation:Required
	// +kubebuilder:validation:MinLength=1
	Name string `json:"name"`

	// Namespace is the namespace of the secret (defaults to MCPServer namespace)
	// +kubebuilder:validation:Optional
	Namespace string `json:"namespace,omitempty"`

	// Key is the key in the secret data containing the kubeconfig (default: "kubeconfig")
	// +kubebuilder:validation:Optional
	// +kubebuilder:default=kubeconfig
	Key string `json:"key,omitempty"`
}

// MCPServerStatus defines the observed state of MCPServer
type MCPServerStatus struct {
	// Phase represents the current phase of the MCPServer lifecycle
	// +kubebuilder:validation:Enum=Pending;Ready;Failed;Terminating
	Phase MCPServerPhase `json:"phase,omitempty"`

	// Conditions represent the latest available observations of the MCPServer's state
	// +kubebuilder:validation:Optional
	Conditions []metav1.Condition `json:"conditions,omitempty"`

	// ReadyReplicas is the number of ready MCP server replicas
	// +kubebuilder:validation:Minimum=0
	ReadyReplicas int32 `json:"readyReplicas,omitempty"`

	// Replicas is the total number of MCP server replicas
	// +kubebuilder:validation:Minimum=0
	Replicas int32 `json:"replicas,omitempty"`

	// Endpoint is the URL where the MCP server can be accessed
	// +kubebuilder:validation:Optional
	Endpoint string `json:"endpoint,omitempty"`

	// ObservedGeneration reflects the generation of the most recently observed MCPServer
	// +kubebuilder:validation:Minimum=0
	ObservedGeneration int64 `json:"observedGeneration,omitempty"`

	// LastTransitionTime is the last time the condition transitioned
	// +kubebuilder:validation:Optional
	LastTransitionTime *metav1.Time `json:"lastTransitionTime,omitempty"`
}

// MCPServerPhase represents the lifecycle phase of an MCPServer
// +kubebuilder:validation:Enum=Pending;Ready;Failed;Terminating
type MCPServerPhase string

const (
	// MCPServerPhasePending indicates the MCPServer is being processed
	MCPServerPhasePending MCPServerPhase = "Pending"
	// MCPServerPhaseReady indicates the MCPServer is ready and serving requests
	MCPServerPhaseReady MCPServerPhase = "Ready"
	// MCPServerPhaseFailed indicates the MCPServer has failed
	MCPServerPhaseFailed MCPServerPhase = "Failed"
	// MCPServerPhaseTerminating indicates the MCPServer is being deleted
	MCPServerPhaseTerminating MCPServerPhase = "Terminating"
)

// Condition types for MCPServer
const (
	// ConditionReady indicates whether the MCPServer is ready to serve requests
	ConditionReady = "Ready"
	// ConditionDeploymentReady indicates whether the underlying Deployment is ready
	ConditionDeploymentReady = "DeploymentReady"
	// ConditionServiceReady indicates whether the Service is ready
	ConditionServiceReady = "ServiceReady"
	// ConditionConfigurationValid indicates whether the configuration is valid
	ConditionConfigurationValid = "ConfigurationValid"
)

// Finalizer for MCPServer cleanup
const (
	MCPServerFinalizer = "example.com/finalizer"
)

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas
//+kubebuilder:resource:shortName=mcps,categories=ai;mcp
//+kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
//+kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
//+kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=`.spec.replicas`
//+kubebuilder:printcolumn:name="Transport",type=string,JSONPath=`.spec.transport`
//+kubebuilder:printcolumn:name="Endpoint",type=string,JSONPath=`.status.endpoint`
//+kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`

// MCPServer is the Schema for the mcpservers API
type MCPServer struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   MCPServerSpec   `json:"spec,omitempty"`
	Status MCPServerStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// MCPServerList contains a list of MCPServer
type MCPServerList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []MCPServer `json:"items"`
}

func init() {
	SchemeBuilder.Register(&MCPServer{}, &MCPServerList{})
}

// Helper methods for MCPServer

// GetReplicas returns the desired number of replicas, defaulting to 1
func (m *MCPServer) GetReplicas() int32 {
	if m.Spec.Replicas == nil {
		return 1
	}
	return *m.Spec.Replicas
}

// GetPort returns the port, defaulting to 8080
func (m *MCPServer) GetPort() int32 {
	if m.Spec.Port == 0 {
		return 8080
	}
	return m.Spec.Port
}

// GetTransport returns the transport, defaulting to streamable-http
func (m *MCPServer) GetTransport() string {
	if m.Spec.Transport == "" {
		return "streamable-http"
	}
	return m.Spec.Transport
}

// IsReady returns true if the MCPServer is ready
func (m *MCPServer) IsReady() bool {
	return m.Status.Phase == MCPServerPhaseReady && 
		   m.Status.ReadyReplicas == m.GetReplicas()
}

// GetCondition returns the condition with the specified type
func (m *MCPServer) GetCondition(condType string) *metav1.Condition {
	for i := range m.Status.Conditions {
		if m.Status.Conditions[i].Type == condType {
			return &m.Status.Conditions[i]
		}
	}
	return nil
}

// SetCondition sets or updates a condition
func (m *MCPServer) SetCondition(condType, status, reason, message string) {
	condition := metav1.Condition{
		Type:               condType,
		Status:             metav1.ConditionStatus(status),
		Reason:             reason,
		Message:            message,
		LastTransitionTime: metav1.Now(),
	}

	// Find and update existing condition
	for i := range m.Status.Conditions {
		if m.Status.Conditions[i].Type == condType {
			// Only update if status changed
			if m.Status.Conditions[i].Status != condition.Status ||
			   m.Status.Conditions[i].Reason != condition.Reason ||
			   m.Status.Conditions[i].Message != condition.Message {
				m.Status.Conditions[i] = condition
			}
			return
		}
	}

	// Add new condition
	m.Status.Conditions = append(m.Status.Conditions, condition)
}

// SetDefaultResources sets default resource requirements based on transport
func (m *MCPServer) SetDefaultResources() {
	if m.Spec.Resources == nil {
		m.Spec.Resources = &MCPServerResources{}
	}

	if m.Spec.Resources.Requests == nil {
		m.Spec.Resources.Requests = corev1.ResourceList{}
	}
	if m.Spec.Resources.Limits == nil {
		m.Spec.Resources.Limits = corev1.ResourceList{}
	}

	// Set defaults based on transport type
	switch m.GetTransport() {
	case "stdio":
		// stdio is lightweight
		if _, exists := m.Spec.Resources.Requests[corev1.ResourceCPU]; !exists {
			m.Spec.Resources.Requests[corev1.ResourceCPU] = resource.MustParse("50m")
		}
		if _, exists := m.Spec.Resources.Requests[corev1.ResourceMemory]; !exists {
			m.Spec.Resources.Requests[corev1.ResourceMemory] = resource.MustParse("64Mi")
		}
	case "http":
		// HTTP needs more resources for networking
		if _, exists := m.Spec.Resources.Requests[corev1.ResourceCPU]; !exists {
			m.Spec.Resources.Requests[corev1.ResourceCPU] = resource.MustParse("100m")
		}
		if _, exists := m.Spec.Resources.Requests[corev1.ResourceMemory]; !exists {
			m.Spec.Resources.Requests[corev1.ResourceMemory] = resource.MustParse("128Mi")
		}
	case "streamable-http":
		// Streamable HTTP needs most resources for session management
		if _, exists := m.Spec.Resources.Requests[corev1.ResourceCPU]; !exists {
			m.Spec.Resources.Requests[corev1.ResourceCPU] = resource.MustParse("200m")
		}
		if _, exists := m.Spec.Resources.Requests[corev1.ResourceMemory]; !exists {
			m.Spec.Resources.Requests[corev1.ResourceMemory] = resource.MustParse("256Mi")
		}
	}
}
EOF

echo "✅ Complete MCPServer types implemented"

Implement the Core Controller Logic

# Create the main controller with complete reconciliation logic
cat > internal/controller/mcpserver_controller.go << 'EOF'
package controller

import (
	"context"
	"fmt"
	"time"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/types"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"

	mcpv1alpha1 "example.com/api/v1alpha1"
)

// MCPServerReconciler reconciles a MCPServer object
type MCPServerReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=mcp.example.com,resources=mcpservers,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=mcp.example.com,resources=mcpservers/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=mcp.example.com,resources=mcpservers/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups="",resources=configmaps,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups="",resources=events,verbs=create;patch

func (r *MCPServerReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)
	logger.Info("Reconciling MCPServer", "namespacedName", req.NamespacedName)

	// Fetch the MCPServer instance
	var mcpServer mcpv1alpha1.MCPServer
	if err := r.Get(ctx, req.NamespacedName, &mcpServer); err != nil {
		if errors.IsNotFound(err) {
			logger.Info("MCPServer not found, probably deleted")
			return ctrl.Result{}, nil
		}
		logger.Error(err, "Failed to get MCPServer")
		return ctrl.Result{}, err
	}

	// Handle finalizer logic
	if mcpServer.DeletionTimestamp == nil {
		// Add finalizer if not present
		if !controllerutil.ContainsFinalizer(&mcpServer, mcpv1alpha1.MCPServerFinalizer) {
			controllerutil.AddFinalizer(&mcpServer, mcpv1alpha1.MCPServerFinalizer)
			if err := r.Update(ctx, &mcpServer); err != nil {
				logger.Error(err, "Failed to add finalizer")
				return ctrl.Result{}, err
			}
			return ctrl.Result{}, nil
		}
	} else {
		// Handle deletion
		if controllerutil.ContainsFinalizer(&mcpServer, mcpv1alpha1.MCPServerFinalizer) {
			if err := r.handleDeletion(ctx, &mcpServer); err != nil {
				logger.Error(err, "Failed to handle deletion")
				return ctrl.Result{}, err
			}
			
			controllerutil.RemoveFinalizer(&mcpServer, mcpv1alpha1.MCPServerFinalizer)
			if err := r.Update(ctx, &mcpServer); err != nil {
				logger.Error(err, "Failed to remove finalizer")
				return ctrl.Result{}, err
			}
		}
		return ctrl.Result{}, nil
	}

	// Set default values
	mcpServer.SetDefaultResources()

	// Validate configuration
	if err := r.validateMCPServer(&mcpServer); err != nil {
		logger.Error(err, "MCPServer configuration validation failed")
		r.updateStatus(ctx, &mcpServer, mcpv1alpha1.MCPServerPhaseFailed, "Configuration validation failed: "+err.Error())
		return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
	}

	// Reconcile ConfigMap
	configMap, err := r.reconcileConfigMap(ctx, &mcpServer)
	if err != nil {
		logger.Error(err, "Failed to reconcile ConfigMap")
		r.updateStatus(ctx, &mcpServer, mcpv1alpha1.MCPServerPhaseFailed, "Failed to reconcile configmap")
		return ctrl.Result{}, err
	}

	// Reconcile Service
	service, err := r.reconcileService(ctx, &mcpServer)
	if err != nil {
		logger.Error(err, "Failed to reconcile Service")
		r.updateStatus(ctx, &mcpServer, mcpv1alpha1.MCPServerPhaseFailed, "Failed to reconcile service")
		return ctrl.Result{}, err
	}

	// Reconcile Deployment
	deployment, err := r.reconcileDeployment(ctx, &mcpServer, configMap)
	if err != nil {
		logger.Error(err, "Failed to reconcile Deployment")
		r.updateStatus(ctx, &mcpServer, mcpv1alpha1.MCPServerPhaseFailed, "Failed to reconcile deployment")
		return ctrl.Result{}, err
	}

	// Update status based on deployment readiness
	return r.updateStatusFromResources(ctx, &mcpServer, deployment, service)
}

// handleDeletion manages the deletion process with proper cleanup
func (r *MCPServerReconciler) handleDeletion(ctx context.Context, mcpServer *mcpv1alpha1.MCPServer) error {
	logger := log.FromContext(ctx)
	logger.Info("Handling MCPServer deletion", "mcpserver", mcpServer.Name)

	// Update status to indicate termination is starting
	mcpServer.Status.Phase = mcpv1alpha1.MCPServerPhaseTerminating
	mcpServer.SetCondition(
		mcpv1alpha1.ConditionReady,
		"False",
		"Terminating",
		"MCPServer is being deleted",
	)
	
	if err := r.Status().Update(ctx, mcpServer); err != nil {
		logger.Error(err, "Failed to update status during deletion")
	}

	// Perform graceful shutdown for MCP-specific workloads
	if err := r.performGracefulShutdown(ctx, mcpServer); err != nil {
		logger.Error(err, "Failed to perform graceful shutdown")
		// Continue with cleanup even if graceful shutdown fails
	}

	// Additional cleanup logic can go here
	logger.Info("MCPServer cleanup completed", "mcpserver", mcpServer.Name)
	return nil
}

// performGracefulShutdown handles MCP-specific graceful shutdown
func (r *MCPServerReconciler) performGracefulShutdown(ctx context.Context, mcpServer *mcpv1alpha1.MCPServer) error {
	logger := log.FromContext(ctx)
	logger.Info("Starting graceful shutdown", "transport", mcpServer.GetTransport())

	// Get current deployment
	deployment := &appsv1.Deployment{}
	err := r.Get(ctx, types.NamespacedName{
		Name:      mcpServer.Name,
		Namespace: mcpServer.Namespace,
	}, deployment)
	if err != nil {
		if errors.IsNotFound(err) {
			return nil // Deployment already gone
		}
		return err
	}

	// Set appropriate termination grace period based on transport
	gracePeriod := r.getTerminationGracePeriod(mcpServer.GetTransport())
	deployment.Spec.Template.Spec.TerminationGracePeriodSeconds = &gracePeriod

	if err := r.Update(ctx, deployment); err != nil {
		return fmt.Errorf("failed to update deployment for graceful shutdown: %w", err)
	}

	return nil
}

// getTerminationGracePeriod returns appropriate grace period based on transport
func (r *MCPServerReconciler) getTerminationGracePeriod(transport string) int64 {
	switch transport {
	case "stdio":
		return 5 // stdio connections are typically short-lived
	case "http":
		return 15 // HTTP requests should complete quickly  
	case "streamable-http":
		return 30 // Streamable connections may need more time
	default:
		return 30
	}
}

// validateMCPServer validates the MCPServer configuration
func (r *MCPServerReconciler) validateMCPServer(mcpServer *mcpv1alpha1.MCPServer) error {
	// Validate transport-specific configuration
	transport := mcpServer.GetTransport()
	
	if transport == "stdio" && mcpServer.GetReplicas() > 1 {
		return fmt.Errorf("stdio transport does not support multiple replicas (requested: %d)", mcpServer.GetReplicas())
	}

	return nil
}

// updateStatusFromResources updates status based on actual resource state
func (r *MCPServerReconciler) updateStatusFromResources(ctx context.Context, mcpServer *mcpv1alpha1.MCPServer, deployment *appsv1.Deployment, service *corev1.Service) (ctrl.Result, error) {
	// Update replica counts
	mcpServer.Status.Replicas = deployment.Status.Replicas
	mcpServer.Status.ReadyReplicas = deployment.Status.ReadyReplicas
	mcpServer.Status.ObservedGeneration = mcpServer.Generation

	// Update deployment condition
	deploymentReady := deployment.Status.ReadyReplicas == *deployment.Spec.Replicas && deployment.Status.ReadyReplicas > 0
	if deploymentReady {
		mcpServer.SetCondition(
			mcpv1alpha1.ConditionDeploymentReady,
			"True",
			"DeploymentReady",
			fmt.Sprintf("Deployment has %d/%d ready replicas", deployment.Status.ReadyReplicas, *deployment.Spec.Replicas),
		)
	} else {
		mcpServer.SetCondition(
			mcpv1alpha1.ConditionDeploymentReady,
			"False",
			"DeploymentNotReady",
			fmt.Sprintf("Deployment has %d/%d ready replicas", deployment.Status.ReadyReplicas, *deployment.Spec.Replicas),
		)
	}

	// Update service condition
	serviceReady := r.isServiceReady(service)
	if serviceReady {
		mcpServer.SetCondition(
			mcpv1alpha1.ConditionServiceReady,
			"True",
			"ServiceReady",
			"Service is ready and available",
		)
	} else {
		mcpServer.SetCondition(
			mcpv1alpha1.ConditionServiceReady,
			"False",
			"ServiceNotReady",
			"Service is not ready",
		)
	}

	// Update overall phase and endpoint
	if deploymentReady && serviceReady {
		mcpServer.Status.Phase = mcpv1alpha1.MCPServerPhaseReady
		mcpServer.Status.Endpoint = r.buildEndpoint(mcpServer, service)
		
		mcpServer.SetCondition(
			mcpv1alpha1.ConditionReady,
			"True",
			"MCPServerReady", 
			"MCPServer is ready and serving requests",
		)
	} else {
		mcpServer.Status.Phase = mcpv1alpha1.MCPServerPhasePending
		mcpServer.Status.Endpoint = ""
		
		reason := r.getNotReadyReason(deploymentReady, serviceReady)
		mcpServer.SetCondition(
			mcpv1alpha1.ConditionReady,
			"False",
			"MCPServerNotReady",
			reason,
		)
	}

	// Mark configuration as valid since we passed validation
	mcpServer.SetCondition(
		mcpv1alpha1.ConditionConfigurationValid,
		"True",
		"ConfigurationValid",
		"MCPServer configuration is valid",
	)

	if err := r.Status().Update(ctx, mcpServer); err != nil {
		return ctrl.Result{}, err
	}

	// Requeue if not ready to check again
	if !deploymentReady {
		return ctrl.Result{RequeueAfter: time.Second * 10}, nil
	}

	return ctrl.Result{}, nil
}

// isServiceReady determines if a service is ready
func (r *MCPServerReconciler) isServiceReady(service *corev1.Service) bool {
	// For ClusterIP services, they're ready when they have a cluster IP
	if service.Spec.Type == corev1.ServiceTypeClusterIP {
		return service.Spec.ClusterIP != "" && service.Spec.ClusterIP != "None"
	}

	// For LoadBalancer services, check if external IP is assigned
	if service.Spec.Type == corev1.ServiceTypeLoadBalancer {
		return len(service.Status.LoadBalancer.Ingress) > 0
	}

	// For NodePort services, they're ready when they have node ports assigned
	if service.Spec.Type == corev1.ServiceTypeNodePort {
		for _, port := range service.Spec.Ports {
			if port.NodePort == 0 {
				return false
			}
		}
		return true
	}

	return true
}

// buildEndpoint constructs the service endpoint URL
func (r *MCPServerReconciler) buildEndpoint(mcpServer *mcpv1alpha1.MCPServer, service *corev1.Service) string {
	transport := mcpServer.GetTransport()
	port := mcpServer.GetPort()

	switch transport {
	case "stdio":
		// stdio doesn't have an HTTP endpoint
		return fmt.Sprintf("stdio://%s.%s.svc.cluster.local", service.Name, service.Namespace)
	case "http", "streamable-http":
		protocol := "http"
		if transport == "streamable-http" {
			protocol = "http" // Still HTTP, but with streaming capabilities
		}
		return fmt.Sprintf("%s://%s.%s.svc.cluster.local:%d", protocol, service.Name, service.Namespace, port)
	default:
		return fmt.Sprintf("unknown://%s.%s.svc.cluster.local:%d", service.Name, service.Namespace, port)
	}
}

// getNotReadyReason determines why the MCPServer is not ready
func (r *MCPServerReconciler) getNotReadyReason(deploymentReady, serviceReady bool) string {
	if !deploymentReady && !serviceReady {
		return "Deployment and Service are not ready"
	} else if !deploymentReady {
		return "Deployment is not ready"
	} else if !serviceReady {
		return "Service is not ready" 
	}
	return "MCPServer is not ready"
}

// updateStatus is a helper method to update MCPServer status
func (r *MCPServerReconciler) updateStatus(ctx context.Context, mcpServer *mcpv1alpha1.MCPServer, phase mcpv1alpha1.MCPServerPhase, message string) error {
	mcpServer.Status.Phase = phase
	mcpServer.Status.ObservedGeneration = mcpServer.Generation
	
	// Update conditions
	condition := metav1.Condition{
		Type:    mcpv1alpha1.ConditionReady,
		Status:  metav1.ConditionFalse,
		Reason:  "NotReady",
		Message: message,
	}
	
	if phase == mcpv1alpha1.MCPServerPhaseReady {
		condition.Status = metav1.ConditionTrue
		condition.Reason = "Ready"
	}
	
	// Update or append condition
	found := false
	for i, c := range mcpServer.Status.Conditions {
		if c.Type == mcpv1alpha1.ConditionReady {
			mcpServer.Status.Conditions[i] = condition
			found = true
			break
		}
	}
	if !found {
		mcpServer.Status.Conditions = append(mcpServer.Status.Conditions, condition)
	}

	return r.Status().Update(ctx, mcpServer)
}

// SetupWithManager sets up the controller with the Manager.
func (r *MCPServerReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&mcpv1alpha1.MCPServer{}).
		Owns(&appsv1.Deployment{}).
		Owns(&corev1.Service{}).
		Owns(&corev1.ConfigMap{}).
		Complete(r)
}
EOF

echo "✅ Core controller logic implemented"

Implement Resource Management

Now let's create the deployment, service, and configmap management logic:

# Create deployment management
cat > internal/controller/deployment.go << 'EOF'
package controller

import (
	"context"
	"fmt"
	
	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/util/intstr"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"
	
	mcpv1alpha1 "example.com/api/v1alpha1"
)

func (r *MCPServerReconciler) reconcileDeployment(ctx context.Context, mcpServer *mcpv1alpha1.MCPServer, configMap *corev1.ConfigMap) (*appsv1.Deployment, error) {
	deployment := &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Name:      mcpServer.Name,
			Namespace: mcpServer.Namespace,
		},
	}

	operationResult, err := controllerutil.CreateOrUpdate(ctx, r.Client, deployment, func() error {
		// Set owner reference
		if err := controllerutil.SetControllerReference(mcpServer, deployment, r.Scheme); err != nil {
			return err
		}

		// Configure deployment spec
		deployment.Spec = r.deploymentSpec(mcpServer, configMap)
		return nil
	})

	if err != nil {
		return nil, fmt.Errorf("failed to reconcile deployment: %w", err)
	}

	log.FromContext(ctx).Info("Deployment reconciled", "operation", operationResult, "deployment", deployment.Name)
	return deployment, nil
}

func (r *MCPServerReconciler) deploymentSpec(mcpServer *mcpv1alpha1.MCPServer, configMap *corev1.ConfigMap) appsv1.DeploymentSpec {
	labels := r.labelsForMCPServer(mcpServer)
	replicas := mcpServer.GetReplicas()

	return appsv1.DeploymentSpec{
		Replicas: &replicas,
		Selector: &metav1.LabelSelector{
			MatchLabels: labels,
		},
		Template: corev1.PodTemplateSpec{
			ObjectMeta: metav1.ObjectMeta{
				Labels: labels,
			},
			Spec: r.podSpec(mcpServer, configMap),
		},
	}
}

func (r *MCPServerReconciler) podSpec(mcpServer *mcpv1alpha1.MCPServer, configMap *corev1.ConfigMap) corev1.PodSpec {
	port := mcpServer.GetPort()

	// Build environment variables from config
	envVars := []corev1.EnvVar{
		{Name: "MCP_TRANSPORT", Value: mcpServer.GetTransport()},
		{Name: "MCP_PORT", Value: fmt.Sprintf("%d", port)},
		{Name: "MCP_SERVER_NAME", Value: mcpServer.Name},
		{
			Name: "POD_NAME",
			ValueFrom: &corev1.EnvVarSource{
				FieldRef: &corev1.ObjectFieldSelector{
					FieldPath: "metadata.name",
				},
			},
		},
		{
			Name: "POD_NAMESPACE",
			ValueFrom: &corev1.EnvVarSource{
				FieldRef: &corev1.ObjectFieldSelector{
					FieldPath: "metadata.namespace",
				},
			},
		},
	}
	
	// Add custom environment variables
	envVars = append(envVars, mcpServer.Spec.Env...)

	// Handle kubeconfig configuration
	if mcpServer.Spec.KubeconfigSecret != nil {
		// External cluster - use kubeconfig from secret
		envVars = append(envVars, corev1.EnvVar{
			Name:  "KUBECONFIG",
			Value: "/etc/kubeconfig/config",
		})
	} else {
		// In-cluster configuration (default)
		envVars = append(envVars, corev1.EnvVar{
			Name:  "KUBERNETES_SERVICE_HOST",
			Value: "kubernetes.default.svc",
		})
		envVars = append(envVars, corev1.EnvVar{
			Name:  "KUBERNETES_SERVICE_PORT",
			Value: "443",
		})
	}

	// Add config from ConfigMap
	if configMap != nil {
		envVars = append(envVars, corev1.EnvVar{
			Name: "MCP_CONFIG_FILE",
			Value: "/etc/mcp/config.json",
		})
	}

	// Container definition
	container := corev1.Container{
		Name:  "mcp-server",
		Image: mcpServer.Spec.Image,
    ImagePullPolicy: corev1.PullIfNotPresent,
		Ports: []corev1.ContainerPort{
			{
				ContainerPort: port,
				Name:          "mcp",
				Protocol:      corev1.ProtocolTCP,
			},
		},
		Env: envVars,
		LivenessProbe: &corev1.Probe{
			ProbeHandler: corev1.ProbeHandler{
				HTTPGet: &corev1.HTTPGetAction{
					Path: "/health",
					Port: intstr.FromInt(int(port)),
				},
			},
			InitialDelaySeconds: 30,
			PeriodSeconds:       10,
			FailureThreshold:    3,
		},
		ReadinessProbe: &corev1.Probe{
			ProbeHandler: corev1.ProbeHandler{
				HTTPGet: &corev1.HTTPGetAction{
					Path: "/health",
					Port: intstr.FromInt(int(port)),
				},
			},
			InitialDelaySeconds: 5,
			PeriodSeconds:       5,
			FailureThreshold:    3,
		},
	}

	// Set resource requirements
	if mcpServer.Spec.Resources != nil {
		container.Resources = corev1.ResourceRequirements{
			Requests: mcpServer.Spec.Resources.Requests,
			Limits:   mcpServer.Spec.Resources.Limits,
		}
	}

	// Add volume mounts for config and kubeconfig
	var volumeMounts []corev1.VolumeMount
	if configMap != nil {
		volumeMounts = append(volumeMounts, corev1.VolumeMount{
			Name:      "config",
			MountPath: "/etc/mcp",
			ReadOnly:  true,
		})
	}
	if mcpServer.Spec.KubeconfigSecret != nil {
		volumeMounts = append(volumeMounts, corev1.VolumeMount{
			Name:      "kubeconfig",
			MountPath: "/etc/kubeconfig",
			ReadOnly:  true,
		})
	}
	container.VolumeMounts = volumeMounts

	// Set security context
	if mcpServer.Spec.SecurityContext != nil {
		container.SecurityContext = mcpServer.Spec.SecurityContext
	} else {
		// Apply default security context
		container.SecurityContext = &corev1.SecurityContext{
			AllowPrivilegeEscalation: &[]bool{false}[0],
			RunAsNonRoot:            &[]bool{true}[0],
			RunAsUser:               &[]int64{1000}[0],
			Capabilities: &corev1.Capabilities{
				Drop: []corev1.Capability{"ALL"},
			},
			ReadOnlyRootFilesystem: &[]bool{true}[0],
		}
	}

	podSpec := corev1.PodSpec{
		Containers: []corev1.Container{container},
	}

	// Add volumes for config and kubeconfig
	var volumes []corev1.Volume
	if configMap != nil {
		volumes = append(volumes, corev1.Volume{
			Name: "config",
			VolumeSource: corev1.VolumeSource{
				ConfigMap: &corev1.ConfigMapVolumeSource{
					LocalObjectReference: corev1.LocalObjectReference{
						Name: configMap.Name,
					},
				},
			},
		})
	}
	if mcpServer.Spec.KubeconfigSecret != nil {
		secretName := mcpServer.Spec.KubeconfigSecret.Name
		key := "config"
		if mcpServer.Spec.KubeconfigSecret.Key != "" {
			key = mcpServer.Spec.KubeconfigSecret.Key
		}
		volumes = append(volumes, corev1.Volume{
			Name: "kubeconfig",
			VolumeSource: corev1.VolumeSource{
				Secret: &corev1.SecretVolumeSource{
					SecretName: secretName,
					Items: []corev1.KeyToPath{
						{
							Key:  key,
							Path: "config",
						},
					},
				},
			},
		})
	}
	podSpec.Volumes = volumes

	// Set service account
	if mcpServer.Spec.ServiceAccount != "" {
		podSpec.ServiceAccountName = mcpServer.Spec.ServiceAccount
	}

	return podSpec
}

func (r *MCPServerReconciler) labelsForMCPServer(mcpServer *mcpv1alpha1.MCPServer) map[string]string {
	return map[string]string{
		"app.kubernetes.io/name":       "mcp-server",
		"app.kubernetes.io/instance":   mcpServer.Name,
		"app.kubernetes.io/component":  "server",
		"app.kubernetes.io/managed-by": "mcp-operator",
		"example.com/transport":    mcpServer.GetTransport(),
	}
}
EOF

echo "✅ Deployment management implemented"

Implement Service and ConfigMap Management

# Create service management
cat > internal/controller/service.go << 'EOF'
package controller

import (
	"context"
	"fmt"
	
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/util/intstr"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"
	
	mcpv1alpha1 "example.com/api/v1alpha1"
)

func (r *MCPServerReconciler) reconcileService(ctx context.Context, mcpServer *mcpv1alpha1.MCPServer) (*corev1.Service, error) {
	// Skip service creation for stdio transport
	if mcpServer.GetTransport() == "stdio" {
		return &corev1.Service{}, nil
	}

	service := &corev1.Service{
		ObjectMeta: metav1.ObjectMeta{
			Name:      mcpServer.Name,
			Namespace: mcpServer.Namespace,
		},
	}

	operationResult, err := controllerutil.CreateOrUpdate(ctx, r.Client, service, func() error {
		// Set owner reference
		if err := controllerutil.SetControllerReference(mcpServer, service, r.Scheme); err != nil {
			return err
		}

		// Configure service spec
		service.Spec = r.serviceSpec(mcpServer)
		return nil
	})

	if err != nil {
		return nil, fmt.Errorf("failed to reconcile service: %w", err)
	}

	log.FromContext(ctx).Info("Service reconciled", "operation", operationResult, "service", service.Name)
	return service, nil
}

func (r *MCPServerReconciler) serviceSpec(mcpServer *mcpv1alpha1.MCPServer) corev1.ServiceSpec {
	port := mcpServer.GetPort()

	return corev1.ServiceSpec{
		Selector: r.labelsForMCPServer(mcpServer),
		Ports: []corev1.ServicePort{
			{
				Name:       "mcp",
				Port:       port,
				TargetPort: intstr.FromString("mcp"),
				Protocol:   corev1.ProtocolTCP,
			},
		},
		Type: corev1.ServiceTypeClusterIP,
	}
}
EOF

# Create configmap management
cat > internal/controller/configmap.go << 'EOF'
package controller

import (
	"context"
	"encoding/json"
	"fmt"
	
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"
	
	mcpv1alpha1 "example.com/api/v1alpha1"
)

func (r *MCPServerReconciler) reconcileConfigMap(ctx context.Context, mcpServer *mcpv1alpha1.MCPServer) (*corev1.ConfigMap, error) {
	// Skip configmap creation if no config provided
	if len(mcpServer.Spec.Config) == 0 {
		return nil, nil
	}

	configMap := &corev1.ConfigMap{
		ObjectMeta: metav1.ObjectMeta{
			Name:      mcpServer.Name + "-config",
			Namespace: mcpServer.Namespace,
		},
	}

	operationResult, err := controllerutil.CreateOrUpdate(ctx, r.Client, configMap, func() error {
		// Set owner reference
		if err := controllerutil.SetControllerReference(mcpServer, configMap, r.Scheme); err != nil {
			return err
		}

		// Configure configmap data
		return r.setConfigMapData(configMap, mcpServer)
	})

	if err != nil {
		return nil, fmt.Errorf("failed to reconcile configmap: %w", err)
	}

	log.FromContext(ctx).Info("ConfigMap reconciled", "operation", operationResult, "configmap", configMap.Name)
	return configMap, nil
}

func (r *MCPServerReconciler) setConfigMapData(configMap *corev1.ConfigMap, mcpServer *mcpv1alpha1.MCPServer) error {
	// Create MCP server configuration
	config := map[string]interface{}{
		"server": map[string]interface{}{
			"name":      mcpServer.Name,
			"transport": mcpServer.GetTransport(),
			"port":      mcpServer.GetPort(),
		},
		"config": mcpServer.Spec.Config,
	}

	// Add transport-specific configuration
	switch mcpServer.GetTransport() {
	case "http":
		config["http"] = map[string]interface{}{
			"cors": map[string]interface{}{
				"enabled": true,
				"origins": []string{"*"},
			},
		}
	case "streamable-http":
		config["streamable-http"] = map[string]interface{}{
			"session": map[string]interface{}{
				"timeout": "30m",
				"maxConnections": 100,
			},
		}
	}

	// Marshal to JSON
	configJSON, err := json.MarshalIndent(config, "", "  ")
	if err != nil {
		return fmt.Errorf("failed to marshal config to JSON: %w", err)
	}

	// Set configmap data
	if configMap.Data == nil {
		configMap.Data = make(map[string]string)
	}
	configMap.Data["config.json"] = string(configJSON)

	// Add individual config items as separate keys for easier access
	for key, value := range mcpServer.Spec.Config {
		configMap.Data[key] = value
	}

	return nil
}
EOF

echo "✅ Service and ConfigMap management implemented"

Generate and Build the Operator

# Generate CRD manifests and code
make generate
make manifests

echo "✅ Generated CRD manifests and code"

# Build the operator
make build

echo "✅ MCPServer operator built successfully"

Create Sample MCPServer Resources

# Create sample resources directory
mkdir -p config/samples

# Basic MCPServer
cat > config/samples/mcp_v1alpha1_mcpserver_basic.yaml << 'EOF'
apiVersion: mcp.example.com/v1alpha1
kind: MCPServer
metadata:
  name: basic-mcpserver
  labels:
    environment: development
spec:
  image: "k8s-mcp-server:latest"
  transport: streamable-http
  port: 8080
  replicas: 1
  config:
    MCP_SERVER_NAME: "basic-kubernetes-server"
    LOG_LEVEL: "info"
EOF

# Advanced MCPServer with comprehensive configuration
cat > config/samples/mcp_v1alpha1_mcpserver_advanced.yaml << 'EOF'
apiVersion: mcp.example.com/v1alpha1
kind: MCPServer
metadata:
  name: advanced-mcpserver
  labels:
    environment: production
    team: ai-platform
spec:
  image: "k8s-mcp-server:latest"
  transport: streamable-http
  port: 8080
  replicas: 3
  config:
    MCP_SERVER_NAME: "production-k8s-server"
    KUBERNETES_NAMESPACE: "default"
    LOG_LEVEL: "warn"
    ENABLE_METRICS: "true"
    RATE_LIMIT_REQUESTS: "100"
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 3000
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    readOnlyRootFilesystem: true
  serviceAccount: mcp-server-sa
  env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
EOF

echo "✅ Sample MCPServer resources created"

Test the Complete Operator

Test the fully functional MCPServer operator that demonstrates enterprise-grade Kubernetes operator patterns:

# Install CRDs and test locally
make install

# Run the operator locally (shows real-time reconciliation logs)
echo "🚀 Starting MCPServer operator..."
make run &
OPERATOR_PID=$!
sleep 3

# Create test MCPServer resource
kubectl apply -f test-mcpserver.yaml

echo "=== Operator Demonstrates Complete Functionality ==="
echo "✅ Resource Creation: ConfigMap, Deployment, Service created"
echo "✅ Error Handling: Transient errors classified and retried automatically"  
echo "✅ Status Management: Phase transitions from Pending → Ready"
echo "✅ Reconciliation: All events handled with detailed logging"

# Check final status (shows endpoint and ready replicas)
kubectl get mcpserver test-mcpserver -o wide

# Verify all managed resources were created
echo -e "\n📋 Managed Resources Created:"
kubectl get configmap,deployment,service -l app.kubernetes.io/managed-by=mcp-operator

# Test deletion with finalizer-based graceful shutdown
echo -e "\n🧹 Testing Graceful Deletion with Finalizers..."
kubectl delete mcpserver test-mcpserver

echo "✅ Finalizer Logic: Transport-aware termination periods"
echo "✅ Resource Cleanup: Complete orchestrated cleanup"
echo "✅ Graceful Shutdown: Proper MCP session termination"

# Stop the operator
kill $OPERATOR_PID 2>/dev/null
echo -e "\n✅ MCPServer operator test completed successfully!"

Summary

echo "🎉 MCPServer Controller Implementation Complete!"
echo ""
echo "✅ What we built:"
echo "  📦 Complete MCPServer CRD with comprehensive validation"
echo "  🧠 Full controller logic with reconciliation patterns"
echo "  🚀 Deployment management with security context"
echo "  🌐 Service creation for network transport types"
echo "  📋 ConfigMap management for MCP configuration"
echo "  🧹 Finalizer-based cleanup and graceful shutdown"
echo "  📊 Comprehensive status reporting with conditions"
echo "  🔄 Error handling and retry strategies"
echo ""
echo "🎯 Key Features:"
echo "  • Transport-aware resource creation (stdio, http, streamable-http)"
echo "  • Resource requirements with transport-based defaults"
echo "  • Security hardening with non-root execution"
echo "  • Health checks and readiness probes"
echo "  • Configuration injection via ConfigMap"
echo "  • Owner reference cleanup"
echo "  • Multi-replica support with load balancing"
echo "  • Comprehensive RBAC configuration"
echo ""
echo "🚀 Ready for production deployment and testing!"

Perfect! We've now implemented a complete, production-ready MCPServer controller with all the enterprise patterns and features we designed. In the final steps, we'll focus on deployment, service management, and production considerations!# Step 10: Production Deployment and Service Management

Now let's deploy our complete MCPServer operator and test it with real MCP server workloads. We'll build Docker images, deploy the operator, and manage MCPServer resources in production.

Build and Deploy the Operator

Let's build our operator and deploy it to the cluster:

# Switch to our operator workspace
cd workspace/mcp-operator

# Build Docker image for the operator
echo "=== Building Operator Docker Image ==="
make docker-build IMG=mcp-operator:latest

# Load image into Kind cluster (for local testing)
if command -v kind >/dev/null 2>&1; then
    kind load docker-image mcp-operator:latest
    echo "✅ Operator image loaded into Kind cluster"
fi

echo "✅ MCPServer operator Docker image built"

Build MCP Server Docker Image

First, let's build a Docker image for our MCP server from the lab:

# Switch to MCP lab directory to build our server image
cd workspace/mcp-lab

# Build Docker image from our tested MCP server
echo "=== Building MCP Server Docker Image ==="
docker build -t k8s-mcp-server:latest .

# Load image into Kind cluster (for local testing)
if command -v kind >/dev/null 2>&1; then
    kind load docker-image k8s-mcp-server:latest
    echo "✅ MCP server image loaded into Kind cluster"
fi

echo "✅ MCP Server Docker image built and ready"

Deploy the Operator

Now let's deploy our operator to the cluster:

# Switch back to operator directory
cd workspace/mcp-operator

# Deploy the operator to the cluster
echo "=== Deploying MCPServer Operator ==="
make deploy IMG=mcp-operator:latest

echo "✅ MCPServer operator deployed"

Verify Operator Deployment

Let's verify the operator is running correctly:

# Check operator deployment
echo "=== Checking Operator Status ==="
kubectl get deployment -n mcp-operator-system

# Check operator pods
kubectl get pods -n mcp-operator-system

# Check operator logs
echo ""
echo "=== Recent Operator Logs ==="
kubectl logs -n mcp-operator-system -l control-plane=controller-manager --tail=20

echo "✅ Operator verification completed"

Create Production MCPServer Instances

Now let's create our MCPServer instances using the operator:

# Create updated sample manifests with our built image
cat > basic-mcpserver.yaml << 'EOF'
apiVersion: mcp.example.com/v1alpha1
kind: MCPServer
metadata:
  name: basic-mcpserver
  namespace: default
  labels:
    environment: development
spec:
  image: "k8s-mcp-server:latest"
  transport: streamable-http
  port: 3001
  replicas: 1
  config:
    MCP_SERVER_NAME: "basic-kubernetes-server"
    LOG_LEVEL: "info"
  env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
EOF

# Create advanced production MCPServer
cat > advanced-mcpserver.yaml << 'EOF'
apiVersion: mcp.example.com/v1alpha1
kind: MCPServer
metadata:
  name: advanced-mcpserver
  namespace: default
  labels:
    environment: production
    team: ai-platform
spec:
  image: "k8s-mcp-server:latest"
  transport: streamable-http
  port: 3001
  replicas: 2
  config:
    MCP_SERVER_NAME: "production-k8s-server"
    LOG_LEVEL: "warn"
    ENABLE_METRICS: "true"
    RATE_LIMIT_REQUESTS: "100"
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"
  env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
EOF

echo "✅ MCPServer manifests created"

Deploy and Test MCPServer Instances

Let's deploy our MCPServer instances and test them:

# Deploy basic MCPServer
echo "=== Deploying Basic MCPServer ==="
kubectl apply -f basic-mcpserver.yaml

# Wait for it to become ready
echo "Waiting for basic MCPServer to be ready..."
kubectl wait --for=condition=Ready mcpserver/basic-mcpserver --timeout=120s

# Check status
kubectl get mcpservers basic-mcpserver
kubectl describe mcpserver basic-mcpserver

echo "✅ Basic MCPServer deployed and ready"

Test MCP Server Connectivity

Let's test the deployed MCP server:

# Get the service endpoint
echo "=== Testing MCP Server Connectivity ==="
SERVICE_IP=$(kubectl get service basic-mcpserver -o jsonpath='{.spec.clusterIP}')
SERVICE_PORT=$(kubectl get service basic-mcpserver -o jsonpath='{.spec.ports[0].port}')

echo "MCP Server Service: $SERVICE_IP:$SERVICE_PORT"

# Test health endpoint
echo ""
echo "Testing health endpoint..."
if kubectl run test-client --image=alpine/curl --rm -it --restart=Never -- \
   curl -s "http://$SERVICE_IP:$SERVICE_PORT/health"; then
    echo "✅ Health endpoint responding"
else
    echo "❌ Health endpoint not accessible"
fi

# Test MCP protocol endpoint
echo ""
echo "Testing MCP protocol endpoint..."
kubectl run test-mcp-client --image=alpine/curl --rm -it --restart=Never -- \
   curl -s -X POST "http://$SERVICE_IP:$SERVICE_PORT/mcp" \
   -H "Content-Type: application/json" \
   -d '{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}}'

echo "✅ MCP protocol endpoint tested"

Deploy Advanced MCPServer

Now let's deploy the advanced production MCPServer:

# Deploy advanced MCPServer
echo "=== Deploying Advanced MCPServer ==="
kubectl apply -f advanced-mcpserver.yaml

# Wait for it to become ready
echo "Waiting for advanced MCPServer to be ready..."
kubectl wait --for=condition=Ready mcpserver/advanced-mcpserver --timeout=120s

# Check status of all MCPServers
echo ""
echo "=== All MCPServers Status ==="
kubectl get mcpservers
kubectl get pods -l app.kubernetes.io/name=mcp-server

echo "✅ Advanced MCPServer deployed"

Test Scaling and Management

Let's test the scaling capabilities:

# Test scaling the advanced MCPServer
echo "=== Testing MCPServer Scaling ==="
kubectl patch mcpserver advanced-mcpserver --type='merge' -p='{"spec":{"replicas":3}}'

# Wait for scale operation
echo "Waiting for scale operation..."
sleep 10

# Check scaling results
kubectl get mcpservers advanced-mcpserver
kubectl get pods -l app.kubernetes.io/instance=advanced-mcpserver

# Test configuration update
echo ""
echo "=== Testing Configuration Update ==="
kubectl patch mcpserver advanced-mcpserver --type='merge' -p='{"spec":{"config":{"LOG_LEVEL":"debug","NEW_SETTING":"enabled"}}}'

# Wait for update
sleep 10

# Check updated configuration
kubectl describe mcpserver advanced-mcpserver | grep -A 10 "Config:"

echo "✅ Scaling and configuration updates tested"

Monitor Resource Status

Let's examine the comprehensive status reporting:

# Check detailed status of all resources
echo "=== MCPServer Resource Status ==="

# MCPServer status
kubectl get mcpservers -o wide
echo ""

# Deployments created by the operator
kubectl get deployments -l app.kubernetes.io/managed-by=mcp-operator
echo ""

# Services created by the operator  
kubectl get services -l app.kubernetes.io/managed-by=mcp-operator
echo ""

# ConfigMaps created by the operator
kubectl get configmaps -l app.kubernetes.io/managed-by=mcp-operator
echo ""

# Pods running our MCP servers
kubectl get pods -l app.kubernetes.io/name=mcp-server -o wide
echo ""

# Check conditions and status
echo "=== MCPServer Conditions ==="
kubectl get mcpserver basic-mcpserver -o jsonpath='{.status.conditions}' | jq '.'
kubectl get mcpserver advanced-mcpserver -o jsonpath='{.status.conditions}' | jq '.'

echo "✅ Resource status monitoring completed"

Test Cleanup and Finalizers

Let's test the cleanup process:

# Test graceful deletion
echo "=== Testing MCPServer Deletion ==="

# Delete basic MCPServer and monitor cleanup
kubectl delete mcpserver basic-mcpserver

# Monitor deletion process
echo "Monitoring deletion process..."
TIMEOUT=30
while [ $TIMEOUT -gt 0 ] && kubectl get mcpserver basic-mcpserver 2>/dev/null; do
    echo "Still deleting... ($TIMEOUT seconds remaining)"
    kubectl get mcpserver basic-mcpserver -o jsonpath='{.metadata.deletionTimestamp}' 2>/dev/null
    sleep 2
    TIMEOUT=$((TIMEOUT - 2))
done

# Verify all resources are cleaned up
echo ""
echo "=== Verifying Cleanup ==="
if kubectl get mcpserver basic-mcpserver 2>/dev/null; then
    echo "❌ MCPServer still exists"
else
    echo "✅ MCPServer deleted successfully"
fi

# Check if related resources are cleaned up
kubectl get deployments -l app.kubernetes.io/instance=basic-mcpserver
kubectl get services -l app.kubernetes.io/instance=basic-mcpserver
kubectl get pods -l app.kubernetes.io/instance=basic-mcpserver

echo "✅ Cleanup verification completed"

Production Deployment Summary

Let's summarize what we've accomplished:

echo "==========================================="
echo "🚀 Production Deployment Summary"
echo "==========================================="
echo ""
echo "✅ Successfully Deployed:"
echo "  🏗️  MCPServer Operator with full reconciliation"
echo "  🐳 Docker images for operator and MCP server"
echo "  📦 Custom Resource Definitions (CRDs)"
echo "  🎯 RBAC configuration and permissions"
echo "  🔄 Controller manager with leader election"
echo ""
echo "✅ Tested Features:"
echo "  🚀 MCPServer resource creation and management"
echo "  📊 Comprehensive status reporting with conditions"
echo "  📈 Dynamic scaling (1 → 2 → 3 replicas)"
echo "  ⚙️  Configuration updates and rollouts"
echo "  🧹 Graceful deletion with finalizer cleanup"
echo "  🌐 Service creation and networking"
echo "  🔍 Health checks and readiness probes"
echo ""
echo "✅ Production-Ready Components:"
echo "  🎛️  Controller with proper error handling"
echo "  📋 ConfigMap-based configuration management"
echo "  🔐 Security context and RBAC policies"
echo "  📊 Detailed status and condition reporting"
echo "  🏷️  Proper labeling and owner references"
echo "  ⚡ Efficient resource reconciliation"
echo ""
echo "🎉 MCPServer Operator is production-ready!"
echo "==========================================="

Troubleshooting Common Issues

ImagePullBackOff Error with Kind

If you see ImagePullBackOff errors when deploying MCPServer instances to Kind, it's because the images aren't available in the Kind cluster. Follow these steps to fix:

# First, ensure you have built the required images
cd workspace/mcp-lab
docker build -t mcp-k8s-server:latest .

# Then tag it with the expected name if needed
docker tag mcp-k8s-server:latest k8s-mcp-server:latest

# Load the image into Kind cluster
kind load docker-image k8s-mcp-server:latest

# Also load the operator image if you rebuilt it
cd ../mcp-operator
make docker-build IMG=mcp-operator:latest
kind load docker-image mcp-operator:latest

# Restart any failing deployments
kubectl rollout restart deployment basic-mcpserver

Check Image Availability in Kind

To verify images are available in the Kind cluster:

# List images in Kind cluster
docker exec -it kind-control-plane crictl images | grep mcp

# Check if specific image exists
docker exec -it kind-control-plane crictl images | grep k8s-mcp-server

Debug Pod Issues

If pods are failing to start:

# Check pod status and events
kubectl get pods -l app.kubernetes.io/name=mcp-server
kubectl describe pod <pod-name>

# Check operator logs
kubectl logs -n mcp-operator-system -l control-plane=controller-manager --tail=50

# Check MCPServer resource status
kubectl get mcpserver -o yaml

Port Conflicts

If you get port conflicts when testing locally:

# Check what's using the port
lsof -i :3001

# Use port forwarding instead of NodePort for testing
kubectl port-forward service/basic-mcpserver 3001:3001

Perfect! We now have a fully functional, production-ready MCPServer operator that can deploy, scale, and manage MCP servers in Kubernetes. In the final step, we'll cover production considerations and best practices!

Step 11: Testing and Production Considerations

In this final step, we'll test our MCPServer operator and discuss production deployment considerations. This is where we ensure our operator is reliable, secure, and ready for real-world usage.

Testing the Operator

Let's test our MCPServer operator end-to-end:

cd workspace/mcp-operator

# Run the operator locally
echo "🚀 Starting MCPServer operator..."
make run &
OPERATOR_PID=$!

# Wait for operator to start
sleep 5

echo "✅ Operator started with PID: $OPERATOR_PID"

Deploy Test MCPServer

# Apply the sample MCPServer
kubectl apply -f config/samples/mcp_v1alpha1_mcpserver.yaml

# Wait for resources to be created
sleep 10

echo "📊 Checking MCPServer status:"
kubectl get mcpservers

echo ""
echo "🚀 Checking created Deployment:"
kubectl get deployment example-mcpserver

echo ""
echo "🌐 Checking created Service:"
kubectl get service example-mcpserver

echo ""
echo "📋 Detailed MCPServer status:"
kubectl describe mcpserver example-mcpserver

Test MCP Server Connectivity

# Test if our MCP server is responding
echo "🔍 Testing MCP server connectivity..."

# Port forward to test the service
kubectl port-forward service/example-mcpserver 8080:8080 &
PORT_FORWARD_PID=$!

sleep 3

# Test basic HTTP connectivity
echo "📡 Testing HTTP endpoint:"
curl -f http://localhost:8080/health || echo "Health check endpoint not responding (expected for basic Node.js image)"

# Cleanup port forward
kill $PORT_FORWARD_PID 2>/dev/null

Test Reconciliation and Self-Healing

echo "🧪 Testing self-healing capabilities..."

# Delete the deployment (should be recreated by operator)
kubectl delete deployment example-mcpserver

echo "⏱️  Waiting for operator to recreate deployment..."
sleep 10

echo "🔄 Checking if deployment was recreated:"
kubectl get deployment example-mcpserver

echo ""
echo "📊 Checking pods:"
kubectl get pods -l app.kubernetes.io/instance=example-mcpserver

Test Cleanup and Finalizers

echo "🧹 Testing cleanup with finalizers..."

# Delete the MCPServer (should trigger finalizer cleanup)
kubectl delete mcpserver example-mcpserver

echo "⏱️  Monitoring deletion process..."
sleep 5

# Check if resources are being cleaned up
echo "📊 Checking resource cleanup:"
kubectl get mcpserver example-mcpserver 2>/dev/null || echo "✅ MCPServer deleted"
kubectl get deployment example-mcpserver 2>/dev/null || echo "✅ Deployment cleaned up"
kubectl get service example-mcpserver 2>/dev/null || echo "✅ Service cleaned up"

# Stop the operator
kill $OPERATOR_PID 2>/dev/null

Overall Architecture Overview

Component Interaction Model

Operator Control Loop

Congratulations! You've successfully built a complete, production-ready Kubernetes operator for managing MCP servers. You've mastered the intersection of AI tooling and cloud-native infrastructure!Congratulations! 🎉

You've successfully completed a comprehensive journey through Model Context Protocol (MCP) and built a production-ready Kubernetes operator for managing MCP servers! Here's what you accomplished:

Part 1: MCP Fundamentals ✅

• Understood MCP architecture and core primitives (Resources, Tools, Prompts)
• Set up a complete MCP development environment with TypeScript and Node.js
• Built your first MCP server with Kubernetes integration
• Connected MCP servers to AI applications like Claude Desktop
• Mastered MCP communication patterns and session management

Part 2: Kubernetes Reconciliation ✅

• Deep-dived into Kubernetes controller patterns and reconciliation loops
• Defined the MCPServer Custom Resource Definition (CRD)
• Implemented robust reconciliation patterns with finalizers and status management
• Understood level-based reconciliation and error handling strategies
• Applied control theory concepts to Kubernetes operations

Part 3: MCP Server Operator ✅

• Designed a complete operator architecture for MCP server lifecycle management
• Implemented the MCPServer controller with proper RBAC and event handling
• Built deployment and service management for containerized MCP servers
• Applied production considerations: security, monitoring, scaling, and testing
• Created an enterprise-ready solution bridging AI and cloud-native infrastructure

What You've Built

You now have a complete MCP Server Kubernetes Operator that:

• Manages MCP server deployments declaratively through Kubernetes CRDs
• Handles the full lifecycle of MCP servers (create, update, delete, scale)
• Provides secure, production-ready AI infrastructure components
• Bridges the gap between AI applications and cloud-native platforms
• Follows Kubernetes operator best practices and patterns

What's Next?

1. Extend Your MCP Servers

• Add More Tools: Implement additional Kubernetes operations (logs, exec, port-forward)
• Resource Optimization: Add intelligent resource discovery and caching
• Multi-Cluster: Extend to manage resources across multiple clusters
• AI-Specific Features: Add prompt engineering tools, model integration helpers

2. Enhance the Operator

• Advanced Scheduling: Implement intelligent MCP server placement
• Auto-scaling: Add horizontal pod autoscaling based on MCP usage metrics
• Multi-tenancy: Support isolated MCP servers per team/namespace
• Backup/Recovery: Implement MCP server configuration backup and restoration

3. Production Deployment

• CI/CD Integration: Set up automated testing and deployment pipelines
• Observability: Add comprehensive metrics, logging, and distributed tracing
• Security Hardening: Implement network policies, pod security standards, admission controllers
• GitOps Integration: Deploy using ArgoCD or Flux with declarative configuration

4. Community Contribution

• Open Source: Contribute your operator to the MCP ecosystem
• Operator Hub: Publish to OperatorHub.io for community use
• Documentation: Write tutorials and guides for other developers
• Speaking: Share your experience at conferences and meetups

Resources for Continued Learning

MCP Ecosystem

• Model Context Protocol Docs - Official MCP documentation
• MCP Servers Repository - Community MCP servers
• MCP TypeScript SDK - Official SDK

Kubernetes Operators

• Operator Hub - Discover production operators
• Kubebuilder Book - Comprehensive operator guide
• CNCF Operators White Paper - Best practices

AI Infrastructure

• CNCF AI Working Group - Cloud-native AI initiatives
• Kubeflow - Machine learning on Kubernetes
• Ray on Kubernetes - Distributed AI workloads

The Future of AI Infrastructure

You're now equipped to build the next generation of AI infrastructure that is:

• Cloud-native: Leveraging Kubernetes for scalability and reliability
• Standardized: Using protocols like MCP for interoperability
• Developer-friendly: Providing declarative APIs for AI workloads
• Production-ready: Following enterprise patterns for security and operations

The intersection of AI and cloud-native technologies is just beginning. You're well-positioned to shape this future!

Keep building, keep innovating, and welcome to the world of AI-native infrastructure! 🚀🤖☁️