Rhoai Workbenches
This project is meant as a bachelors thesis about MCP servers in RHOAI
Installation
npx mcp-server-rhoai-workbenchesAsk AI about Rhoai Workbenches
Powered by Claude Β· Grounded in docs
I know everything about Rhoai Workbenches. Ask me about installation, configuration, usage, or troubleshooting.
0/500
Reviews
Documentation
MCP Server for Red Hat OpenShift AI
This project implements a Model Context Protocol (MCP) server for managing Red Hat OpenShift AI (RHOAI) workbenches. It provides tools for listing, creating, and managing workbenches, images, hardware profiles, and storage in OpenShift clusters, also it provides tools for monitoring resource consumption by workbenches per user, workbench, namespace or cluster. This project is part of a Bachelor's thesis.
Prerequisites
- Go 1.24.0 or later
- Access to an OpenShift cluster with RHOAI installed
ocCLI logged in to your cluster- (Optional) golangci-lint for code linting
Installation
- Clone the repository:
git clone https://github.com/amaly/mcp-server-rhoai.git
cd mcp-server-rhoai
- Download dependencies:
go mod download
- Ensure you're logged in to your OpenShift cluster:
oc login <your-cluster-url>
Building
The project includes a Makefile for easy building:
Build with Make
# Build with linting and testing (recommended)
make build
# Build without running lint and tests (faster)
go build -o mcp-server-rhoai
The build process will:
- Run linting checks (golangci-lint)
- Execute all tests
- Compile the binary as
mcp-server-rhoai
Build Outputs
- Binary location:
./mcp-server-rhoai
Testing
The project has a comprehensive unit test suite (60 tests across 7 test files) covering every tool module. Tests use Kubernetes fake clients (dynamicfake.NewSimpleDynamicClient) and function-variable mocking to isolate logic from live cluster dependencies, so no running cluster is needed.
Run tests using Make or directly with Go:
# Run all tests with Make
make test
# Run tests directly with verbose output
go test -v ./tools/... ./resources/... ./prompts/...
Test coverage includes:
- Workbench tools β CRUD operations, status transitions, and helper functions (image tag parsing, resource extraction, PVC name resolution)
- Storage tools β PVC creation, deletion, and size updates including rejection of size decreases
- Image tools β Custom image CRUD, default-image and in-use protection checks, image listing
- Hardware profile tools β Profile CRUD and resource identifier parsing
- Resource consumption tools β Value parsing (millicores, Gi suffixes, decimals), aggregation, and per-workbench/namespace/user/cluster queries
- Namespace & pod tools β Listing and filtering by namespace
Each tool module has a dedicated *_test.go file with both success and error-path cases. Table-driven subtests are used where multiple input variations need to be verified.
Evaluation
This project uses Promptfoo for automated evaluation of tool selection and execution quality. Promptfoo is an open-source testing framework specifically designed for evaluating LLM applications, tool calling, and agent behavior. It provides metrics-based testing with visual reports.
Run Evaluation
# Run evaluation tests
make eval
# Run evaluation and view results in browser
make eval-view
API Key Requirements:
The evaluation uses AI models to test the MCP server. You'll need an API key for one of the supported providers:
-
OpenAI (default): Set
OPENAI_API_KEYenvironment variableexport OPENAI_API_KEY=your-key-here make eval -
Anthropic Claude: Uncomment the Claude provider in
promptfoo.yamland setANTHROPIC_API_KEYexport ANTHROPIC_API_KEY=your-key-here make eval -
Google Gemini: Uncomment the Gemini provider in
promptfoo.yamland setGOOGLE_API_KEYexport GOOGLE_API_KEY=your-key-here make eval
The evaluation measures:
- Tool Selection Accuracy - Does the system choose the correct tool for each prompt?
- Parameter Extraction - Are the tool parameters extracted correctly?
- Execution Success Rate - Do the tools execute without errors?
Results are displayed in an interactive web UI showing pass/fail rates, detailed comparisons, and performance metrics.
Prerequisites: Node.js (^20.20.0 or >=22.22.0) and npm must be installed.
Evaluation Findings
Evaluation was run against Google Gemini models (2.5 Flash and 2.5 Pro). Key findings:
| Test Case | Gemini 2.5 Flash | Gemini 2.5 Pro |
|---|---|---|
| List workbenches in a namespace | PASS | PASS |
| Resource consumption per namespace | PASS | PASS |
| List all workbenches across namespaces | FAIL | FAIL |
| Create a workbench | FAIL | FAIL |
Observations:
-
list_all_workbenchestool confusion β Both models refused to call this tool despite it being available. The tool's input schema requires anamespaceparameter, which contradicts its description ("list workbenches across all namespaces"). Models interpret this inconsistency as a reason to fall back to the per-namespacelist_workbenchestool or refuse entirely. -
create_workbenchtool refusal β Both models declined to callcreate_workbench, claiming they could only list resources. The tool has a complex input schema with many required fields (hardwareProfileas a nested object,imageTag,pvcName, etc.). When the test prompt does not supply all required parameters, the models choose not to attempt the call rather than inferring or requesting missing values. -
Model quality vs. cost trade-off β Gemini 2.5 Flash and 2.5 Pro produced identical results for tool selection tasks. Lighter models may struggle with complex tool schemas. Note that
gemini-2.0-flashhas been deprecated for new API users as of April 2026. -
Free tier limitations β The Google Gemini free tier quota can be exhausted quickly. When the quota is hit (
limit: 0), promptfoo enters an infinite retry loop. A pay-as-you-go billing plan is recommended for running evaluations reliably.
Recommendations for improving eval pass rates:
- Remove the contradictory
namespaceparameter from thelist_all_workbenchesinput schema, or rename the tool to clarify its behavior. - Enrich test prompts for write operations to include all required parameters (e.g., hardware profile, image tag, PVC name), or simplify the tool's required fields so the model can attempt the call with partial information.
- Consider testing with OpenAI GPT-4o or Anthropic Claude, which may handle complex tool schemas more reliably.
Configuration
The MCP server can be configured in AI assistants to enable workbench management through natural language.
VSCode/Cursor Setup
- Build the server:
make build
-
Open MCP Settings in Cursor:
- Press
Ctrl + Shift + P(Windows/Linux) orCmd + Shift + P(Mac) - Type "Open MCP Settings"
- Click "New MCP server"
- Press
-
Add configuration:
Read-only mode (default, safer):
{
"mcpServers": {
"mcp-server-rhoai": {
"command": "/home/amaly/mcp-server-rhoai/mcp-server-rhoai"
}
}
}
Read-write mode (full access):
{
"mcpServers": {
"mcp-server-rhoai": {
"command": "/home/amaly/mcp-server-rhoai/mcp-server-rhoai",
"env": {
"MCP_RHOAI_MODE": "write"
}
}
}
}
-
Verify installation:
- Check that the server is enabled in Cursor MCP settings
- You should see available tools listed
-
Start using:
- Ask the AI assistant: "List all workbenches in namespace mcp-test"
- Or: "Show me resource consumption per namespace"
Claude Desktop Setup
- Build the server:
make build
-
Locate Claude Desktop config file:
- macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json - Linux:
~/.config/Claude/claude_desktop_config.json
- macOS:
-
Edit the config file and add the MCP server:
Read-only mode (default):
{
"mcpServers": {
"mcp-server-rhoai": {
"command": "/home/amaly/mcp-server-rhoai/mcp-server-rhoai"
}
}
}
Read-write mode (full access):
{
"mcpServers": {
"mcp-server-rhoai": {
"command": "/home/amaly/mcp-server-rhoai/mcp-server-rhoai",
"env": {
"MCP_RHOAI_MODE": "write"
}
}
}
}
-
Restart Claude Desktop for changes to take effect
-
Verify connection:
- Look for the MCP connection indicator in Claude Desktop
- Ask Claude: "What workbenches are available in my cluster?"
Important Notes
Permissions: The MCP server uses your current oc login credentials. All operations are performed with your user's permissions.
Security: Start with read-only mode to explore safely. Only enable write mode when you need to create or modify resources.
Tool Modes
The server supports two operation modes for safety and flexibility:
Read-Only Mode (Default)
Only listing and querying tools are available. This mode is safer and prevents accidental modifications to your cluster.
Available operations:
- List workbenches, images, hardware profiles, PVCs
- Query resource consumption
- View namespace and pod information
Read-Write Mode
All tools are available, including create, delete, and modify operations.
Additional operations:
- Create/Delete workbenches
- Create/Delete custom images
- Create/Delete hardware profiles
- Create PVCs
- Start/Stop workbenches
Configuring the Mode
The mode is controlled by the MCP_RHOAI_MODE environment variable:
| Variable Value | Mode | Description |
|---|---|---|
Not set or any value except write | Read-only | Default, safe mode |
write | Read-write | Full access to all tools |
Set the environment variable in your MCP server configuration (see Configuration section above).
Available Tools
Read-Only Tools (Always Available)
- List Pods - List pods in a namespace
- List Namespaces - List all namespaces in the cluster
- List Workbenches - List workbenches in a specific namespace
- List All Workbenches - List workbenches across all namespaces
- List Images - List all available notebook images
- List Hardware Profiles - List available hardware profiles
- List PVCs - List persistent volume claims in a namespace
- List Resource Consumption Per Workbench - Get resource usage for a specific workbench
- List Resource Consumption Per Namespace - Get resource usage for all workbenches in a namespace
- List Resource Consumption Per User - Get resource usage for a specific user
- List Resource Consumption Per Cluster - Get cluster-wide resource usage
Write Tools (Only in Write Mode)
- Create Workbench - Create a new workbench with specified configuration
- Delete Workbench - Delete an existing workbench
- Change Workbench Status - Start or stop a workbench
- Create Custom Image - Create a custom notebook image
- Delete Image - Delete a custom image
- Create Hardware Profile - Create a new hardware profile
- Delete Hardware Profile - Delete a hardware profile
- Create PVC - Create a persistent volume claim
Development
Project Structure
mcp-server-rhoai/
βββ main.go # Entry point
βββ core/ # Shared types and utilities
β βββ gvr.go # Kubernetes GroupVersionResource definitions
β βββ common_types.go # Shared output/namespace types
β βββ workbench_types.go # Workbench-related types
β βββ image_types.go # Image-related types
β βββ hardware_profile_types.go # Hardware profile types
β βββ pvc_types.go # PVC-related types
β βββ resource_consumption_types.go # Resource consumption types
β βββ logging.go # Logging utilities
βββ tools/ # MCP tool implementations
β βββ registry.go # Tool registration (read-only & write modes)
β βββ workbench_tools.go # Workbench CRUD & status tools
β βββ image_tools.go # Image management tools
β βββ hardware_profile_tools.go # Hardware profile tools
β βββ storage_tools.go # PVC management tools
β βββ namespace_tools.go # Namespace listing tools
β βββ pod_tools.go # Pod listing tools
β βββ resource_consumption_tools.go # Resource monitoring tools
β βββ common.go # Shared tool helpers
βββ resources/ # MCP resource definitions
βββ prompts/ # MCP prompts
βββ Makefile # Build automation
Adding New Tools
- Implement the tool function in the appropriate file in
tools/ - Register it in
tools/registry.go:- For read-only: Add to
RegisterReadOnlyTools() - For write: Add to
RegisterWriteTools()
- For read-only: Add to
- Add tests in
tools/*_test.go - Update this README
Linting
This repository uses golangci-lint for code quality checks.
Install golangci-lint
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
Run Linting
make lint
