MCPBench - Model Context Protocol Compatibility Testing Framework

Overview

MCPBench is the open standard for compatibility testing of various Model Context Protocol (MCP https://github.com/modelcontextprotocol) client and server SDK implementations. As the MCP ecosystem grows, not every server or client will be written in the same language, creating an N×N compatibility matrix where N represents the number of language SDKs available.

Our goal is to ensure seamless interoperability between different MCP implementations, regardless of the programming language used.

Current Status

Initial SDK Support:

• Python SDK (Official)
• TypeScript SDK (Official)
• Rust SDK (Official)

Future Expansion: Additional language SDKs will be added as the MCP ecosystem grows.

Milestone M0 - Foundation

We have successfully demonstrated key foundational capabilities:

1. Git Package Information Extraction

• Automatically fetch and parse git package information from official MCP SDK repositories
• Construct language-specific compatibility matrices
• Track version information and dependencies across SDKs

2. Docker Compose Generation

• Generate docker-compose configurations to run client-server combinations across the language matrix
• Support for cross-language testing scenarios
• Automated environment setup for compatibility testing

3. End-to-End (E2E) Testing Framework

• Working Python client ↔ TypeScript server communication
• Streamable HTTP transport validation
• Tool invocation and response verification
• Session management and protocol compliance testing

Project Structure

mcpbench/
├── README.md                           # This file
├── matrix/                             # Core compatibility testing framework
│   ├── templates/                      # Test templates for different scenarios
│   │   ├── python/                     # Python client/server templates
│   │   │   ├── client/
│   │   │   │   ├── e2e/               # End-to-end client tests
│   │   │   │   ├── connection/        # Connection tests
│   │   │   │   └── ...                # Other test categories
│   │   │   └── server/
│   │   │       ├── e2e/               # End-to-end server implementations
│   │   │       └── ...                # Other server templates
│   │   ├── typescript/                 # TypeScript client/server templates
│   │   │   ├── client/
│   │   │   └── server/
│   │   │       └── e2e/               # TypeScript server implementations
│   │   └── rust/                       # Rust client/server templates
│   │       ├── client/
│   │       └── server/
│   ├── driver/                         # Test orchestration and automation
│   │   ├── update_sdk_info.py         # SDK information extraction
│   │   ├── setup.py                   # Environment setup
│   │   └── docker_generator.py        # Docker compose generation
│   └── requirements.txt               # Python dependencies
└── docs/                              # Documentation and guides

Key Components

matrix/templates/: Contains test templates organized by language and test type. Each template provides a working example of MCP client/server implementation for specific testing scenarios.

matrix/driver/: Core automation scripts that:

• Extract SDK information from git repositories
• Generate compatibility matrices
• Create docker-compose configurations
• Orchestrate cross-language testing

matrix/templates/*/e2e/: End-to-end test implementations demonstrating full client-server communication using the MCP protocol.

How to Run E2E Tests

Prerequisites

• Python 3.8+ with pip
• Node.js 18+ with npm
• Docker (for containerized testing)

Quick Start: Python Client ↔ TypeScript Server

1. Install Python Dependencies

   cd mcpbench/matrix/templates/python/server
   pip install -r requirements.txt
   

2. Start the TypeScript Server

   cd mcpbench/matrix/templates/typescript/server/e2e
   npm install
   npm run dev
   

   The server will start on http://localhost:8000

3. Run the Python Client Test

   cd mcpbench/matrix/templates/python/client/e2e
   python3 test_connection.py
   

Expected Output

You should see:

• Successful connection establishment
• Tool discovery and listing
• Tool invocation and response verification
• Clean session termination

Testing Other Language Combinations

Similar patterns can be followed for other language combinations:

• Python Client ↔ Rust Server
• TypeScript Client ↔ Python Server
• Rust Client ↔ TypeScript Server

Each combination will have its own template in the matrix/templates/ directory.

Roadmap

Milestone M1 - Matrix Expansion

• Goal: Complete N×N compatibility matrix for Python, TypeScript, and Rust
• Deliverables:
  • All client-server combinations working
  • Automated matrix testing
  • Performance benchmarking
  • Protocol compliance validation

Milestone M2 - Advanced Testing

• Goal: Comprehensive test coverage and edge case handling
• Deliverables:
  • Authentication testing
  • Error handling scenarios
  • Resource management testing
  • Notification and streaming tests
  • Load testing and stress testing

Milestone M3 - Ecosystem Integration

• Goal: Integration with CI/CD and developer workflows
• Deliverables:
  • GitHub Actions integration
  • Automated compatibility reports
  • SDK version compatibility tracking
  • Performance regression detection

Milestone M4 - Extended Language Support

• Goal: Support for additional language SDKs
• Deliverables:
  • Go SDK support
  • Java SDK support
  • C# SDK support
  • Community SDK integration

Milestone M5 - Production Readiness

• Goal: Enterprise-grade compatibility testing
• Deliverables:
  • Comprehensive documentation
  • Performance SLAs
  • Security testing
  • Compliance validation

Contributing

We welcome contributions from the MCP community! Here's how you can help:

Getting Started

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature-name
3. Make your changes
4. Add tests for new functionality
5. Update documentation as needed
6. Submit a pull request

Contribution Areas

• New Language SDK Support: Add templates and tests for additional MCP SDKs
• Test Improvements: Enhance existing tests or add new test scenarios
• Documentation: Improve guides, examples, and API documentation
• Automation: Enhance the testing framework and CI/CD integration
• Performance: Optimize test execution and add performance benchmarks

Development Guidelines

• Code Style: Follow the existing code style for each language
• Testing: Ensure all new features include appropriate tests
• Documentation: Update relevant documentation for new features
• Compatibility: Verify that changes don't break existing functionality

Reporting Issues

• Use GitHub Issues to report bugs or request features
• Include detailed reproduction steps
• Provide relevant logs and error messages
• Specify the affected language combinations

Community

• Join our discussions on GitHub Discussions
• Participate in MCP community events
• Share your use cases and requirements
• Help other contributors

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Model Context Protocol team for the specification and official SDKs
• The MCP community for feedback and contributions
• Contributors who have helped build and test this framework

MCPBench - Ensuring MCP compatibility across all languages and implementations.