Azure MCP Server: Multi-Tool Integration with Cosmos DB and AI Search

A comprehensive Azure AI Agent system that integrates multiple Model Context Protocol (MCP) servers with intelligent search capabilities, featuring automatic fallback between Azure Cosmos DB and Azure AI Search.

🌟 Features

• 🧠 Intelligent Search with Fallback: Automatically switches from Azure Cosmos DB to Azure AI Search when needed
• 🔧 Multiple MCP Servers: Weather, time, system logging, and Azure data services
• 📊 Source Transparency: Clear indication of data source (Cosmos DB vs AI Search) in responses
• 🔐 Secure Configuration: Environment-based configuration with Azure managed identity support
• 📈 Quality Evaluation: Comprehensive agent evaluation framework using Azure AI Evaluation SDK
• 🏗️ Production Ready: Scalable architecture with proper error handling and monitoring

🏛️ Architecture

graph TB
    subgraph "User Interface"
        CF["Chainlit Frontend<br/>Port 8081"]
    end
    
    subgraph "Backend"
        BK["FastAPI + Semantic Kernel<br/>Port 8091"]
    end
    
    subgraph "MCP Servers"
        WM["Weather MCP<br/>Port 8086"]
        TM["Time MCP<br/>Port 8087"] 
        SM["System Log MCP<br/>Port 8089"]
        AM["Azure Integrated MCP<br/>Port 8090"]
    end
    
    subgraph "Azure Services"
        CDB["Azure Cosmos DB"]
        AIS["Azure AI Search"]
        AOI["Azure OpenAI"]
        AAI["Azure AI Foundry"]
    end
    
    CF --> BK
    BK --> WM & TM & SM & AM
    AM --> CDB & AIS & AOI
    BK --> AAI

🚀 Quick Start

Prerequisites

• Python 3.10+
• Azure subscription with:
  • Azure AI Foundry project
  • Azure Cosmos DB account
  • Azure AI Search service
  • Azure OpenAI resource

1. Clone the Repository

git clone https://github.com/sureshpaulraj/azure-mcp-server-multiple-tools.git
cd azure-mcp-server-multiple-tools

2. Environment Setup

# Create conda environment
conda create -n azure-mcp python=3.10 -y
conda activate azure-mcp

# Install dependencies
pip install -r requirements.txt

3. Configuration

Copy the example environment file and configure your Azure resources:

cp .env.example .env
# Edit .env with your Azure resource details

Required environment variables:

# Azure AI Foundry
AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME=gpt-4o
AZURE_AI_AGENT_ENDPOINT=https://your-project.services.ai.azure.com/api/projects/your-project-id

# Azure Cosmos DB
COSMOSDB_ENDPOINT=https://your-cosmos-account.documents.azure.com:443/
COSMOSDB_KEY=your-cosmos-key
COSMOSDB_DATABASE=your-database-name
COSMOSDB_CONTAINER=your-container-name

# Azure AI Search
AZURE_SEARCH_ENDPOINT=https://your-search-service.search.windows.net
AZURE_SEARCH_API_KEY=your-search-api-key
AZURE_SEARCH_INDEX=your-search-index

# Azure OpenAI (for embeddings)
AZURE_OPENAI_EMBEDDING_ENDPOINT=https://your-openai.openai.azure.com/openai/deployments/text-embedding-ada-002/embeddings
AZURE_OPENAI_EMBEDDING_API_KEY=your-embedding-api-key

4. Start the Services

Run all MCP servers and the backend:

# Terminal 1: Weather MCP Server
python src/mcpservers/weather.py

# Terminal 2: Time MCP Server  
python src/mcpservers/localtime.py

# Terminal 3: System Log MCP Server
python src/mcpservers/azuredataexproler.py

# Terminal 4: Azure Integrated MCP Server
python src/mcpservers/azuremcpserver.py

# Terminal 5: FastAPI Backend
python src/backend/server.py

# Terminal 6: Chainlit Frontend
cd src/frontend && chainlit run app.py

5. Access the Application

• Web Interface: http://localhost:8081
• API Documentation: http://localhost:8091/docs

🎯 Key Components

Azure Integrated MCP Server

The core component that provides intelligent search with automatic fallback:

# Intelligent search with fallback logic
async def intelligent_search_with_fallback(
    query: str,
    search_type: str = "vector",
    similarity_threshold: float = 0.7,
    fallback_threshold: int = 0
) -> Dict[str, Any]:
    """
    Smart search that tries Cosmos DB first, then falls back to AI Search.
    Returns results from only ONE source with explicit source marking.
    """

Key Features:

• 🔄 Automatic Fallback: Cosmos DB → Azure AI Search
• 📍 Source Transparency: Every document marked with source information
• 🎯 Single Source Response: No mixing of results from different sources
• ⚡ Performance Optimized: Efficient query execution with proper indexing

MCP Server Ecosystem

📊 Agent Quality Evaluation

The project includes a comprehensive evaluation framework using Azure AI Evaluation SDK:

Evaluation Dimensions

• Intent Resolution (1-5): How well the agent understands user requests
• Tool Call Accuracy (0-1): Accuracy in tool selection and parameter handling
• Task Adherence (1-5): Following instructions and completing tasks
• Response Completeness (1-5): Quality and thoroughness of responses

Running Evaluations

# Run the evaluation notebook
jupyter notebook Azure_AI_Agent_Quality_Evaluation_MCP.ipynb

🛠️ Development

Project Structure

azure-mcp-server-multiple-tools/
├── src/
│   ├── backend/
│   │   └── server.py              # FastAPI backend with Semantic Kernel
│   ├── frontend/
│   │   ├── app.py                 # Chainlit web interface
│   │   └── chainlit.yaml          # Frontend configuration
│   └── mcpservers/
│       ├── azuremcpserver.py      # Azure Cosmos DB + AI Search MCP
│       ├── weather.py             # Weather information MCP
│       ├── localtime.py           # System time MCP
│       └── azuredataexproler.py   # System logging MCP
├── infra/
│   └── terraform/                 # Infrastructure as Code
├── docs/
│   ├── ARCHITECTURE_DIAGRAM.md    # System architecture
│   └── MERMAID_ARCHITECTURE_DIAGRAMS.md
├── Azure_AI_Agent_Quality_Evaluation_MCP.ipynb
├── requirements.txt
├── .env.example
└── README.md

Adding New MCP Servers

1. Create a new Python file in src/mcpservers/
2. Implement MCP server using FastMCP framework
3. Add server configuration to backend
4. Update environment variables if needed

Example MCP server template:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("Your-MCP-Server")

@mcp.tool(description="Your tool description")
async def your_tool_function(param: str) -> str:
    """Your tool implementation"""
    return f"Result for {param}"

if __name__ == "__main__":
    mcp.run()

🔒 Security Best Practices

• ✅ Environment Variables: No hardcoded secrets
• ✅ Azure Managed Identity: Recommended authentication method
• ✅ Least Privilege: Minimal required permissions
• ✅ Secure Communication: HTTPS for all external calls
• ✅ Input Validation: Comprehensive parameter validation

🤝 Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

Development Setup

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

Code Style

• Follow Python PEP 8 conventions
• Use type hints for all functions
• Add docstrings for public functions
• Include unit tests for new features

📚 Documentation

• Architecture Overview
• Mermaid Diagrams
• API Documentation (when server is running)
• Azure AI Evaluation Guide

🐛 Troubleshooting

Common Issues

MCP Server Connection Errors

# Check if all MCP servers are running
curl http://localhost:8086/health  # Weather MCP
curl http://localhost:8087/health  # Time MCP
curl http://localhost:8089/health  # System Log MCP
curl http://localhost:8090/health  # Azure MCP

Azure Authentication Issues

# Login to Azure CLI
az login

# Verify authentication
az account show

Missing Dependencies

# Reinstall all dependencies
pip install -r requirements.txt --force-reinstall

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Model Context Protocol (MCP) by Anthropic
• FastMCP framework
• Azure AI Platform
• Semantic Kernel

📞 Support

• 📧 Issues: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 📖 Documentation: Wiki

⭐ Star this repository if you find it helpful!