Fetch MCP Server

A Model Context Protocol server that provides web content fetching capabilities using browser automation, OCR, and multiple extraction methods. This server enables LLMs to retrieve and process content from web pages, even those that require JavaScript rendering or use techniques that prevent simple scraping.

Available Tools

• fetch - Fetches a URL from the internet using browser automation and multi-method extraction (including OCR).
  • url (string, required): URL to fetch
  • raw (boolean, optional): Get the actual HTML content if the requested page, without simplification (default: false)

The server uses multiple methods to extract content:

1. Browser automation with undetected-chromedriver
2. OCR using pytesseract with layout detection
3. HTML extraction using requests/BeautifulSoup
4. Document parsing (PDF, DOCX, PPTX)
5. Original markdown conversion method

The server uses a sophisticated scoring system to select the best result, considering:

1. Base content score (up to 50 points)

   • Points awarded based on content length (1 point per 100 characters, max 50)
   • Penalizes extremely short content (<100 characters)

2. Structure bonus (up to 20 points)

   • Awards points for well-structured content with paragraphs
   • More paragraphs indicate better content organization

3. Quality penalties

   • Detects and penalizes error messages
   • Reduces score for content containing error indicators
   • Validates content structure and readability

The scoring system ensures the most reliable and high-quality content is selected, regardless of the extraction method used. Debug logging is available to track scoring decisions.

Prompts

• fetch
  • Fetch a URL and extract its contents as markdown using browser automation
  • Arguments:
    • url (string, required): URL to fetch

Installation

Using Docker

To install and run mcp-server-fetch using Docker, follow these steps:

1. Build the Docker image:

   docker build -t mcp-server-fetch .
   

2. Run the Docker container:

   docker run --rm -i mcp-server-fetch
   

Configuration

Configure Roo Code or Claude App

Add to your Claude settings:

{
  "mcpServers": {
    "fetch": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "mcp-server-fetch"
      ],
      "disabled": false,
      "alwaysAllow": []
    }
  }
}

Customization - User-agent

By default, depending on if the request came from the model (via a tool), or was user initiated (via a prompt), the server will use either the user-agent

ModelContextProtocol/1.0 (Autonomous; +https://github.com/modelcontextprotocol/servers)

or

ModelContextProtocol/1.0 (User-Specified; +https://github.com/modelcontextprotocol/servers)

This can be customized by adding the argument --user-agent=YourUserAgent to the args list in the configuration.

Browser Automation and OCR

The server now includes advanced content extraction capabilities:

• Automated handling of cookie consent banners
• Full-page screenshot capture
• OCR with layout detection using pytesseract
• Multiple extraction methods with automatic selection of best results

Contributing

We encourage contributions to help expand and improve mcp-server-fetch. Whether you want to add new tools, enhance existing functionality, or improve documentation, your input is valuable.

For examples of other MCP servers and implementation patterns, see: https://github.com/modelcontextprotocol/servers

Pull requests are welcome! Feel free to contribute new ideas, bug fixes, or enhancements to make mcp-server-fetch even more powerful and useful.

License

mcp-server-fetch is licensed under the MIT License. This means you are free to use, modify, and distribute the software, subject to the terms and conditions of the MIT License. For more details, please see the LICENSE file in the project repository.