About io.github.sharozdawa/robotstxt-ai
io.github.sharozdawa/robotstxt-ai is an MCP server in the Scraping category: analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
Install
claude mcp add io-github-sharozdawa-robotstxt-ai -- npx -y robotstxt-ai-mcpnpx -y robotstxt-ai-mcpThis server has no ConduID identity, so agent calls to it are not receipted. Pin the version you install and review the source before granting it credentials.
Tools (5)
analyze_robotsAnalyze pasted robots.txt content for AI bot blocking status
Parameters not published.
check_bot_statusCheck if a specific bot is blocked on a given website
Parameters not published.
fetch_robotsFetch and analyze a robots.txt from any URL
Parameters not published.
generate_robotsGenerate a robots.txt with specified blocked bots and custom rules
Parameters not published.
list_ai_botsList all known AI bots with user-agents, companies, and descriptions
Parameters not published.
Ask AI
Ask AI about io.github.sharozdawa/robotstxt-ai
Powered by Claude · Grounded in docs
Security checks
- ✓README presentThe repository ships documentation.
5 checks have not been run yet.
README
Manage which AI bots can crawl your website — visually.
Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.
Features
- Visual Toggle UI — Block or allow AI bots with simple on/off switches
- 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
- Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
- Generate robots.txt — Create a complete robots.txt with your chosen rules
- MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
- Check Bot Status — Verify if a specific bot is blocked on any website
MCP Tools
| Tool | Description |
|---|---|
fetch_robots |
Fetch and analyze a robots.txt from any URL |
analyze_robots |
Analyze pasted robots.txt content for AI bot blocking status |
generate_robots |
Generate a robots.txt with specified blocked bots and custom rules |
list_ai_bots |
List all known AI bots with user-agents, companies, and descriptions |
check_bot_status |
Check if a specific bot is blocked on a given website |
Installation
Web App
git clone https://github.com/sharozdawa/robotstxt-ai.git
cd robotstxt-ai
npm install
npm run dev
Open http://localhost:3000 in your browser.
MCP Server — Claude Desktop
Add to your claude_desktop_config.json:
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": ["-y", "robotstxt-ai-mcp"]
}
}
}
MCP Server — Cursor
Add to .cursor/mcp.json:
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": ["-y", "robotstxt-ai-mcp"]
}
}
}
Tracked Bots
The server knows about 25+ bots including:
- AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
- AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
- Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot
Why robotstxt.ai vs Manual Editing
| Feature | robotstxt.ai | Manual Editing |
|---|---|---|
| Visual toggle UI | Yes | No |
| 20+ AI bots database | Yes | Research yourself |
| Analyze existing robots.txt | Yes | No |
| MCP Server | Yes | No |
| Price | Free | Free but tedious |
More Open Source SEO Tools
| Tool | Description |
|---|---|
| awesome-seo-mcp-servers | Curated list of SEO MCP servers and agent skills |
| indexnow-mcp | Instant URL indexing via IndexNow |
| schema-gen | Schema.org JSON-LD markup generator |
| ai-visibility | AI brand visibility tracker |
License
MIT
Built by Sharoz Dawa — SEO Professional & Digital Marketing Expert
README copy captured when this server was indexed; it may lag the repository. The original is authoritative.