About ByteCrawl
ByteCrawl is an MCP server published by abrahamperz in the Scraping category: the focused-crawling MCP server for AI agents. Rank a whole site by your topic, read any page as clean Markdown, extract anything.
The repository has 12 stars and 0 forks, with the last commit a month ago. Recent commit activity is the strongest signal that reported issues will get attention.
Install
From source
git clone https://github.com/abrahamperz/ByteCrawlThis server has no ConduID identity, so agent calls to it are not receipted. Pin the version you install and review the source before granting it credentials.
Ask AI
✦
Ask AI about ByteCrawl
Powered by Claude · Grounded in docs
I know everything about ByteCrawl. Ask me about installation, configuration, usage, or troubleshooting.
Security checks
6 checks have not been run yet.
Releases
v1.4.0v1.4.0 · 11 Sep 2026Added A bot wall now says so.** When a site's protection refuses the fetch, the error names the wall instead of a bare `403 Client Error: Forbidden` — e.g. "Blocked by Cloudflare — tapology.com uses bot protection and refused this request…
v1.3.0v1.3.0 · 11 Sep 2026Changed CI is split into separate checks instead of one "test" job.** Lint (`ruff check`), format (`ruff format --check`), type check (`mypy`) and security (`bandit`) each run as their own top-level check alongside the unit-test matrix…
v1.2.2v1.2.2 · 10 Sep 2026Fixed `fetch_markdown` could leak raw JavaScript into its output.** On pages where the main-content extractor bails (ad-heavy listings, like tapology.com's rankings), the fallback converter kept `<script>` contents as loose text instead of…
v1.2.1v1.2.1 · 9 Sep 2026Wording, and one change to how the agent skill behaves. Nothing in the library changed — `pip install bytecrawl` gets the same code as 1.2.0. Changed The agent skill no longer instructs an agent to install it.** It opened by telling…
v1.2.0v1.2.0 · 9 Sep 2026Added `Page.selectors()` — extraction without having read the markup.** Writing an `extract()` call means knowing the classes on the page, and an agent connected over MCP could not: `fetch_markdown` strips exactly the classes and…