Daedra

Daedra is a self-contained web search MCP server in Rust. It gives search and page-fetch tools to AI agents. It works from any IP address: datacenter, VPS, or residential. Basic search needs no API keys.

Why Daedra

Major search engines block datacenter and VPS IP addresses with CAPTCHAs. Daedra solves this with a multi-backend fan-out. All available backends run the query at once, and the results merge by relevance:

Mwmbl → Brave → Marginalia → Bing RSS → Bing → Google News → Hacker News → Google → Wikipedia → StackOverflow → GitHub → Wiby → DDG Instant → DuckDuckGo HTML

Three backend groups exist. Scraper backends read HTML and sometimes meet a CAPTCHA. Machine-format backends read RSS and JSON feeds that engines publish for integrations. Knowledge backends query Wikipedia, StackOverflow, GitHub, Wiby, and DDG Instant. All fourteen backends need no paid search key.

Unkeyed general coverage depends on where daedra runs. Mwmbl and Marginalia answer from any IP. Brave HTML may rate-limit. The knowledge and machine-format backends (wiki, HN, StackOverflow, GitHub, Bing RSS, Google News) work from any IP.

Per-backend circuit breakers and per-backend rate limits keep the chain stable under load. The chain retries only transient errors. Bot protection and rate-limit errors fail fast, so the next backend starts at once.

Features

• 14 unkeyed search backends with automatic fallback (see the table below)
• Circuit breaker (BackendHealth): opens after repeated failures, with a 30 second cooldown
• Per-backend rate limits via governor, with separate quotas for knowledge and scraper backends
• Classified retry: the chain retries only transient errors
• Readability extraction: dom_smoothie extracts the article body from HTML pages
• PDF support: infer detects the MIME type, pdf-inspector extracts Markdown (OCR-needing pages are reported)
• Content classification: FetchedContent (Html / Pdf / Binary) on every fetch
• URL classification: src/url_classification.rs maps search result URLs to content types
• MCP tools: web_search, visit_page, crawl_site, and the search_duckduckgo alias

Install

cargo install daedra

Search backends

The merge ranks every result by how much of the query it shares. Results that share no query token land after every matched result. When no result matches at all, the search reports this. It does not return unrelated feed noise. Backends that ignore the query cannot outvote backends that found nothing relevant.

This crate does not accept paid search keys. Mwmbl and Marginalia are the unkeyed general indexes. HTML scrapers may meet a CAPTCHA. Knowledge and feed backends stay on wiki, HN, StackOverflow, GitHub, and RSS.

--backend and --exclude select or skip backends by name. --time-range d|w|m|y filters by recency on the backends that support it (Hacker News, Google News).

Usage

MCP server (for Claude, Cursor, pawan, and similar agents)

{
  "mcpServers": {
    "daedra": {
      "command": "daedra",
      "args": ["serve", "--transport", "stdio", "--quiet"]
    }
  }
}

CLI

# Search: pick backends, filter by recency
daedra search "rust async runtime" --num-results 5
daedra search "crate release" --backend hn --time-range w
daedra search "tokio" --exclude bing-rss --exclude gnews

# Fetch a webpage as Markdown (HTML via Readability, PDF via pdf-inspector)
daedra fetch https://rust-lang.org
daedra fetch https://example.com/report.pdf --timeout 60
daedra fetch https://example.com/report.docx   # docx, odt, epub, pptx, xlsx, csv too

# Crawl with the optional crawlberg engine (rebuild with --features crawlberg)
daedra crawl https://rust-lang.org --engine crawlberg -m 10 -d 3

# Crawl: root page always included; robots.txt honored; depth follows links
daedra crawl https://rust-lang.org -m 5 --depth 2 --delay-ms 250

# Check backend health (reports missing API keys; JSON with -f json)
daedra check

# Server info (four MCP tools; JSON with -f json)
daedra info

As a Rust library

use daedra::tools::SearchProvider;
use daedra::types::SearchArgs;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let provider = SearchProvider::auto();
    let args = SearchArgs {
        query: "rust programming".to_string(),
        options: None,
    };
    let results = provider.search(&args).await?;
    for r in &results.data {
        println!("{} — {}", r.title, r.url);
    }
    Ok(())
}

MCP tools

web_search

Search the web with automatic backend fallback.

{
  "query": "search terms",
  "options": {
    "region": "wt-wt",
    "safe_search": "MODERATE",
    "num_results": 10,
    "time_range": "w"
  }
}

search_duckduckgo is an alias for web_search. It exists for backward compatibility.

visit_page

Fetch a page and extract the content as Markdown. HTML pages use dom_smoothie Readability extraction. The infer crate detects PDFs, and pdf-inspector extracts Markdown. Office documents (docx, doc, odt, rtf, epub, pptx, xlsx, csv) convert via anydoc.

{
  "url": "https://example.com",
  "selector": "article.main",
  "include_images": false
}

crawl_site

Crawl a site from a root URL and return Markdown for each page. The crawler always fetches the root page, honors robots.txt (longest-match rule; ignore_robots opts out), expands one level of sitemap indexes, and can follow same-origin link layers with depth (max 5). delay_ms staggers fetches. The command exits non-zero when it fetched zero pages.

Architecture

Daedra
├── SearchProvider (fallback chain, circuit breakers, keyed rate limits)
│   ├── MwmblBackend (unkeyed general-web JSON)
│   ├── BraveBackend / BingBackend / GoogleBackend (HTML scraping)
│   ├── BingRssBackend / GoogleNewsBackend / HnAlgoliaBackend (machine formats)
│   ├── WikipediaBackend / StackExchangeBackend / GitHubBackend
│   ├── WibyBackend / DdgInstantBackend
│   └── SearchClient (DuckDuckGo HTML, last resort)
├── FetchClient (FetchedContent: Html / Pdf / Binary → Markdown)
│   ├── dom_smoothie (Readability), infer (MIME), pdf-inspector (PDF), anydoc (Office)
├── soft_block (classifies zero-result scraper pages: genuine empty or bot block)
├── url_classification (search result URL → ContentType)
├── SearchCache (moka async cache)
├── MCP Server (DaedraHandler: handle_web_search, handle_visit_page, handle_crawl_site)
│   ├── STDIO transport (JSON-RPC)
│   └── SSE transport (Axum HTTP)
└── CLI (Commands::run, CheckReporter)

Key dependencies

Configuration

# Optional quota tokens (raise rate limits; backends work without them)
export GITHUB_TOKEN=...          # Higher GitHub API rate limit
export MARGINALIA_API_KEY=...    # Higher Marginalia rate limit

# Logging
export RUST_LOG=daedra=info

Ecosystem

Built by DIRMACS.

License

MIT