1. Conduid
  2. Scraping
  3. Researchpowerpack Http
MCP server · Scraping

Researchpowerpack Http

The ultimate research MCP toolkit: Reddit mining, web search with CTR aggregation, and intelligent web scraping - all in one modular package

37Low

Scored 6 months ago · breakdown

About Researchpowerpack Http

Researchpowerpack Http is an MCP server in the Scraping category: the ultimate research MCP toolkit: Reddit mining, web search with CTR aggregation, and intelligent web scraping - all in one modular package.

Install

Claude Code
claude mcp add researchpowerpack-http -- npx -y mcp-researchpowerpack-http
npx
npx -y mcp-researchpowerpack-http

This server has no ConduID identity, so agent calls to it are not receipted. Pin the version you install and review the source before granting it credentials.

Ask AI

✦

Ask AI about Researchpowerpack Http

Powered by Claude · Grounded in docs

I know everything about Researchpowerpack Http. Ask me about installation, configuration, usage, or troubleshooting.

Security checks

  • ✓README presentThe repository ships documentation.

5 checks have not been run yet.

README

MseeP.ai Security Assessment Badge

mcp-researchpowerpack

http mcp server for research. five tools, orientation-first, built for agents that run multi-pass research loops.

ships on mcp-use. tool-level provider and llm orchestration flows through effect ts service layers for typed concurrency, typed errors, scoped kernel cleanup, and step timeouts. provider clients remain small promise/sdk adapters behind those layers. no stdio — http only.

tools

tool what it does needs
start-research returns a goal-tailored brief: primary_branch (reddit / web / both), exact first_call_sequence, 25–50 keyword seeds, iteration hints, gaps to watch, stop criteria. call first every session. LLM_API_KEY + LLM_BASE_URL + LLM_MODEL for the non-degraded brief (optional — server falls back to a static playbook)
raw-web-search parallel search, up to 50 keywords per call. serper is primary; jina search is the fallback when serper is missing, fails, or returns empty. returns the raw ranked markdown list directly — no llm pass. use for broad discovery, audit trails, and reddit permalink probes via explicit site:reddit.com/r/.../comments. SERPER_API_KEY or JINA_API_KEY
smart-web-search parallel search, up to 50 keywords per call, plus required extract. same provider order as raw. always runs llm classification and returns tiered markdown (HIGHLY_RELEVANT / MAYBE_RELEVANT / OTHER) + grounded synthesis + gaps + refine suggestions. supports scope: "web" | "reddit" | "both". SERPER_API_KEY or JINA_API_KEY + llm env triple
raw-scrape-links fetch urls in parallel, return full markdown directly. reddit post permalinks route through the reddit api with threaded comments. non-reddit urls hit jina reader first, then jina reader through scrape.do proxy mode, then optional kernel browser rendering for web pages. pdf / docx / pptx / xlsx urls go straight through jina reader. optional REDDIT_CLIENT_ID / REDDIT_CLIENT_SECRET, SCRAPEDO_API_KEY, JINA_API_KEY, KERNEL_API_KEY
smart-scrape-links same fetch stack as raw scrape, then per-url llm extraction with required extract. returns focused evidence packs with ## Source, ## Matches, ## Not found, and ## Follow-up signals. raw scrape providers + llm env triple

also exposes /health (simplified for proxies) and health://status (full json: planner/extractor reachability, consecutive failure counters, uptime, active sessions).

workflow

call start-research once per session with your goal. the server returns a brief that names the first tool to fire (reddit-first for sentiment/migration, web-first for spec/bug/pricing, both when opinion-heavy and you also need official sources), the keyword seeds to fan out, and stop criteria.

for search fan-out, think bad → better before calling raw-web-search or smart-web-search. turn broad phrases like <feature> support, <product> pricing, <library> bug fix, or <tool> reviews into source-aware probes like:

  • site:<official-docs-domain> "<feature>" "<platform-or-version>"
  • site:<vendor-domain> "<product>" pricing "enterprise" OR "free tier"
  • "<exact error text>" "<library-or-package>" "<version>" site:github.com
  • site:reddit.com/r/<community>/comments "<tool>" "migration" OR "regression"

pair the server with the run-research skill for the full agentic playbook:

npx -y skills add -y -g https://github.com/yigitkonur/skills-by-yigitkonur --skill /run-research

quickstart

# from npm
HOST=127.0.0.1 PORT=3000 npx -y mcp-researchpowerpack

# from source
git clone https://github.com/yigitkonur/mcp-researchpowerpack.git
cd mcp-researchpowerpack
pnpm install && pnpm dev

point your client at http://localhost:3000/mcp:

{
  "mcpServers": {
    "research-powerpack": {
      "url": "http://localhost:3000/mcp"
    }
  }
}

or skip the install entirely and hit the hosted deployment at https://research-mcp.yigitkonur.com/mcp.

config

copy .env.example, set only what you need. blank/whitespace keys are treated as absent. missing keys don't crash the server — they disable the affected capability with a clear error at call time.

server

var default
PORT 3000 http port
HOST 127.0.0.1 bind address; cloud runtimes that set PORT auto-switch to 0.0.0.0. public binds require ALLOWED_ORIGINS, MCP_URL, CSP_URLS, or FLY_APP_NAME
ALLOWED_ORIGINS unset comma-separated origins for host validation / cors; merged with MCP_URL and platform CSP_URLS when present
MCP_URL unset public mcp url; contributes its origin to host validation and well-known resource urls
CSP_URLS unset platform-provided comma-separated public origins; also contributes to host validation, including the derived mcp-use --br-main host
FLY_APP_NAME unset Fly runtime app name; when present, https://<app>.fly.dev is added to host validation for Manufact's deploy verifier
MCP_DISABLE_OAUTH_METADATA unset local loopback only; set to 1, true, yes, or on to skip .well-known/oauth-protected-resource routes for anonymous local mcpc --no-profile tests
NODE_ENV unset production also requires ALLOWED_ORIGINS, MCP_URL, CSP_URLS, or FLY_APP_NAME, even on a local bind
DEBUG unset 1 or 2 to bump mcp-use debug verbosity

providers

var enables
SERPER_API_KEY primary raw/smart web search provider
SCRAPEDO_API_KEY scrape.do proxy-mode retry for jina reader (X-Proxy-Url)
REDDIT_CLIENT_ID + REDDIT_CLIENT_SECRET raw/smart scrape for reddit.com permalinks (threaded post + comments)
JINA_API_KEY jina search fallback and authenticated jina reader requests
KERNEL_API_KEY optional kernel browser-render fallback after jina direct + proxy fail
KERNEL_PROJECT optional kernel project scoping header for org-wide api keys
LLM_API_KEY + LLM_BASE_URL + LLM_MODEL goal-tailored brief, smart-web-search, smart-scrape-links

llm

any openai-compatible endpoint. LLM_API_KEY, LLM_BASE_URL, and LLM_MODEL are required together. reasoning effort is hardcoded to low.

var required?
LLM_API_KEY yes api key for the endpoint
LLM_BASE_URL yes base url for the openai-compatible endpoint (e.g. https://server.up.railway.app/v1)
LLM_MODEL yes primary model (e.g. gpt-5.4-mini)
LLM_FALLBACK_MODEL no model to use after primary exhausts retries — gets 3 more attempts (e.g. gpt-5.4). also receives oversized inputs that exceed the primary's context window

concurrency

all optional. provider limits are clamped 1–200; kernel is clamped 1–20. malformed numeric values are rejected instead of partially parsed.

var default controls
CONCURRENCY_SEARCH 50 parallel serper / jina search queries
CONCURRENCY_SCRAPER 50 parallel scrape.do (proxy mode) requests
CONCURRENCY_JINA_READER 50 parallel jina reader fetches
CONCURRENCY_REDDIT 50 parallel reddit api fetches
CONCURRENCY_KERNEL 3 parallel kernel browser-render fallbacks
LLM_CONCURRENCY 50 parallel llm extraction / classification calls

evals

pnpm test:evals writes a json artifact to test-results/eval-runs/<timestamp>.json. when an openai api key is present, it runs a live responses-api + remote-mcp eval. without one, it exits successfully in explicit skip mode and records the skip in the artifact.

useful env vars:

  • EVAL_MCP_URL
  • EVAL_MODEL
  • EVAL_API_KEY or OPENAI_API_KEY

mcpc smoke tests

pnpm test:mcpc:contract builds the package, starts the compiled server on a free loopback port, connects with the pinned @apify/mcpc CLI, and verifies the deterministic client-visible MCP contract: tools, prompts, resources, degraded start-research, and validation failures. It does not need provider credentials.

pnpm test:mcpc runs the deterministic mcpc contract smoke.

See docs/testing/mcpc-e2e.md for the harness lifecycle, environment isolation contract, and the deterministic MCP contract coverage model.

dev

pnpm install
pnpm dev          # watch mode, serves :3000/mcp
pnpm typecheck    # tsc --noEmit
pnpm test:unit    # deterministic unit tests
pnpm test:http    # below-mcp server startup / http checks
pnpm test:mcpc:contract  # deterministic built-server mcp contract smoke
pnpm test:mcpc    # deterministic built-server mcp contract smoke
pnpm test         # unit + http + deterministic mcpc contract tests
pnpm build        # compile to dist/
pnpm inspect      # mcp-use inspector

prepublishOnly runs pnpm typecheck, pnpm test:unit, pnpm test:http, pnpm test:mcpc:contract, and pnpm build, matching the required publish gate without requiring provider credentials.

deploy

deploy to manufact cloud via the mcp-use cli (github-backed):

pnpm deploy       # runs the package script: mcp-use deploy

the canonical hosted endpoint is https://research-mcp.yigitkonur.com/mcp. the raw manufact server slug is calm-wave-3gtvb.

or self-host anywhere with node 20.19+ / 22.12+:

HOST=0.0.0.0 ALLOWED_ORIGINS=https://app.example.com pnpm start
# or derive production origin protection from the public mcp url:
NODE_ENV=production MCP_URL=https://research.example.com/mcp pnpm start

architecture

index.ts                 server startup, cors, health, shutdown
src/
  config/                env parsing, capability detection, lazy proxy config
  effect/                typed service tags + Live layers; runExternalEffect()
                         only accepts fully-provided programs at the async
                         tool boundary
  clients/               provider api clients (serper, jina, kernel, reddit,
                         scrapedo) — wrapped by Live layers in src/effect/
  tools/
    registry.ts          registerAllTools() — wires the five tools
    start-research.ts    goal-tailored brief + static playbook + planner
                         circuit-breaker
    search.ts            raw/smart search handlers (ctr ranking + optional
                         llm classification)
    scrape.ts            raw/smart scrape handlers (reddit api, jina reader,
                         scrape.do proxy retry, optional kernel, optional
                         llm extraction)
    mcp-helpers.ts       markdown response builders
  services/
    llm-processor.ts     llm extraction, classification, brief generation —
                         primary + fallback model, always low reasoning,
                         oversized inputs route straight to fallback
    markdown-cleaner.ts  html/markdown cleanup (readability + turndown)
  schemas/               zod v4 input validation per tool
  utils/                 errors, retry, ctr aggregator, response builders,
                         logger (stderr-only, mcp-safe)

key patterns: capability detection at startup, description-led tool routing (no bootstrap gate), markdown-only mcp tool output for search/scrape, raw/smart tool split, tiered classified output in smart-web-search, reddit api routing in scrape tools, jina reader first for non-reddit urls, scrape.do proxy-mode retry through X-Proxy-Url, optional kernel browser-render fallback with scoped session cleanup, bounded concurrency via Effect.forEach, ctr-based url ranking, tools never throw (always return toolFailure), and structured errors with retry classification.

license

MIT

README copy captured when this server was indexed; it may lag the repository. The original is authoritative.

Questions

About Researchpowerpack Http

How do I install Researchpowerpack Http?

Run claude mcp add researchpowerpack-http -- npx -y mcp-researchpowerpack-http, then add the server to your MCP client's configuration.

Is Researchpowerpack Http safe to use with an AI agent?

Its trust score is 37 out of 100 (low). It passes 1 of the 1 static security checks run so far. It has no ConduID identity yet, so agent calls to it are not receipted.

Is Researchpowerpack Http still maintained?

Conduid hasn't recorded a commit date for this repository yet. Check the repository directly for recent activity.