polite-crawler MCP Server

by mark1russell7io.github.mark1russell7/polite-crawlerv0.1.4

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

0Node.jsstdioofficial registry

context tax

pending

security

pending

cold start

pending

freshness

pending

Our reading of polite-crawler, from the data

polite-crawler is a browser automation MCP server from mark1russell7, listed in the official MCP Registry.

We haven't measured its tool list yet, so its token cost is unknown.

With 0 GitHub stars it ranks #560 of 1,199 browser automation servers we index.

It runs locally on your machine over stdio.

Install polite-crawler MCP server

claude mcp add polite-crawler -- npx -y polite-crawler mcp

{
  "mcpServers": {
    "polite-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler",
        "mcp"
      ]
    }
  }
}

Add to ~/.cursor/mcp.json (global) or .cursor/mcp.json (project).

{
  "mcpServers": {
    "polite-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler",
        "mcp"
      ]
    }
  }
}

Settings → Developer → Edit Config (claude_desktop_config.json), then restart.

{
  "servers": {
    "polite-crawler": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler",
        "mcp"
      ]
    }
  }
}

Add to .vscode/mcp.json in your workspace.

{
  "mcpServers": {
    "polite-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler",
        "mcp"
      ]
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json.

Configuration

Variable	Required	Secret	Description
POLITE_CRAWLER_DIR	—	—	The folder of the store. The default is ~/.polite-crawler.
POLITE_CRAWLER_CONTACT	—	—	An email address or URL for the From header of each request.
POLITE_CRAWLER_ALLOW_PRIVATE	—	—	Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them.
POLITE_CRAWLER_ALLOW_USER_PROVIDED	—	—	Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm.
POLITE_CRAWLER_SIGN_KEY	—	—	The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT.
POLITE_CRAWLER_SIGNATURE_AGENT	—	—	Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set.

Freshness

Not enough public signals yet to judge maintenance.

Last commit (default branch)
2026-10-18 · today · GitHub
Latest release
2026-10-18 · today · GitHub · v0.1.4
Package published
no data · npm/PyPI
Registry entry updated
2026-10-18 · today · official registry · v0.1.4

How it compares in Browser Automation

Against the most-starred browser automation servers we index. Token cost is per model turn, measured from each server's live tool list.

Server	Stars	Tools	Tokens / turn	Freshness	Runs
polite-crawler (this)	0	—	not measured	—	local
Browser Use	118,444	—	not measured	Active	local
Scrapling	88,517	—	not measured	Active	local
Browser	39,318	—	not measured	Active	local + hosted
Playwright	38,349	—	not measured	Active	local
Browser	36,690	—	not measured	Active	—

All browser automation MCP servers

FAQ

›How do I install the polite-crawler MCP server in Claude Code?

Run: claude mcp add polite-crawler -- npx -y polite-crawler mcp. For Cursor, VS Code, Claude Desktop and Windsurf, use the install tabs above.

›Does polite-crawler require an API key?

No required environment variables are declared in its published metadata.

›Can I use polite-crawler as a remote (hosted) MCP server?

No hosted endpoint is published; it runs locally over stdio.

›Is polite-crawler in the official MCP registry?

Yes, as io.github.mark1russell7/polite-crawler.

Sources and method

Official MCP Registry entry: updated 2026-10-18
GitHub repository: fetched 2026-10-19
Package registry (npm / PyPI): listed
Last checked against all sources: 2026-10-19

The description, packages and links come from the maintainer's own published metadata. Token cost, security signals, freshness and the reading above are MCPSkillHub's own measurements; see the methodology. Spot an error? Request a correction or claim the listing.