Robotstxt AI MCP Server
by sharozdawaio.github.sharozdawa/robotstxt-aiv1.0.1
Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
context tax
queued
security
queued
cold start
queued
freshness
Stale7mo ago
Install Robotstxt AI MCP server
Install in Claude Code
claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcpInstall in Cursor
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Add to ~/.cursor/mcp.json (global) or .cursor/mcp.json (project).
Install in Claude Desktop
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Settings → Developer → Edit Config (claude_desktop_config.json), then restart.
Install in VS Code
{
"servers": {
"robotstxt-ai": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Add to .vscode/mcp.json in your workspace.
Install in Windsurf
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json.
Freshness
Stale — last maintenance signal 7mo ago. The newest of the signals below sets the band.
Last commit (default branch)
2026-03-22 · 7mo ago · GitHub
Latest release
2026-03-21 · 7mo ago · GitHub · v1.0.0
Package published
no data · npm/PyPI
Registry entry updated
2026-03-21 · 7mo ago · official registry · v1.0.1
FAQ
›How do I install the Robotstxt AI MCP server in Claude Code?
Run: claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp. For Cursor, VS Code, Claude Desktop and Windsurf, use the install tabs above.
›Does Robotstxt AI require an API key?
No required environment variables are declared in its published metadata.
›Can I use Robotstxt AI as a remote (hosted) MCP server?
No hosted endpoint is published; it runs locally over stdio.
›Is Robotstxt AI in the official MCP registry?
Yes, as io.github.sharozdawa/robotstxt-ai.
Alternatives to Robotstxt AI
Other browser automation MCP servers.
Browser UseActiveControl a real Chrome browser to complete any task: fill forms, extract data, book flights.
ScraplingActiveWeb scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.
BrowserActiveA fast, lightweight MCP server that empowers LLMs with browser automation via Puppeteer’s structured accessibility data, featuring optional vision mode for complex visual understanding and flexible, cross-platform configuration.
PlaywrightActivePlaywright browser automation: interact with web pages through structured accessibility snapshots.
BrowserActiveHeadless browser for AI and web automation with a built-in MCP server.
AIHawkActiveAI browser agent: browses, clicks, types, and reads real web pages from plain-English instructions.