The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
Crawl Readiness
Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.
Install
$
npx -y crawl-readiness-mcpClaude Code
claude mcp add crawl-readiness -- npx -y crawl-readiness-mcpCursor / Claude Desktop (mcp.json)
{
"mcpServers": {
"crawl-readiness": {
"command": "npx",
"args": [
"-y",
"crawl-readiness-mcp"
]
}
}
}Install command taken from the project's README or registry entry. Always review code before running it.
Similar plugs
Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.
Real-time global intelligence dashboard. AI-powered news aggregation, geopolitical monitoring, and infrastructure tracking in a unified situational awareness interface
An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here…
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.
