Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.
ActableSite AI Crawler Monitor
Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
Install
One-click install
Remote serverClaude Code (no install link, copy and run)
$
claude mcp add --transport http actablesite-ai-crawler-monitor https://actablesite.com/mcp/Buttons open the app on your computer and ask you to confirm before anything is added.
$
claude mcp add --transport http actablesite-ai-crawler-monitor https://actablesite.com/mcp/mcp.json (Cursor, Claude Desktop, Windsurf)
{
"mcpServers": {
"actablesite-ai-crawler-monitor": {
"url": "https://actablesite.com/mcp/"
}
}
}Install command taken from the project's README or registry entry. Always review code before running it.
Similar plugs
An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here…
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.
The Open-Source Multimodal AI Agent Stack: Connecting Cutting-Edge AI Models and Agent Infra