PyScrappy

Adaptive Python web scraping toolkit (self-healing, stealth)+ MCP server for AI agents

PyScrappy is an AI-native web scraping toolkit that turns websites into structured, LLM-ready data. Use it as a Python library or expose it as an MCP server for AI agents.

📖 Documentation: pyscrappy.vercel.app

Key features

• Generic scraper — give it any URL, get back structured text, links, images, tables, and metadata
• LLM-ready output — .to_markdown() turns any result into clean Markdown; also .to_json() and .to_dataframe()
• MCP server — expose the scrapers as tools for AI agents (Claude, Cursor, local LLMs, …)
• JS rendering — optional Playwright backend for JavaScript-heavy sites
• Custom selectors — pass CSS selectors to extract exactly what you need
• Chainable Selector — navigate HTML directly with CSS/XPath, find_all, find_by_text, and find_similar (Scrapy/BeautifulSoup-style)
• Adaptive (self-healing) selectors — remember an element and relocate it by similarity when a site changes its markup, so scrapers don't silently break
• Concurrent scraping — scrape_many / scrape_all run scrapes in parallel
• Sitemap crawling — enumerate and scrape a whole site from its sitemap.xml (index + gzip aware)
• Proxy & scraping-API support — route through a proxy or ScraperAPI/ScrapeOps for blocked sites
• TLS-fingerprint impersonation — impersonate="chrome" gets past anti-bot filters that block plain clients (optional curl_cffi backend)
• Command-line extract — pyscrappy extract <url> out.md scrapes a URL straight to a file, no code
• Retry & rate-limiting — built-in exponential backoff and per-domain rate limiting
• Type-safe — full type hints, py.typed marker
• 20+ built-in scrapers — Wikipedia, IMDB, stocks, news, GitHub, Amazon/IKEA, YouTube, and more

Installation

pip install pyscrappy

Optional extras:

# Browser support (for JS-rendered pages)
pip install 'pyscrappy[browser]'
playwright install chromium

# DataFrame support
pip install 'pyscrappy[dataframe]'

# MCP server (use PyScrappy's scrapers as AI-agent tools)
pip install 'pyscrappy[mcp]'

# Stealth (TLS-fingerprint impersonation to bypass anti-bot filters)
pip install 'pyscrappy[stealth]'

# Parquet / Excel export (ScrapeResult.to_parquet() / .to_excel())
pip install 'pyscrappy[parquet]'
pip install 'pyscrappy[excel]'

# Everything
pip install 'pyscrappy[all]'

For AI agents

PyScrappy ships an MCP server that exposes its scrapers as tools, so an agent (Claude, Cursor, an OpenAI agent, a local LLM) can pull structured web data from any URL and hand it straight to the model:

AI agent  ──MCP tool call──▶  PyScrappy  ──fetch + extract──▶  Any website
   ▲                                                                │
   └──────────────  clean Markdown / JSON  ◀───────────────────────┘

pip install 'pyscrappy[mcp]'
claude mcp add pyscrappy pyscrappy-mcp

Then just ask: "use pyscrappy to summarize the latest headlines from bbc.com." See MCP server for the full setup and tool list.

Local models (Ollama), no MCP host needed