🚀🤖 Crawl4AI: the open-source web crawler for LLMs and AI agents

Latest: v0.9.5 (14 Oct 2026) · all releases →

Crawl4AI turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines. Run the open-source web crawler and scraper yourself, free forever, or use it hosted with one key: /scrape a page to Markdown, /extract it to JSON, /search the web, /serp for Google's results page as data, /answer a question with sources; MCP for your agent. Free: 10 a minute, 2 at a time, 1,000 credits every month. Plans: 120 a minute, 10 at a time, from $10 a month.

Two ways to use Crawl4AI

🐍 Run it yourself: open source, forever

pip install -U crawl4ai
crawl4ai-setup        # installs the browser, once

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com")
        print(result.markdown)

asyncio.run(main())

Docker server, CLI and every option: Installation · docs.crawl4ai.com

☁️ Or use the cloud: no browsers, no proxies

1. 
   Verify your email and 1,000 free credits are yours every month. No card. Soft launch: prices can change, what you buy stays yours.

2. Get any page as Markdown:

   curl -s https://api.crawl4ai.com/scrape \
     -H "Authorization: Bearer $CRAWL4AI_KEY" \
     -H "Content-Type: application/json" \
     -d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown
   

   The same key works for /extract, /search, /serp, /answer and many URLs at once (/scrape/batch, /scrape/jobs). Pay as you go: live prices.

3. Give it to your AI agent. Claude Code shown; Codex, Cursor and OpenCode →

   claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \
     --header "Authorization: Bearer $CRAWL4AI_KEY"
   

Which one?

	🐍 Library	🐳 Your own server	☁️ Crawl4AI Cloud
Runs the browsers	you, in your Python process	you, in Docker on your machine	we do
JS-heavy pages and bot walls	your settings, your proxies	your settings, your proxies	handled for you, automatically
Web search	–	–	/search, /serp and /answer
Price	free, forever	free (your hosting)	1,000 free credits every month; plans from $10 a month

🤓 My Personal Story

I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.

In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.