searxNcrawl

MCP server and CLI toolkit for web search and crawling, built on Crawl4AI and SearXNG.

Published at github.com/DasDigitaleMomentum/searxNcrawl — maintained by DDM – Das Digitale Momentum GmbH & Co KG. Successor to searxng-mcp.

Quick Start

Pick your setup:

Docker Compose

MCP server with Playwright/Chromium, ready in one command. SearXNG required separately for search.

cp .env.example .env          # set SEARXNG_URL to your SearXNG instance
docker compose up --build

➜ MCP server at http://localhost:9555/mcp

uvx (no clone)

Run the MCP server from PyPI, no clone or virtualenv. Chromium is downloaded automatically on the first crawl.

SEARXNG_URL=http://your-searxng:8888 uvx searxncrawl

The latest development version runs straight from GitHub with uvx --from git+https://github.com/DasDigitaleMomentum/searxNcrawl searxncrawl.

pip (standalone)

CLI tools, Python API, and MCP server. SearXNG required for search.

python -m venv .venv && source .venv/bin/activate
pip install -e .
playwright install chromium

uv (standalone)

Same capabilities as pip.

uv sync
uv run playwright install chromium

What you get

Feature	Docker Compose	pip / uv
MCP Server (STDIO)	—	✅
MCP Server (HTTP)	✅	✅
Web Crawl	✅	✅
Web Search	✅¹	✅¹
CLI Tools	via exec²	✅
Python API	—	✅
CORS (HTTP)	✅	✅

¹ Requires a SearXNG instance. ² docker compose exec searxncrawl crawl ...

Features

Crawling

• Single page, multi-page, and site crawling (DFS with depth/page limits)
• Production-tested extraction config optimized for documentation sites
• Configurable timeouts with graceful error handling

Content Quality

• Markdown deduplication — exact (default) removes repeated blocks, off disables it
• Link removal — strip all links for cleaner LLM context (--remove-links)
• Dedup guardrails — non-destructive metadata signals when removal is unusually aggressive

Web Search

• SearXNG metasearch integration (privacy-respecting)
• Configurable language, time range, categories, engines, safe search

MCP Server

• STDIO transport — for MCP harnesses (Zed, opencode, VS Code, Claude Code, etc.)
• HTTP transport — for remote access and browser clients
• CORS support — configurable origins for browser-based MCP clients
• Noise-free startup with UTF-8 encoding (cross-platform, incl. Windows)

CLI Tools

• crawl — crawl pages from the command line
• search — search the web via SearXNG
• crawl-capture — session capture for authenticated crawling

Installation

Docker Compose

The Compose stack includes searxNcrawl + Playwright/Chromium. SearXNG must be provided separately.

cp .env.example .env
# Edit .env: set SEARXNG_URL to your SearXNG instance
docker compose up --build