Crawl4AI
LibraryOpen-source crawler for LLMs
Use it when you need clean Markdown or JSON from web pages to feed an LLM or RAG pipeline
An open-source web crawler and scraper that turns websites into LLM-ready Markdown, runnable yourself or hosted.
Added to the shelf
From the makersREADME, read 15 Oct
Crawl4AI is an open-source web crawler and scraper that turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines. It runs as a Python library, a CLI (crwl), or a self-hosted Docker server. Crawl4AI Cloud is a hosted version of the same idea, where the browsers, proxies and bot-wall handling run for you, with search, SERP and answer endpoints added.
Install
pip install -U crawl4aiCommands
pip install -U crawl4aiInstall the Python library
crawl4ai-setupInstall the browser, once
crwl https://news.ycombinator.com -o markdownPrint a page as Markdown from the CLI