C
Crawl4AI
Open-source web crawler optimized for LLMs and AI agents — 62K+ stars
Open SourceFree
About
Crawl4AI is a free, open-source web crawling and scraping tool optimized for LLMs. Handles JavaScript rendering, generates clean markdown, supports structured data extraction with LLMs, and runs 6x faster than alternatives. Features multi-URL crawling, session management, proxy support, and custom hooks. Fully local with no API keys required.
Details
| Type | web-scraping, crawler, data-extraction |
| Integrations | any-llm, langchain, llamaindex |
| Language | python |
Tags
web-scrapingcrawlerragmarkdownlocalpythonopen-source
Quick Info
- Organization
- unclecode
- Pricing
- open-source
- Free Tier
- Yes
- Updated
- Apr 1, 2026
63.1Ktoday72
Also in Dev Tools
F
Firecrawl
Web scraping API built for LLMs — turn any website into LLM-ready data — 89K+ stars
OSSfreemium
Mendable
102.1Ktoday138
H
Headroom Context Optimization
Reduce LLM API costs by 50-90% through advanced context compression
OSSFree
Shubham Saboo
104.2Ktoday74
M
Mem0
Persistent, adaptive memory layer for AI agents and assistants
OSSfreemium
Mem0
51.6Ktoday284