C
Crawl4AI
Open-source web crawler optimized for LLMs and AI agents — 62K+ stars
Open SourceFree
About
Crawl4AI is a free, open-source web crawling and scraping tool optimized for LLMs. Handles JavaScript rendering, generates clean markdown, supports structured data extraction with LLMs, and runs 6x faster than alternatives. Features multi-URL crawling, session management, proxy support, and custom hooks. Fully local with no API keys required.
Details
| Type | web-scraping, crawler, data-extraction |
| Integrations | any-llm, langchain, llamaindex |
| Language | python |
Tags
web-scrapingcrawlerragmarkdownlocalpythonopen-source
Quick Info
- Organization
- unclecode
- Pricing
- open-source
- Free Tier
- Yes
- Updated
- Aug 13, 2026
78.0Ktoday82
Also in Dev Tools
F
Firecrawl
Web scraping API built for LLMs — turn any website into LLM-ready data — 89K+ stars
OSSfreemium
Mendable
166.6Ktoday160
H
Headroom Context Optimization
Reduce LLM API costs by 50-90% through advanced context compression
OSSFree
Shubham Saboo
132.4K3d ago93
L
LlamaIndex
Data framework for LLM applications — RAG, agents, and workflows — 47K+ stars
OSSFree
LlamaIndex
51.6Ktoday475