ICAE-Bench
Benchmark for evaluating coding agents on end-to-end project building from vague intent
About
ICAE-Bench is a research benchmark designed to evaluate coding agents as interactive project builders rather than single-task completers. It tests agents on transforming incomplete product specifications into working software through multi-step workflows including requirement clarification, planning, tool use, debugging, and repository-level code construction. Addresses the emerging paradigm of 'vibe-coding' where users provide high-level intent instead of precise specifications, making it essential for evaluating next-generation AI coding assistants.
Details
| Type | |
| Integrations | |
| Language |
Tags
Quick Info
- Organization
- Research Community
- Pricing
- open-source
- Free Tier
- Yes
- Updated
- Jul 24, 2026
Also in Dev Tools
Crawl4AI
Open-source web crawler optimized for LLMs and AI agents — 62K+ stars
Firecrawl
Web scraping API built for LLMs — turn any website into LLM-ready data — 89K+ stars
Headroom Context Optimization
Reduce LLM API costs by 50-90% through advanced context compression