Evaluation
57 tools across 4 categories tagged with “evaluation”
dev tools
(38)RAG Failure Diagnostics Clinic
Diagnose and fix common RAG pipeline failure modes
Langfuse
Open-source LLM engineering platform — traces, evals, prompt management — 23K+ stars
promptfoo
Test and evaluate LLM prompts and agents — 11K+ stars
Arize Phoenix
Open-source LLM observability with tracing, evaluation, and datasets — 8K+ stars
Braintrust
AI evaluation and experiment tracking platform for production LLM apps
LangSmith
Observability, tracing, and evaluation platform for LLM applications
AgentBeats
Agent-based evaluation framework where AI judges assess other agents through standardized protocols
AgentDS
Benchmark framework measuring AI agent performance vs human experts on data science tasks
Agora Consensus Agent
Multi-agent framework for autonomous bug detection in distributed consensus protocols
Agora Consensus
Multi-agent framework for autonomous bug detection in distributed consensus protocols
CapCode
Cheat-proof coding evaluation framework with performance-capped randomized tests
CausalFlow
Causal debugging framework that converts failed LLM agent traces into minimal counterfactual fixes
ClawTrap
MITM red-teaming framework for testing autonomous web agent security in real-world scenarios
CUA-Gym
Reinforcement learning benchmark suite for training GUI automation agents with verifiable rewards
CUA-Suite
Large-scale human-annotated video dataset for training computer-use agents
DeltaBox
Millisecond-level checkpoint/rollback system for stateful AI agent exploration
Echo
Learning system that refines AI agents through user feedback on interaction logs
EpiBench
Verifiable benchmark for testing AI agents on epigenomics analysis workflows
EVMbench
Benchmark for evaluating AI agents on smart contract security tasks
EvoRepair
Self-evolving AI agent that learns from past fixes to automatically repair code vulnerabilities
FinToolBench
Specialized benchmark for evaluating LLM agents on real-world financial tool use and compliance
Front-End Checklist
Comprehensive web development checklist for HTML, CSS, and JavaScript best practices
JetBrains AI Toolkit
In-IDE LLM tracing and evaluation for JetBrains developers
LH-Bench
Evaluation framework for measuring long-horizon agent workflows on enterprise tasks
MalSkillBench
Security benchmark for detecting malicious AI agent skills with runtime verification
MCP-in-SoS
Security risk assessment framework for evaluating open-source MCP server implementations
PhysAssistBench
Benchmark for evaluating LLM agents in doctor-patient-EHR clinical assistance workflows
PostTrainBench
Benchmark for evaluating autonomous post-training of LLMs under compute constraints
SlopCodeBench
Benchmark for measuring coding agent performance degradation across iterative tasks
SpecOps
Automated testing framework for GUI-based AI agents in real-world environments
SWE-Explore
Benchmark for evaluating coding agents' repository exploration and code understanding abilities
TDAD
Pre-change impact analysis for AI coding agents using graph-based dependency mapping
TML-Bench
Benchmark for evaluating data science agents on Kaggle-style tabular ML tasks
ToolBench-X
Benchmark framework testing agent robustness with unreliable tools and error conditions
Translate-R1
RL-powered cost-aware translation optimizer for multilingual LLM agents
Vera
Automated safety testing framework for LLM agents with risk discovery and verification
Veritas
LLM-powered binary analysis for detecting memory corruption vulnerabilities in stripped code
ZEBRAARENA
Diagnostic simulation environment for testing LLM reasoning and tool use capabilities
agents
(10)Agents-A1
Open-source multimodal agent model with image-text reasoning on Qwen 3.5 MoE architecture
Code-Augur
Autonomous LLM agent for security vulnerability detection through specification inference
ImageEdit-R1
RL-powered multi-agent system for complex, instruction-based image editing
JetBrains Mellum2
12B parameter coding model with dedicated thinking mode for IDE integration
KARL
RL-trained enterprise search agents with multi-regime evaluation benchmark
MLEvolve
Self-evolving multi-agent system for automated ML algorithm discovery and optimization
QBugLM
Multi-agent framework for automated quantum software debugging with silent error detection
RepoLaunch
Automated repository testing agent for cross-language SWE dataset construction
ReViSQL
Text-to-SQL agent achieving human-level accuracy through step-by-step reasoning
Socratic-SWE
Self-improving coding agent that learns from its execution traces to evolve continuously
frameworks
(8)Ares
Adaptive reasoning framework that optimizes LLM inference costs by dynamically scaling effort
Automat
Autonomous feature engineering for materials science via LLM-driven genetic programming
CAST
Case-driven framework that learns from execution trajectories to optimize LLM tool use
EvoTool
Evolutionary framework that self-optimizes tool-use policies for long-horizon LLM agent tasks
Glite ARF
Verifier-driven framework for parallel LLM coding agents with built-in audit capabilities
HarnessX
Composable runtime scaffolding for adaptive AI agents that learn from execution traces
TITAN Agent
Full-featured TypeScript agent framework with multi-agent orchestration and Mission Control UI
Vellum
Visual AI workflow builder with SDK and enterprise governance