Endorphin QA Agent Bench

An open set of hard browser challenges for QA agents. Every challenge has a goal in one sentence, the steps an Endorphin test would take, a clear pass condition, and its own page to run against.

Results

• Endorphin v2: endorphin v2.0.0-alpha.0, four challenges at a time; the grammar decides most steps, so the model was called 49 times.
• Endorphin v1: endorphin-ai 1.1.1 from npm, seven runs in parallel; an AI agent picks every action, so the model was called 2,090 times. It acted on 7 of the 18 must-refuse challenges.
• Endorphin Pro: Endorphin Go Pro (feature/accuracy-100), one challenge at a time; every step is decided by local models on this machine, nothing is sent to a cloud model. Four at a time it scored 53 because some clicks timed out under load.

A result is published only with the model it ran on, the date of the run, the time it took and what it cost. Time is the wall-clock time of the whole run; cost is what the model calls cost. A must-refuse challenge counts when the engine stops instead of claiming success.

Challenges by category

See all 161 challenges