HarmBench: standardized benchmark for evaluating LLM jailbreaks and defenses
In one sentence UCSB publishes HarmBench: 400+ harmful behaviors, 18 attack methods, 33 models tested. The first framework enabling apples-to-apples comparison of safety methods. Reveals that most safety fine-tuning is easily circumvented.
How do you know if an AI model is truly safe against jailbreaks? Before HarmBench, the honest answer was: you cannot reliably compare. Every research paper used its own tests, success criteria, and models. It was like comparing athletes across different disciplines without ever using the same track.
HarmBench, developed by researchers at the University of California Santa Barbara, created the first standardized playing field for this problem. It defined a set of over 400 harmful behaviors divided into categories — weapons, cyberattacks, illegal content, manipulation — and tested 18 different attack methods on 33 different models.
The results sparked debate: almost all models tested, including those with advanced safety training, were vulnerable to at least some attack methods. In many cases, sophisticated techniques like GCG (Greedy Coordinate Gradient) or PAIR succeeded in bypassing protections systematically.
HarmBench is not a tool for causing harm — it is a tool for measuring safety in a reproducible way. Like crash tests for cars: it tells you how robust the system is before it falls into the hands of someone who wants to misuse it.
Companies
UCSB
Tools
—
Tags
Sources