UK and US AISI publish the pre-deployment evaluation of the new Claude 3.5 Sonnet
In one sentence On November 19, 2024 the UK and US AI Safety Institutes publish their first joint pre-deployment evaluation: the upgraded Claude 3.5 Sonnet, tested on biology, cyber, software and AI development and safeguard efficacy, compared with the previous Sonnet, o1-preview and GPT-4o.
Before Anthropic released the upgraded Claude 3.5 Sonnet (October 22, 2024), the UK and US government AI safety institutes had limited access to the model and tested it independently. The results were shared with Anthropic before launch and published on November 19.
The tests covered four areas: biological capabilities, offensive cyber capabilities, software and AI development, and safeguard efficacy. To measure how much the model had improved, it was compared with the previous Sonnet, o1-preview and OpenAI's GPT-4o.
Among the results: the new Sonnet solves more entry-level cyber tasks than the reference models, and its built-in safeguards can often be bypassed with known jailbreaks. The institutes describe the findings as preliminary.
Companies
UK AI Safety Institute, US AI Safety Institute, Anthropic
Tools
Claude 3.5 Sonnet
Tags
Sources