WizardCoder-Python-34B: 73.2% on HumanEval with Evol-Instruct applied to code

In one sentence The WizardLM team releases WizardCoder-Python-34B-V1.0, scoring 73.2% pass@1 on HumanEval and, according to the authors, surpassing GPT-4 (March 15, 2023 version), ChatGPT-3.5 and Claude 2. It follows June's WizardCoder-15B, introduced in the paper that adapts Evol-Instruct to code.

Verified Official source

ShareLinkedIn X

Evol-Instruct is a technique from the WizardLM team: you take a set of seed instructions, ask a language model to rewrite them into progressively harder versions, then train a model on these evolved examples. WizardCoder applies the idea to programming.

The first version, WizardCoder-15B (June 2023), according to the authors outperformed closed models such as Claude-Plus and Bard on HumanEval. On August 26, 2023, two days after Meta released Code Llama, WizardCoder-Python-34B-V1.0 arrives with 73.2% pass@1 on HumanEval.

The GPT-4 comparison needs care: 73.2% beats the figure published for GPT-4 in March 2023, but later measurements of GPT-4 on the same benchmark are higher (88.4 in the table of the same repository).