Skip to content
AImpact
IT EN
Inference Intermediate

HumanEval

/human-eval/

An OpenAI benchmark of 164 Python programming problems scored by running unit tests against the code generated by the model.

ShareLinkedInX

In practice

It was the standard for measuring LLM coding ability since 2021. It too is now saturated (over 90% pass@1), and the community has moved to SWE-bench, more realistic because it is based on real repositories.

Related terms

SWE-bench MMLU

Seen in the wild

5 entries mentioning it
  1. Qwen2.5-Coder-32B: the open source model that beats GPT-4o on code
    High
  2. Code Llama 70B: Meta brings the Llama 2 code branch to GPT-3.5 level
    Medium
  3. WizardCoder: evolutionary instructions for GPT-4 level code generation
    Medium
  4. Phi-1: 1.3B parameters beating models 10x larger on code
    High
  5. Codex paper: OpenAI publishes HumanEval and the model behind Copilot
    High
← All terms