Skip to content
AImpact
IT EN
Inference Intermediate Also known as: Software Engineering Bench

SWE-bench

/swee-bench/

A benchmark of over 2,000 real GitHub issues from Python repositories: the model must produce a patch that makes the project's tests pass.

ShareLinkedInX

In practice

It measures real software-engineering ability (reading a codebase, debugging, cross-file edits), not isolated coding. It has become the reference for agents like Devin, Claude Code, and OpenAI Codex.

Related terms

HumanEval Agent

Seen in the wild

7 entries mentioning it
  1. SWE-bench Verified: OpenAI cleans up the reference benchmark for coding agents
    Medium
  2. Agentless: less agent complexity, more results on SWE-bench
    Medium
  3. DeepSeek-Coder-V2: GPT-4 Turbo coding quality with open weights
    High
  4. Aider: CLI coding agent with automatic git integration and SOTA benchmark
    High
  5. SWE-agent: an AI agent that resolves real GitHub issues at 12.5%
    High
  6. Devin: the first 'autonomous AI engineer' goes viral
    High
  7. Devin: 13.86% on SWE-bench, the first autonomous AI software engineer
    Landmark
← All terms