E

LLM Evaluation Tools

EvalPlus Software Review

github.com · checked 11 Oct 2026

Open-source framework for checking code generated by large language models.

How it plugs in

—Public API
NoOpen source
YesFree tier

Signal on LLM Evaluation Tools5.8#18 of 29

EvalPlus evaluates code written by large language models. HumanEval and MBPP receive expanded test coverage, while EvalPerf measures code efficiency and the project publishes result leaderboards. You can install it with pip, run evaluations in Docker, or host it yourself on Linux. It supports several model-serving backends and is free to use.

EvalPlus home page

Review

In a line

EvalPlus is an open-source framework for assessing code produced by large language models. The project is designed for teams assessing model performance on coding tasks.

What it does

The framework covers HumanEval, MBPP, and EvalPerf evaluations. HumanEval+ increases the original HumanEval test set by a factor of 80, while MBPP+ expands the original MBPP set 35-fold.

EvalPerf checks how efficiently generated code runs by using tasks and test inputs designed to exercise performance. The project also describes RepoQA, which focuses on evaluating understanding of code across long files or projects. EvalPlus publishes leaderboards covering HumanEval+, MBPP+, and EvalPerf outcomes.

How it connects

EvalPlus is available via an API, on Linux, and as a self-hosted deployment. Its documented backends cover Hugging Face Transformers, vLLM, OpenAI-compatible servers, OpenAI, Anthropic, Google, Amazon Bedrock, Ollama, and Intel Gaudi.

The code is open source under the Apache License 2.0. Files in the evalplus/eval folder also follow the MIT License. You can install it with pip, adding optional vLLM and performance extras when needed. Docker examples show how to run code evaluation in a container.

Good fit

EvalPlus suits model teams and other people who need repeatable checks for code-generating language models. It is useful when benchmark coverage, efficiency checks, self-hosting, or Docker-based execution matter.

Watch for

Because EvalPerf is designated *nix only, verify that your operating system is supported before using its efficiency tests.

You will need to choose a supported model backend and decide whether to install with pip, run evaluations in Docker, or host the framework yourself.

Pricing

EvalPlus is free and open source. There is no paid plan listed in the facts.

Verdict

Choose EvalPlus if you need an open tool for checking generated code against expanded HumanEval or MBPP tests, measuring execution efficiency, or viewing benchmark results. Its range of documented backends and self-hosted options can fit teams that control their own setup.

Look elsewhere if you need a hosted service, a broad non-code evaluation suite, or efficiency testing on an operating system outside the stated *nix support.

Our take

EvalPlus fits teams assessing large language models on programming tasks that need more extensive HumanEval and MBPP coverage. It also adds a separate efficiency evaluation and publishes leaderboards for its main tests. Free, open-source code, Docker examples, and self-hosting give you control over deployment. Before using it, account for the *nix requirement on EvalPerf and configure one of the documented model backends. It is not a general-purpose hosted testing service.

  • Free and open-source framework
  • HumanEval+ provides 80 times the original HumanEval test coverage
  • MBPP+ provides 35 times the original MBPP test coverage
  • Includes code-efficiency evaluation through EvalPerf
  • Supports Docker-based code execution
  • EvalPerf runs only on *nix systems
  • Requires your own setup and model backend
  • Focused on code-related language-model evaluation
  • Pick it ifTeams benchmarking models that generate code; Self-hosted evaluation workflows; Checking code quality and execution efficiency
  • Skip it ifYou need a hosted evaluation service; You need tests unrelated to generated code; Your system cannot run the *nix-only EvalPerf evaluation

Pricing

EvalPlus is free and open source, with no paid plan listed. You can install it with pip or run it in your own Docker or self-hosted setup.

Questions

What is EvalPlus?

EvalPlus is an open-source framework for evaluating code generated by large language models.

Which benchmarks does EvalPlus support?

Its evaluation coverage includes HumanEval, MBPP, and EvalPerf. HumanEval+ and MBPP+ enlarge the original HumanEval and MBPP test sets.

How many extra tests do HumanEval+ and MBPP+ add?

HumanEval+ contains 80 times the original HumanEval test coverage, and MBPP+ contains 35 times the original MBPP coverage.

Can EvalPlus measure code efficiency?

Yes. EvalPerf uses performance-focused tasks and test inputs to evaluate the efficiency of generated code.

Which model backends are documented?

Documented options include Hugging Face Transformers, vLLM, OpenAI-compatible servers, OpenAI, Anthropic, Google, Amazon Bedrock, Ollama, and Intel Gaudi.

How can I install or run EvalPlus?

The README describes pip installation with optional vLLM and performance extras, along with Docker-based code evaluation. Self-hosted deployment is supported.

Is EvalPlus free?

Yes. The project is free and open source under Apache License 2.0; files in evalplus/eval additionally comply with the MIT License.