CyberSecEval 4 Review
meta-llama.github.io · checked 15 Oct 2026
A free, open-source benchmark suite for testing cybersecurity risks and defenses in large language models.
How it plugs in
Signal on AI Guardrail Software7.6#2 of 23
CyberSecEval 4 is Meta’s free benchmark suite for measuring cybersecurity risks and defensive abilities in large language models. It covers attack-request compliance, false refusals, insecure code, prompt injection, code-interpreter abuse, security operations, and automated patching. It runs through Python on Linux, an API, or self-hosted models, with support for several model providers.

Review
In a line
CyberSecEval is Meta’s benchmark suite for checking cybersecurity weaknesses and defensive capabilities in large language models. CyberSecEval 4 adds tests for agents that generate patches for vulnerable C and C++ code.
What it does
The suite measures whether models comply with cyberattack requests, refuse borderline safe questions incorrectly, suggest insecure code, or follow injected instructions. Prompt-injection tests include text, multiple languages, and images, with another language model judging whether the attack worked.
It also tests malicious prompts sent to code interpreters, automated patch generation for fuzzed C and C++ vulnerabilities, malware analysis, and threat-intelligence reasoning. Its metrics cover cybersecurity risk, insecure-code suggestion frequency, and help given to attackers.
How it connects
CyberSecEval runs from a Python command-line module in a Python 3.10 virtual environment. It supports API specifications for OpenAI, Google, Anthropic, Llama, and Together, plus custom OpenAI-compatible base URLs. The getting-started guide also lists OpenAI, Anyscale, and Together, and explains how to add self-hosted providers.
The code is open source under the MIT license for research and commercial use. Self-hosted models can be connected by adding query, chat, and system-prompt chat methods. AutoPatch requires Podman, and threat-intelligence report processing may require Poppler.
Good fit
CyberSecEval suits security teams, model developers, and researchers who need repeatable checks for cybersecurity behavior in language models. It is also suited to organizations building responsibly with open generative AI models.
The suite is useful when you need coverage of prompt attacks, jailbreaks, unsafe outputs, insecure coding, offensive operations, or security-operations tasks.
Watch for
AutoPatch needs substantial resources: the page recommends at least 80 CPUs and 8 TB of storage. Apple Silicon MacBooks are not currently supported for this benchmark.
The insecure-code detector uses 189 rules and cannot understand code context, which can create false positives. A related critique reports a narrower Semgrep ruleset than its comparison repository and found some prompts measured refusal behavior instead of genuine insecure-code tendency.
Pricing
CyberSecEval 4 is free and open source.
Running benchmarks can still use compute, storage, or model API capacity. One reported estimate puts a full one-epoch CyberSecEval 4 run at roughly $29 to $213 per model, depending on the model.
Verdict
Choose CyberSecEval if you need a broad, inspectable way to measure how language models handle cyberattacks, unsafe code, prompt injection, and defensive security work. Its MIT license and provider extensions make it practical for teams that can run Python tools and manage their own infrastructure.
Skip it if you need a lightweight hosted scanner, cannot provide the resources required by AutoPatch, or need code analysis that understands program context.
Our take
CyberSecEval is a strong fit for teams comparing language models on security behavior with an open benchmark they can run and extend. It covers more than one attack type and includes defensive tasks, but setup is technical and some tests have known measurement limits. The free software does not remove infrastructure or API costs, and AutoPatch has demanding hardware requirements. Choose it when breadth and self-hosting matter; avoid it when you need a simple hosted product or context-aware code review.
- Free and open source under the MIT license
- Covers prompt injection, jailbreaks, unsafe code, and attack-request compliance
- Supports several model APIs and self-hosted providers
- Includes security-operations and automated patching benchmarks
- AutoPatch recommends at least 80 CPUs and 8 TB of storage
- Apple Silicon MacBooks are not supported for AutoPatch
- The insecure-code detector can produce false positives
- Some prompts may test refusal behavior instead of insecure-code propensity
- Pick it ifSecurity teams evaluating language-model behavior; Model developers needing self-hosted, repeatable benchmarks; Researchers comparing cybersecurity defenses
- Skip it ifYou need a hosted scanner with little setup; You cannot provide AutoPatch’s compute and storage; You need code analysis that understands full program context
Pricing
CyberSecEval 4 is free and open source. Benchmark runs may still require paid model API usage and infrastructure; one estimate puts a full one-epoch run at roughly $29 to $213 per model.
- CyberSecEval 4Open source benchmark suiteFree
As github.com lists them, checked 8 Oct 2026.
Questions
Is CyberSecEval free?
Yes. CyberSecEval 4 is listed as a free, open-source benchmark suite under the MIT license.
Which models can CyberSecEval connect to?
The runner supports OpenAI, Google, Anthropic, Llama, and Together specifications, plus custom OpenAI-compatible base URLs. Self-hosted models can also be added.
What security tests does it include?
It tests attack-request compliance, false refusals, insecure code, prompt injection, jailbreaks, unsafe outputs, code-interpreter abuse, security operations, and automated patching.
What does AutoPatchBench require?
AutoPatchBench needs Podman and substantial compute and storage. The page recommends at least 80 CPUs and 8 TB of storage, and Apple Silicon MacBooks are not supported.
What operating systems does it run on?
The listed deployment modes are API, Linux, and self-hosted. The setup uses Python 3.10 and a virtual environment.
Can I add my own model provider?
Yes. Developers can add a provider by implementing an LLM class, registering it in the factory, and supplying query and chat methods.
Near it
All alternatives →- NN
NVIDIA NeMo Guardrails#1 on this board - P
PromptGuard#3 on this board - OMOracle Mobile Authenticator#4 on this board
- OPOpenSecureAI Prompt Firewall#5 on this board
- VVotalAI#6 on this board
- GA
Guardrails AI#7 on this board - EA
Enkrypt AI Guardrails#8 on this board - FG
Fiddler Guardrails#9 on this board - PAPangea AI Guard#10 on this board
- OGOpenlayer Guardrails#11 on this board