RedAmon

Benchmark explainer

What is the XBOW validation benchmark?

The XBOW validation benchmark is a public set of 104 web-application security challenges, XBEN-001-24 to XBEN-104-24, that XBOW released under the Apache 2.0 license in 2024. Each challenge is a small Docker Compose app that hides one flag behind a real, exploitable vulnerability. An AI agent solves a challenge only when it recovers that exact flag from the running app.

Last updated

challenges, XBEN-001-24 to XBEN-104-24
104challenges, XBEN-001-24 to XBEN-104-24
flag hidden in each one
1flag hidden in each one
difficulty levels
3difficulty levels
license, free to use
Apache 2.0license, free to use
released by XBOW
2024released by XBOW

What vulnerabilities does it cover?

The challenges span the classes a web pentest meets in practice. Counts are by tag across all 104 challenges, and a challenge can carry several tags, so they add up to more than 104. Single challenges also cover cryptography, broken authentication, NoSQL injection, race conditions and HTTP request smuggling.

Vulnerability classChallenges
Cross-site scripting (XSS)23
Default credentials15
Insecure direct object reference (IDOR)14
Server-side template injection (SSTI)13
Privilege escalation11
Command injection10
Remote code execution8
SQL injection7
Insecure deserialization6
Local file inclusion6
Path traversal6
Information disclosure6
Business logic flaws6
Arbitrary file upload5
Known CVEs4
JWT, GraphQL, SSRF, XXE3 each

How is a challenge scored?

Every challenge hides exactly one flag of the form FLAG{…}, injected when the image is built, so it cannot be looked up in the source in advance. Most challenges have a flag win condition; a few ask a question instead, and the answer string is what counts. Either way the rule is the same: the expected string must appear, verbatim, in the agent's output.

The benchmark sets no other rules: no time limit, no cap on tool calls or tokens, no required model and no limit on attempts.

Black-box or white-box: why it matters

The challenges were written as Capture The Flag tasks: each ships a description of the kind a competitor would be given, and its full source. A run that shows the agent any of that is white-box. In a black-box run the agent gets only the target URL and has to discover everything else, so the harness must hide all of the following, because each one leaks the answer or the vulnerability class.

  • The challenge metadata: name, description, tags, level

    It names the vulnerability class, and the flag is derived from the challenge name.

  • The README in each challenge folder

    Many carry a deliberate hint, such as where to look or what to fuzz.

  • The application source code

    It shows the exploit path and where the flag is stored.

  • The XBEN identifier

    It is public, so one web search returns write-ups of the solution.

  • Descriptive Docker service names

    A host named after the vulnerability gives the class away. An opaque alias such as lab-XXXXXXXX does not.

How do you run it yourself?

  1. 01

    Get the challenges

    Clone xbow-engineering/validation-benchmarks. You need Docker, the Docker Compose plugin and openssl.

  2. 02

    Build and start one

    In a challenge folder, make build injects the flag at build time (make build FLAG=… sets your own) and make run starts it and waits until its health checks pass.

  3. 03

    Keep it isolated

    The apps contain deliberate, unpatched vulnerabilities. Run them on an isolated network, never on a production interface, and give the agent only the exposed URL.

  4. 04

    Log every fix

    A few images need small compatibility fixes on current Docker, such as pointing old Debian package mirrors to the archive. Record each one, and never touch the vulnerability, the win condition or the flag.

The repository embeds canary strings so its data can be detected in AI training sets. Don't paste challenge source or metadata into public posts or chatbots: publish results only.

How did RedAmon do?

RedAmon, an open-source AI penetration testing framework, solved 101 of the 104 challenges fully black-box. The agent received only an opaque http://lab-XXXXXXXX/ URL, with no credentials, source code, README hint or challenge name. Every solve ships the complete, unedited agent transcript and a decoded step-by-step walkthrough, so anyone can check how each flag was recovered.

A solved challenge took about 23 minutes of wall time and 0.75 million tokens on average. Most runs used DeepSeek V4 Pro, the rest DeepSeek V4 Flash and Kimi K3. The three it did not solve are XBEN-066-24 (HTTP request smuggling), XBEN-089-24 (privilege escalation through business logic) and XBEN-101-24 (cryptography).

Full scorecard and transcriptsHow the agent works

Frequently asked questions

Who created the XBOW validation benchmark?

XBOW released it in 2024. Its README says external contractors built the challenges to mirror the vulnerability classes the XBOW security team meets in routine pentesting and bug bounty work, and that the set was kept confidential until release so no model had trained on it.

Is the XBOW benchmark free to use?

Yes. It is licensed under Apache 2.0. If you redistribute it, keep the LICENSE and NOTICE files and mark your changes. The license does not let you use the XBOW trademark to imply endorsement.

How many challenges does the XBOW benchmark have?

104, numbered XBEN-001-24 to XBEN-104-24. Each one is a Docker Compose stack with a single objective, rated level 1 (easy), 2 (medium) or 3 (hard) by its author.

What is the XBOW benchmark used for?

Testing AI pentest agents against real, exploitable web vulnerabilities in a free, reproducible lab. When the full transcripts are published, it also shows step by step how an agent finds and exploits each flag.

What does fully black-box mean on the XBOW benchmark?

The agent gets only the target URL. It does not see the source code, the challenge name, description, tags or README hints. The XBEN identifier is hidden too, because the flag is derived from the challenge name and write-ups are one web search away.

Where can I download the XBOW benchmark?

From github.com/xbow-engineering/validation-benchmarks. Run it only on an isolated network: the apps contain deliberate, unpatched vulnerabilities.

Attribution

Results on this page come from RedAmon's XBOW benchmark evaluation, published with this notice:

RedAmon is evaluated against the XBOW Validation Benchmarks
(https://github.com/xbow-engineering/validation-benchmarks),
Copyright 2024- XBOW USA Inc., Apache License 2.0.
The benchmarks are used as-is except for minimal compatibility fixes
required by modern Docker and by retired Debian suites (compose-syntax
updates and image-rot fixes such as redirecting EOL apt repositories to
archive.debian.org, all logged at xbow-patches/), which do not alter the
attack surface or the flag.
RedAmon and XBOW are independent projects; this evaluation is not endorsed
or certified by XBOW.