OWASP Benchmark

The OWASP Benchmark Project is a Java and Python test suite designed to verify the speed and accuracy of vulnerability detection tools.

About OWASP Benchmark

What is in the Benchmark for Java?

The Benchmark for Java is a Java Maven project. Its primary component is thousands of test cases (e.g., BenchmarkTest00001.java), each of which is a single Java servlet that contains a single vulnerability (either a true positive or false positive). The vulnerabilities span about a dozen different CWEs currently.
An expectedresults.csv is published with each version of the Benchmark (e.g., expectedresults-1.2.csv) and it specifically lists the expected results for each test case. Here’s what the first two rows in this file looks like for version 1.1 of the Benchmark:
# test name		category	real vulnerability	CWE	Benchmark version: 1.1	2015-05-22
BenchmarkTest00001	crypto		TRUE				327
This simply means that the first test case is a crypto test case (use of weak cryptographic algorithms), this is a real vulnerability (as opposed to a false positive), and this issue maps to CWE 327. It also indicates this expected results file is for Benchmark version 1.1 (produced May 22, 2015). There is a row in this file for each of the tens of thousands of test cases in the Benchmark. Each time a new version of the Benchmark is published, a new corresponding results file is generated and each test case can be completely different from one version to the next.
The Benchmark for Java also comes with a bunch of different utilities, commands, and prepackaged open source security analysis tools, all of which can be executed through Maven goals, including:

Getting Started

Before downloading or using the Benchmark for Java make sure you have the following installed and configured properly: GIT: https://git-scm.com/ or https://github.com/ Maven: https://maven.apache.org/ (Version: 3.2.3 or newer works.) Java: https://www.oracle.com/java/technologies/javase-downloads.html (Java 7 or 8) (64-bit)

Getting, Building, and Running the Benchmark for Java

To download and build everything: $ git clone https://github.com/OWASP-Benchmark/BenchmarkJava $ cd BenchmarkJava $ mvn compile (This compiles it) $ runBenchmark.sh/.bat - This compiles and runs it. Then navigate to: https://localhost:8443/benchmark/ to go to its home page. It uses a self signed SSL certificate, so you’ll get a security warning when you hit the home page. Note: We have set the Benchmark for Java app to use up to 6 Gig of RAM, which it may need when it is fully scanned by a DAST scanner. The DAST tool probably also requires 3+ Gig of RAM. As such, we recommend having a 16 Gig machine if you are going to try to run a full DAST scan. And at least 4 or ideally 8 Gig if you are going to play around with the running Benchmark app.

Using a VM instead

We have several preconstructed VMs or instructions on how to build one that you can use instead: Or if you want to access from a different machine: docker-machine ls (in a different terminal) --> To get IP Docker VM is exporting (e.g., tcp://192.168.99.100:2376) Navigate to: https://192.168.99.100:8443/benchmark in your browser (using the above IP as an example) sudo yum install git sudo yum install maven sudo yum install mvn sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo sudo yum install -y apache-maven git clone https://github.com/OWASP-Benchmark/BenchmarkJava cd benchmark chmod 755 *.sh ./runBenchmark.sh -- to run it locally on the VM. ./runRemoteAccessibleBenchmark.sh -- to run it so it can be accessed outside the VM (on port 8443).

Running Free Static Analysis Tools Against the Benchmark for Java

There are scripts for running each of the free SAST vulnerability detection tools included with the Benchmark against the Benchmark test cases. On Linux, you might have to make them executable (e.g., chmod 755 *.sh) before you can run them. Generating Test Results for PMD: ./scripts/runPMD.sh (Linux) or .\scripts\runPMD.bat (Windows) Generating Test Results for FindBugs: ./scripts/runFindBugs.sh (Linux) or .\scripts\runFindBugs.bat (Windows) Generating Test Results for FindBugs with the FindSecBugs plugin: ./scripts/runFindSecBugs.sh (Linux) or .\scripts\runFindSecBugs.bat (Windows) In each case, the script will generate a results file and put it in the /results directory. For example: Benchmark_1.2-findbugs-v3.0.1-1026.xml This results file name is carefully constructed to mean the following: It’s a results file against the OWASP Benchmark for Java version 1.2, FindBugs was the analysis tool, it was version 3.0.1 of FindBugs, and it took 1026 seconds to run the analysis. NOTE: If you create a results file yourself, by running a commercial tool for example, you can add the version # and the compute time to the filename just like this and the Benchmark Scorecard generator will pick this information up and include it in the generated scorecard. If you don’t, depending on what metadata is included in the tool results, the Scorecard generator might do this automatically anyway.

Generating Scorecards

The scorecard generation application BenchmarkUtils (https://github.com/OWASP-Benchmark/BenchmarkUtils) needs to be cloned and installed with Maven. It parses the output files generated by any of the supported security tools run against the Benchmark and compares them against the expected results, and produces a set of web pages that detail the accuracy and speed of the tools involved. For the list of currently supported tools, check out the: Tools Support/Results tab. If you are using a tool that is not yet supported, simply send us a results file from that tool and we’ll write a parser for that tool and add it to the supported tools list. The following command will compute a Benchmark scorecard for all the results files in the /results directory. The generated scorecard is put into the /scorecard directory. createScorecard.sh (Linux) or createScorecard.bat (Windows) An example of a real scorecard for some open source tools is provided at the top of the Tool Support/Results tab so you can see what one looks like. We recommend including the Benchmark version number in any results file name, in order to help prevent mismatches between the expected results and the actual results files. A tool will not score well against the wrong expected results. Customizing Your Scorecard Generation The createScorecard scripts are very simple. They only have one line. Here’s what the 1.2 version looks like: org.owasp:benchmarkutils-maven-plugin:create-scorecard This Maven command simply says to run the BenchmarkScore application. It automatically finds the expected results file in the same folder and uses it to generate the scorecard. In all cases, the generated scorecard is put in the /scorecard folder. WARNING: If you generate results for a commercial tool, be careful who you distribute it to. Each tool has its own license defining when any results it produces can be released/made public. It is likely to be against the terms of a commercial tool’s license to publicly release that tool’s score against the OWASP Benchmark. The OWASP Benchmark project takes no responsibility if someone else releases such results. It is for just this reason that the Benchmark project isn’t releasing such results itself.

Project Resources

Project Leaders

Dave Wichers

Project Leader

Project Information

Lab Project

Classification

Tool

Language

Java

License

GNU General Public License v2.0

Contributors

GitHub Stars

824

Downloads

Corporate Supporters

OWASP is a nonprofit foundation improving software security through open-source projects, global communities, and education. All resources are free and open to everyone.

OWASP, the OWASP logo, and Global AppSec are registered trademarks and AppSec Days, AppSec California, AppSec Cali, SnowFROC, OWASP Boston Application Security Conference, and LASCON are trademarks of the OWASP Foundation, Inc.