Skip to content

Latest commit

 

History

History
146 lines (105 loc) · 7.43 KB

blackbox.md

File metadata and controls

146 lines (105 loc) · 7.43 KB
Raw

Black-Box Testing

RESTful APIs

Informally, in Black-Box Testing of a RESTful API, we generate test cases without knowing the internal details of the API. Still, we need to know the schema of the API to determine which endpoints can be called, and how. Otherwise, sending random bits on a TCP socket will unlikely result in any meaningful, well formatted HTTP message.

Even with a schema, there is still the issue of generating inputs, like HTTP query parameters and body payloads (e.g., in JSON/XML). This can be done at random, but still within the constraints defined in the schema (e.g., the structure of the body payloads).

To do this kind of testing, there is only the need to have an API up and running, and specify where the schema can be found. For example, https://apis.guru lists many APIs online. Such website provides an API itself to query info on existing APIs. Such small API (only 2 endpoints) can be easily tested by running the following on a command-line:

java -jar core/target/evomaster.jar  --blackBox true --bbSwaggerUrl https://api.apis.guru/v2/openapi.yaml  --outputFormat JAVA_JUNIT_4 --maxTime 30s --ratePerMinute 60

The command is doing the following:

  • java -jar core/target/evomaster.jar: execute the EvoMaster core process. The executable evomaster.jar must be either downloaded or built from source.
  • --blackBox true: by default, EvoMaster does white-box testing. Here, we specify that we do black-box testing instead.
  • --bbSwaggerUrl ...: URL of where the OpenAPI/Swagger schema is. The location of the API will be inferred from this schema (e.g., from host and servers tags). If such info is missing, then the API is assumed to be on same host as the schema. If needed, the API host location can be changed with the optional --bbTargetUrl (which overrides what specified in the schema).
  • --outputFormat JAVA_JUNIT_4: must specify how the tests will be generated, e.g., in Java using JUnit 4 in this case. Note: the language of the generated tests is not necessarily related to the language in which the tested application is implemented.
  • --maxTime 30s: for how long to run the search, i.e., just 30 seconds in this very simple example.
  • --ratePerMinute 60: avoid doing a DoS attack by bombarding the remote service with too many HTTP calls in quick rapid succession. Limit to max 1 per second (i.e., 60 per minute) in this example. Note: if you are testing an API running on your machine (e.g., on localhost) then this parameter is not only not required, but also detrimental for performance (i.e., do not use it).

This command will create the following test suite, in which 2 GET calls are executed:

public class EvoMasterTest {

    private static String baseUrlOfSut = "https://api.apis.guru";
    
    @BeforeClass
    public static void initClass() {
        RestAssured.enableLoggingOfRequestAndResponseIfValidationFails();
        RestAssured.useRelaxedHTTPSValidation();
        RestAssured.urlEncodingEnabled = false;
        RestAssured.config = RestAssured.config()
            .jsonConfig(JsonConfig.jsonConfig().numberReturnType(JsonPathConfig.NumberReturnType.DOUBLE))
            .redirect(redirectConfig().followRedirects(false));
    }

    
    @Test
    public void test_0() throws Exception {
        
        given().accept("application/json")
                .get(baseUrlOfSut + "/v2/list.json")
                .then()
                .statusCode(200)
                .assertThat()
                .contentType("application/json")
                .body("size()", numberMatches(1605));
    }
    
    
    @Test
    public void test_1() throws Exception {
        
        given().accept("application/json")
                .get(baseUrlOfSut + "/v2/metrics.json")
                .then()
                .statusCode(200)
                .assertThat()
                .contentType("application/json")
                .body("'numAPIs'", numberMatches(1605.0))
                .body("'numEndpoints'", numberMatches(46276.0))
                .body("'numSpecs'", numberMatches(2869.0));
    }
}

CLI Parameters

Note that there are several further parameters that can be configured (see documentation). These for example include options to specify where to store the generated files (--outputFolder), filter endpoints to test (e.g., --endpointPrefix), specify how test files will be named (e.g., --outputFilePrefix and --outputFileSuffix), etc.

Since version 3.0.0 these options can be specified in the generated em.yaml configuration files (so they do not need to be typed each time).

Issues with JDKs Above 8 and EvoMaster before version 3.2.0

The previous example run EvoMaster directly from its JAR file, using the command:

java -jar core/target/evomaster.jar

To do this, you need to have a JDK installed on your machine, version 8 or later. An easier approach is to download and install EvoMaster through its installer files (e.g., .msi for Windows), as those embed a JDK as well, with right configuration. Then EvoMaster can be run with for example evomaster.exe (for Windows).

If you want to run the JAR file directly with the JDK, you might encounter issues with versions from 17 on, due integrity constraints on the JDK. This is no longer the case since EvoMaster version 3.2.0. If for any reason you need to use an older version (not recommended!) on JDK 17 or later, you will have to manually add --add-opens commands like:

java --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED -jar core/target/evomaster.jar

If you fail to do that, EvoMaster will crash, but at least it will tell you what to do (with the most update requirements, in case more --add-opens are required since this documentation was written).

GraphQL APIs

Black-box fuzzing of GraphQL APIs uses the same options as for RESTful APIs. One difference is that --bbTargetUrl is used to specify the entry point of the GraphQL API. Another difference is that we must specify the --problemType to be GRAPHQL, as the default is REST. An example on GitLab's API is:

evomaster.exe  --problemType GRAPHQL --bbTargetUrl https://gitlab.com/api/graphql --blackBox true --outputFormat JAVA_JUNIT_4 --maxTime 30s --ratePerMinute 60

AUTH

Since version 1.3.0, it is possible to specify custom HTTP headers (e.g., to pass auth tokens), using the options from --header0 to --header2 (in case more than one HTTP header is needed). Many more options are now available since version 3.0.0. This is clarified in details in the authentication documentation.

WARNING

Black-box testing is easy to do: first download the tool, and then just specify where the OpenAPI/GraphQL schema can be found. However, as it does know nothing about the internal details of the tested application, it is unlikely that black-box testing will get good results in terms of code coverage. Albeit it can still detect many kinds faults (especially related to input validation).

The first time you try EvoMaster, use black-box testing to get an idea of what EvoMaster could do for you. However, after an initial trial, if your APIs run on the JVM, we recommend to switch to white-box testing, as it can lead to much, much better results. However, as previously stated, for the time being only programs running on the JVM (e.g., written in Java or Kotlin) are supported for white-box testing.