A Black Box Tool for Robustness Testing of REST Services

Abstract

REST services are nowadays being used to support many businesses, with most major companies exposing their services via REST interfaces (e.g., Google, Amazon, Instagram, and Slack). In this type of scenarios, heterogeneity is prevalent and software is sometimes exposed to unexpected conditions that may activate residual bugs, leading service operations to fail. Such failures may lead to financial or reputation losses (e.g., information disclosure). Although techniques and tools for assessing robustness have been thoroughly studied and applied to a large diversity of domains, REST services still lack practical approaches that specialize in robustness evaluation. In this paper, we present a tool (named bBOXRT) for performing robustness tests over REST services, solely based on minimal information expressed in their interface descriptions. We used bBOXRT to evaluate an heterogeneous set of 52 REST services that comprise 1,351 operations and fit in distinct categories (e.g., public, private, in-house). We were able to disclose several different types of robustness problems, including issues in services with strong reliability requirements and also a few security vulnerabilities. The results show that REST services are being deployed preserving software defects that harm service integration, and also carrying security vulnerabilities that can be exploited by malicious users.

foREST: A Tree-based Approach for Fuzzing RESTful APIs

“…Based on the previous results, we can conclude that foREST has non-trivial performance advantages over RESTler and EvoMaster in fuzzing RESTful APIs. Note that that are also other tools that support RESTful API fuzzing, such as RestTestGen [25] and bBOXRT [18]. However, an empirical study [8] has shown that these tools are either preliminary or not robust compared to RESTler, and RESTler is the best one over them.…”

Empirical Comparison of Black-box Test Case Generation Tools for RESTful APIs

“…bBOXRT is a black-box robustness testing tool for RESTful APIs proposed by Laranjeiro et al [7], written in Java and available on the authors' website 7 . The aim of bBOXRT is to assess the robustness of REST APIs observing the behavior of services under test when providing invalid requests.…”

“…To the best of our knowledge, the only black-box testing approaches for REST APIs which provide an implementation, i.e., a usable testing tool, are the one we have taken into account in our comparison (RestTestGen [5], RESTler [6], bBOXRT [7] and RESTest [8], presented in Section III). Regarding test coverage, the only work proposing a systematic approach to assess the coverage of REST APIs testing tools is the framework of Martin-Lopez et al [3], that we have taken as basis for our comparison.…”

“…https://github.com/microsoft/restler-fuzzer (cloned on Dec. 27th, 2020) 7. https://eden.dei.uc.pt/ ∼ cnl/papers/2020-access.zip 8 https://github.com/isa-group/RESTest/ (cloned on Dec. 27th, 2020).…”

Harvesting Production GraphQL Queries to Detect Schema Faults

“…Recently, deep learning models have been proposed to determine the validity of these inputs [32]. The generated tests can assess the robustness of the API through invalid requests [33], detect regressions across API versions [34], verify the data dependencies among sequences of requests [35], or verify the constraints imposed on their parameters [36]. Metamorphic relations among requests may also serve as the oracle [37].…”