Augmenting Fuzzing With Metamorphic Testing: The Case Of REST APIs

Défense de mémoire de Eunice Mojuye Toukam

Metamorphic testing is a software testing approach that identifies and exploits relationships among multiple inputs and their corresponding outputs to detect inconsistencies in software behavior. This thesis explores the  integration of metamorphic testing with fuzzing techniques to enhance the testing of REST APIs. The primary aim is to address the research questions : (1) How can Metamorphic relations enhance the capacity of an API fuzzer in functional testing? and (2) How can Metamorphic relations enhance the capacity of an API fuzzer in non-functional testing?

In functional testing, Metamorphic Relations such as MROPEquality, MROPEquivalence, and MROPDisjoint are implemented to ensure consistent behavior across repeated operations, equivalent inputs, and resource isolation. These relations enable the detection of subtle and complex bugs that traditional testing methods often overlook. MROPEquality verifies the consistency of outputs from repeated identical inputs, MROPEquivalence ensures different but logically equivalent inputs produce the same results, and MROPDisjoint confirms that operations on one resource do not affect another.

For non-functional testing, MROPTimePerformance is employed to assess and maintain the performance and efficiency of API operations. This relation helps in identifying performance regressions and ensures that the API can handle requests efficiently under various conditions, highlighting issues such as unexpected delays or bottlenecks.

Integrating these Metamorphic Relations into Restler has demonstrated significant improvements in detecting issues in APIs. When testing APIs with the tool, several critical bugs were detected, including cross-data contamination where GET item1 returned data from item2, and improper handling of equivalent sequences where POST followed by DELETE item1 did not result in GET item1 returning "not found". Additionally, discrepancies were found in data consistency when performing equivalent operations, such as differing results from (POST, PUT) vs. POST, unintended data updates where PUT item1 inadvertently modified item2, and random delays experienced in GET requests. These issues were successfully identified using bug seeding, demonstrating the tool’s effectiveness in uncovering deep-seated API flaws.

However, the process involves challenges such as adapting grammar files and configurations for each unique API structure, indicating a need for more automated and flexible testing frameworks.

This research contributes to API testing by providing a systematic approach to uncovering a broader range of bugs, thereby enhancing the reliability and robustness of RESTful APIs.

 

A replication package with the implementation and evaluation data is available at the following

GitHub repository URL: https://github.com/vianeyMojuye/restler-fuzzer-mrops/tree/mrops

 

Keywords : API, Fuzzer, Metamorphic Relations

Contact : Isabelle Daelman - isabelle.daelman@unamur.be
Télecharger : vCal