SUMMARY Regression testing can be performed by reexecuting a test suite on different software versions and comparing the outcome. For functional testing, the outcome of such tests is either pass (correct behavior) or fail (incorrect behavior). For nonfunctional testing, such as performance testing, this is more challenging as correct and incorrect are not clearly defined concepts for these types of testing. In this paper, we present an approach for detecting and analyzing input/output (I/O) performance regressions. Our method is supplemental to existing profilers, and its goal is to analyze the effect of source code changes on the performance of a system. In this paper, we focus on analyzing the amount of I/O writes being performed. The open source implementation of our approach, SPECTRAPERF, is available for download. We evaluate our approach in a field user study on Tribler, an open source peer‐to‐peer client and its decentralized solution for synchronizing messages, Dispersy. In this evaluation, we show that our approach can guide the performance optimization process, as it helps developers to find performance bottlenecks on the one hand and, on the other, allows them to validate the effect of performance optimizations. In addition, we perform a feasibility study on Django, the most popular Python project on Github, to demonstrate our applicability on other projects. Copyright © 2014 John Wiley & Sons, Ltd.