Are your dependencies code reviewed?: Measuring code review coverage in dependency updates

Abstract

As modern software extensively uses free open source packages as dependencies, developers have to regularly pull in new third-party code through frequent updates. However, without a proper review of every incoming change, vulnerable and even malicious code can sneak into the codebase through these dependencies. The goal of this study is to aid developers in securely accepting dependency updates by measuring if the code changes in an update have passed through a code review process. We implement DepDive, an update audit tool for packages in Crates.io, npm, PyPI, and RubyGems registry. DepDive first (i) identifies the files and the code changes in an update that cannot be traced back to the package's source repository, i.e., phantom artifacts; and then (ii) measures what portion of changes in the update, excluding the phantom artifacts, has passed through a code review process, i.e., code review coverage.Using DepDive, we present an empirical study across the latest ten updates of the most downloaded 1000 packages in each of the four registries. Our study unveils interesting insights while also providing an evaluation of our proposed approach. We find that phantom artifacts are not uncommon in the updates (20.1% of the analyzed updates had at least one phantom file). The phantoms can appear either due to legitimate reasons, such as in the case of programmatically generated files, or from accidental inclusion, such as in the case of files that are ignored in the repository. However, without provenance tracking, we cannot audit if the changes in these phantom artifacts were code-reviewed or not.Regarding code review coverage (CRC), we find the updates are typically only partially code-reviewed (52.5% of the time). Further, only 9.0% of the packages had all their updates in our data set fully code-reviewed, indicating that even the most used packages can introduce non-reviewed code in the software supply chain. We also observe that updates either tend to have very high CRC or very low CRC, suggesting that packages at the opposite end of the spectrum may require a separate set of treatments.