Multifactor Citation Analysis over Five Years: A Case Study of SIGMETRICS Papers

Abstract

Performance evaluation is a broad discipline within computer science, combining deep technical work in experimentation, simulation, and modeling. The field’s subjects encompass all aspects of computer systems, including computer architecture, networking, energy efficiency, and machine learning. This wide methodological and topical focus can make it difficult to discern what attracts the community’s attention and how this attention evolves over time. As a first attempt to quantify and qualify this attention, using the proxy metric of paper citations, this study looks at the premier conference in the field, SIGMETRICS. We analyze citation frequencies at monthly intervals over a five-year period and examine possible associations with myriad other factors, such as time since publication, comparable conferences, peer review, self-citations, author demographics, and textual properties of the papers. We found that in several ways, SIGMETRICS is distinctive not only in its scope, but also in its citation phenomena: papers generally exhibit a strongly linear rate of citation growth over time, few if any uncited papers, a large gamut of topics of interest, and a possible disconnect between peer-review outcomes and eventual citations. The two most-cited papers in the dataset also exhibit larger author teams, higher than typical self-citations, and distinctive citation growth curves. These two papers, sharing some coauthors and a research focus, could either signal the area where SIGMETRICS had the most research impact, or they could represent outliers; their omission from the analysis reduces some of the otherwise distinctive observed metrics to nonsignificant levels.

Citation analysis of computer systems papers

“…It did find, as this study confirms, that papers that are electronically available tend to collect more citations. In another single-conference study, Frachtenberg (2022a) analyzed the citations of papers in the SIGMETRICS’17 conference and found similar citation dynamics to those of the larger systems field in this more generalized study. Similarly focused, but taking a reverse perspective, Lister & Box (2008) looked at the outgoing citations of the SIGCSE’07 conference and examined in detail the venues and venue types of those cited papers.…”

“…The bibliometrics literature is rich with studies analyzing citations in various disciplines and fields, including CS as a whole ( Devarakonda et al, 2020 ; Hirst & Talent, 1977 ; Mattauch et al, 2020 ). Even within CS, several fields, subfields, and specific venues have received bibliometric analyses ( Broch, 2001 ; Frachtenberg, 2022a ; Iqbal et al, 2019a ; Iqbal et al, 2019b ; Lister & Box, 2008 ; Rahm & Thor, 2005 ; Wang et al, 2016 ). The purpose of this article is to apply these methods to the CS field of computer systems and to understand the distribution and characteristics of its citations.…”