Experimental evaluation in computer science: A quantitative study

Abstract

A survey of over 400 recent research articles suggests that computer scientists publish relatively few papers with experimentally validated r esults.The survey includes complete volumes of several refereed c omputer science journals, a conference, and 50 titles drawn at random from all articles published b y A CM in 1993. The journals Optical Engineering OE and Neural Computation NC were used for comparison.Of the papers in the random sample that would require experimental validation, 40 have none at all. In journals related to software engineering, this fraction is over 50 . In comparison, the fraction of papers lacking quantitative evaluation in OE and NC is only 15 and 12 , respectively.Conversely, the fraction of papers that devote one fth or more of their space to experimental validation is almost 70 for OE and NC, while it is a mere 30 for the CS random sample and 20 for software engineering.The low ratio of validated r esults appears to be a serious weakness in computer science r esearch. This weakness should be r e cti ed for the long-term health of the eld.

On the Status of Experimental Research on the Semantic Web

“…For this purpose, we took the papers published at ISWC since 2002 and manually classified them according to the scheme proposed by [5]. In addition, we had a closer look at papers containing descriptions of experimental work with respect to the data used and the claims made.…”

“…In order to be able to compare our findings to previous studies on the role of experimental work in computer science as a whole, we used the classification scheme proposed in [5] with the modifications described in [11], i.e. the merge of the two categories 'Empirical Work' and 'Hypothesis Testing'.…”

“…While the amount of such papers is lower in certain subareas of computer science, like software engineering (55%), still a significant amount of work in computer science is formulative and requires some evaluation. So far, the most detailed and systematic investigation of experimental research as a means for evaluating formulative research has been carried out by Tichy and others in 1995 [5]. Based on a sample of publications from major computer science journals the authors categorize papers into formal theory, design and modeling, as well as empirical work and others.…”

“…As described above, we take the relative number of pages describing experiments (RELPAGES) as a general proxy for the importance of the experimental part within a paper. Following [5], we thus make the assumption that the better the experiments in a paper are, the more the paper reports about the experiments.…”

“…In particular, we present descriptive statistics of the ISWC paper collection and results of investigating possible correlations with research impact. [5] refer to papers published in 1995, and [6] used papers from 2005, our study covers 11 consecutive years from 2002 -2012. This also explains the comparably high number of papers (500) included in our study.…”