Reporting computing projects through structured abstracts: a quasi-experiment

Abstract

Previous work has demonstrated that the use of structured abstracts can lead to greater completeness and clarity of information, making it easier for researchers to extract information about a study. In academic year 2007/08, Durham University's Computer Science Department revised the format of the project report that final year students were required to write, from a 'traditional dissertation' format, using a conventional abstract, to that of a 20-page technical paper, together with a structured abstract. This study set out to determine whether inexperienced authors (students writing their final project reports for computing topics) find it easier to produce good abstracts, in terms of completeness and clarity, when using a structured form rather than a conventional form. We performed a controlled quasi-experiment in which a set of 'judges' each assessed one conventional and one structured abstract for its completeness and clarity. These abstracts were drawn from those produced by four cohorts of final year students: two preceding the change, and the two following. The assessments were performed using a form of checklist that is similar to those used for previous experimental studies. We used 40 abstracts (10 per cohort) and 20 student 'judges' to perform the evaluation. Scored on a scale of 0.1-1.0, the mean for completeness increased from 0.37 to 0.61 when using a structured form. For clarity, using a scale of 1-10, the mean score increased from 5.1 to 7.2. For a minimum goal of scoring 50% for both completeness and clarity, Empir Software Eng (2011) 16:244-277 245 only 3 from 19 conventional abstracts achieved this level, while only 3 from 20 structured abstracts failed to reach it. We conclude that the use of a structured form for organising the material of an abstract can assist inexperienced authors with writing technical abstracts that are clearer and more complete than those produced without the framework provided by such a mechanism.

Robust Statistical Methods for Empirical Software Engineering

“…This suggests that the median score is increased by approximately 0.1 which is equivalent to getting one additional Yes answer in the 8 completeness questions. In contrast, Budgen et al (2011) observed a median difference between conventional and structured abstracts of just over 0.2 using a similar scoring method. In addition, the abstracts in the Budgen study were all written by undergraduates who would have had little experience of writing abstracts, whereas the abstracts in our multi-site experiment were written by the authors of the papers.…”

“…These were respectively assessed using a set of 8 questions similar to those employed in previous studies, Budgen et al (2011Budgen et al ( , 2008, and a 10-point Likert-like scale. The completeness score for a specific abstract for each judge was calculated as:…”

“…In addition, the title and keywords were removed from each abstract. The questions were derived from those used in the previous studies (Budgen et al 2011(Budgen et al , 2008, with modifications to address the restriction of using only those papers that had an empirical element. For the purpose of data collection, each student judge was required to first complete a consent form, then a short form asking for demographic information, and would then receive the four data collection forms in the defined order, 11 and one at a time.…”

Review of James Hartley’s research on structured abstracts

“…In addition, he applied the structured format to the teaching of how to write laboratory reports with the first-year psychology students at Keele University [30]. These particular studies were taken up and extended by colleagues in the Department of Computing at Keele to the writing of abstracts in both firstyear and final-year reports [31,32]. Here are the titles of Hartley's five research papers in this regard: 'Is It Appropriate to Use Structured Abstracts in Social Science Journals?…”

“…Furthermore, teaching students how to write structured abstracts is conducive to improved abstracts [30][31][32].…”