The Peer Review Process of Teaching Materials

Abstract

When an instructor adopts teaching materials, he/she wants some measure of confidence that the resource is effective, correct, and robust. The measurement of the quality of a resource is an open problem. It is our thesis that the traditional evaluative approach to peer review is not appropriate to insure the quality of teaching materials, which are created with different contextual constraints. This Working Group report focuses on the evaluation process by detailing a variety of review models. The evolution of the development and review of teaching materials is outlined and the contexts for creation, assessment, and transfer are discussed. We present an empirical study of evaluation forms conducted at the ITiCSE 99 conference, and recommend at least one new review model for the validation of the quality of teaching resources.

Developing and Using a Guide to Assess Learning Resource Quality in Educational Digital Libraries

“…Following the recommendations above for establishing a rubric to assess content (Knox et al, 1999;Recker, Walker, Lawless, 2003;Sumner et al, 2003), we began with a literature review in the disciplines of computer science, library and information science, education (specifically online learning objects), and digital libraries. We selected only articles that included or referenced rubrics used to review online educational resources.…”

“…We selected only articles that included or referenced rubrics used to review online educational resources. The literature review yielded articles with descriptions of 12 rubrics (Liu, & Ward, 2007;Fitzgerald, Lovin, & Branch, 2003;Nesbit, Belfer, Leacock, 2003;Ohio Resource Center;Knox et al, 1999;Recker, Walker, & Lawless, 2003;Sumner, Khoo, Recker, & Marlino, 2003;Custard & Sumner, 2005;Kastens, DeFelice, Devaul, DiLeonardo, Ginger, Larsen, et al, 2005;McMartin, 2004;Muramatsu & Agogino, 1999). In keeping with our first goal (to synthesize the various dimensions of existing digital library rubrics in order to identify a standardized set of criteria that could potentially be used by any digital library with online educational resources), our initial plan was to identify review criteria at the most granular level and then do a frequency analysis to identify broad topics that applied across criteria (e.g., pedagogy or usability).…”

“…The resource is attractive (Liu & Ward, 2007) The resource is easy to navigate (Ohio Resource Center) The resource contains no advertising (Custard & Sumner, 2005;Sumner, Khoo, Recker, Marlino, 2003) The resource contains links that work (Liu & Ward, 2007) The resource is designed to accommodate disabled and mobile learners (Nesbit, Belfer, Leacock, 2003) Technical Reliability (Muramatsu & Agogino, 1999) The resource uses multimedia (Flash, sound clips, videos, or applets) that work (Custard & Sumner, 2005) The resource clearly identifies the supporting technical resources required (Knox et al, 1999) The resource contains help features (Nesbit, Belfer, Leacock, 2003 The resource is engaging (Liu & Ward, 2007) The reource is motivating (Fitzgerald, Lovoing, & Branch, 2003) The resource is interactive (Muramatsu & Agogino, 1999) The resources includes assessment(s) (Ohio Resource Center) The resource provides feedback (Nesbit, Belfer, & Leacock, 2003) The resource supports learners proceeding at their own pace (Muramatsu & Agogino, 1999) The resource supports introductory, reinforcing, or summative activities Administrative…”

“…Many rubric creators believe it is important to map out stakeholders and end-users, review previous work by other digital libraries (Knox et al, 1999), and learn how to gather data (Recker, Walker, Lawless, 2003;Sumner et al, 2003). Testing a rubric with users, for formative evaluation, further refines the rubric, yielding usability information (Fitzgerald, Lovin, & Branch, 2003) as well as real time feedback from target users (Recker, Walker, Lawless, 2003;Sumner et al, 2003), which ultimately assists with the process of implementing the rubric with a larger group of users.…”