Analyzing Components of Work Samples to Evaluate Performance

Abstract

A method is described in which work sample data were analyzed to reveal how aircraft maintenance technicians differed in troubleshooting performance after a training intervention. During the training phase of the study, approximately half the technicians received troubleshooting instruction from an intelligent tutoring system, and the remaining technicians received informal on-the-job training. Data from work samples, in which technicians verbally isolated simulated equipment failures, were collected from both groups before and after the training. The work sample data were then examined by subject matter experts who scored performance for overall proficiency and noted specific violations of good troubleshooting practice. These violations provided the basis for the components of performance examined in the work sample data. The results of this componential analysis allow more specific conclusions to be drawn regarding the effectiveness of the training intervention by revealing how the tutoring system improved technicians' troubleshooting (e.g., tutored technicians were more likely than untutored technicians to test components thoroughly to ensure they were faulty before replacing them).

Assembling and Analyzing the Building Blocks of Problem‐Based Learning Environments

“…This approach may work for helping novices solve simple troubleshooting problems; however, it is inadequate for training competent, professional troubleshooters. Effective troubleshooting requires system knowledge (conceptual knowledge of how the system works), procedural knowledge (how to perform problem-solving procedures), and strategic knowledge (strategies such as search-and-replace, serial elimination, and space splitting) (Pokorny, Hall, Gallaway, & Dibble, 1996). These different kinds of knowledge comprise the troubleshooter's mental model of the process.…”

Learning to Troubleshoot: A New Theory-Based Design Architecture

“…Its instructional model was based on dynamic assessment of the learner while troubleshooting problems (Lajoie and Lesgold, 1992a). After 20-30 hr of troubleshooting problems, the overall proficiency scores of learners were no better than for trainees receiving on-the-job training (Pokorny et al, 1996). In another study, trainees who used SHERLOCK for 20 hr over 2 weeks performed as well on troubleshooting tasks as experienced technicians (Gott et al, 1993).…”

Instructional design for learning to troubleshoot

“…Because effective troubleshooting requires more than a series of decisions. Effective troubleshooting requires system knowledge, or conceptual knowledge of how the system works, procedural knowledge, or how to perform problem-solving procedures and test activities, and strategic knowledge, that is, strategies such as search and replace, serial elimination, and space splitting (Pokorny, Hall, Gallaway, & Dibble, 1996). These different kinds of knowledge comprise the troubleshooter's mental model of the process.The troubleshooter's mental model consists of conceptual, functional, and declarative knowledge, including knowledge of system components and interactions, flow control, fault states (including fault characteristics, symptoms, contextual information, and probabilities of occurrence), and fault-testing procedures.…”

“…Because effective troubleshooting requires more than a series of decisions. Effective troubleshooting requires system knowledge, or conceptual knowledge of how the system works, procedural knowledge, or how to perform problem-solving procedures and test activities, and strategic knowledge, that is, strategies such as search and replace, serial elimination, and space splitting (Pokorny, Hall, Gallaway, & Dibble, 1996). These different kinds of knowledge comprise the troubleshooter's mental model of the process.…”