Performance Indicators for the Assessment of Industrial Operators

Manca, Davide; Nazir, Salman; Lucernoni, Federico; Colombo, Simone

doi:10.1016/b978-0-444-59520-1.50143-3

Cited by 18 publications

(20 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The use of computers in assessment has been widely debated concerning its use for summative or formative assessment, but as discussed in Gekara, Bloor and Sampson (2011), many researchers, e.g., Conole and Warburton (2005), argue that computer assisted assessment can be used in both summative and formative assessment purposes. Benefits such as automated assessment, enabled by the use of computers, are also increasingly valued, e.g., Manca, Nazir, Colombo and Kluge (2014), and it seems that considering all the benefits, that the use of computers in assessment is beneficial.…”

Section: Using Computers In Assessmentmentioning

confidence: 99%

Consistency in the development of performance assessment methods in the maritime domain

Ernstsen

Nazir

2018

WMU J Marit Affairs

Self Cite

View full text Add to dashboard Cite

Maritime navigation involves the process of monitoring and controlling the movement of a ship from one position to another. When the ship is approaching or leaving a port, a local navigational expert, the pilot, is often provided for assisting the bridge team to safely and efficiently navigate the littoral waters. However, statistics concerning maritime accidents continues to associate human errors with accidents, which advocates a scrutiny of maritime education and training (MET) for pilotage operations. The pilotage operation is a unique phase of the voyage in which the bridge team has to rely on an external expert with whom they have no prior working experience with. Also, the pilot-bridge team often has to perform immediately, and the consequences of not performing can be calamitous. These aspects suggest a need for dedicated research on how to ensure that bridge teams have the necessary competencies for carrying out safe and efficient pilotage operations. Training and assessment are quintessential instruments for this mission. MET facilities increasingly rely on full-scale simulators for their benefits in training for complex operations. Meanwhile, the assessment of training performance is often based on subjective criteria, which has implications on the reliability and validity of the assessment. Consequently, lacking a reliable and valid assessment method could have repercussions on the educational quality, as well as subsequent developments to the MET facilities' training and education programs. The aim of this research is to understand and to advance performance assessment related to the use of full-scale maritime simulators in MET, with a main objective to reduce the subjective impact that can be present in the performance assessment of pilotage operations. This thesis presents a computer-assisted assessment tool based on a structural probabilistic network, in which the assessment criteria are weighted by using an analytical hierarchical process. This tool is designed for flexibility, so it can easily be Ernstsen: Reducing the subjective impact in performance assessment ___ VI employed across a multitude of pilotage scenarios, and to be less subjective so that it can present more reliable and valid performance information to its stakeholders. The results show that the presented assessment tool can be used for higher reliability in the assessment of technical performance, but that more research is required to assess teamwork performance reliably. The tool's content validity is considered adequate, and that studies over time are required to assess its criterion validity effectively. Regardless, the tool is deemed opportune for generating precise and accurate assessment of training performance and could serve as a steppingstone to objective assessment. Accurate and precise assessment of training performance are imperative for stakeholders that make executive decisions concerning the development of training programs, competency mappings of the workforce, as well as for the trainee to know his or her strengt...

show abstract

Section: Using Computers In Assessmentmentioning

confidence: 99%

Consistency in the development of performance assessment methods in the maritime domain

Ernstsen

Nazir

2018

WMU J Marit Affairs

Self Cite

View full text Add to dashboard Cite

show abstract

“…Commonly, expert instructors are experienced operators who have accumulated a great knowledge of the system through years. This dependency on expert instructors has raised concern in different industrial sectors given that many of the experienced operators have retired or will retire in the near future (Alamo and Ross, 2017, Dozortsev, 2013, Manca et al, 2012. Consequently, the availability of expert instructors is continuously decreasing while the demand for operator training continues increasing.…”

Section: Introductionmentioning

confidence: 99%

“…The topic of automated feedback for simulator training has been mentioned in several studies, some of them indicate that the use of instructional tools embedded in the simulator can improve the efficiency of training (Bell et al, 2008, Malakis andKontogiannis, 2012). Further, there are studies that present a method (Manca et al, 2014) or an already developed tool (Dozortsev, 2013) to give automated feedback.…”

Section: Introductionmentioning

confidence: 99%

Using the concept of data enclosing tunnel as an online feedback tool for simulator training

Marcano

Yazidi

Manca

et al. 2018

Linköping Electronic Conference Proceedings

Self Cite

View full text Add to dashboard Cite

Feedback is one of the key factors that makes industrial simulator training an effective learning tool. Usually, the trainees receive feedback from the instructor, who guides them through the simulation tasks. However, nowadays the availability of expert instructors is scarce while the training demand is increasing. Therefore, there is a need for new simulator training practices that could allow the trainees to be more independent and decrease the need to rely so often on the instructor. This could be achieved by offering the trainees online automated feedback. This article presents a method for developing a tool meeting those requirements is presented. Simulation data were gathered representing different execution paths of the same scenario. Data were then analyzed and clustered using different clustering techniques. Interestingly, "good" and "bad" performances are shown to be separable using different techniques for clustering multivariate time series. Furthermore, we introduce the concept of enclosing data tunnel representing the trajectory of well-behaving execution paths in a reduced dimensional space. By conditioning the mal-behaving performances to be less than 20 % of the total simulation time inside the tunnel, an accuracy on 68 % was obtained. Being more flexible and allowing the mal-behaving performances to be inside the tunnel for a maximum of 35 % of the total simulation time, the accuracy of the enclosing tunnel was increased to 84 %.

show abstract

“…A limited number of current objective assessment algorithms is employed on procedure-based work scenarios where the sequence and timings of actions can be pre-defined. Examples of assessment systems for procedure-based scenarios are the K-SIM ® Polaris -Ships Bridge simulator (Kongsberg Maritime, 2017) and systems for the automated assessment of operators' performance in a petrochemical process simulator (Manca et al, 2012;Manca & Brambila, 2011;Nazir et al, 2013;. However, both systems focus on procedure-based work tasks, and are not designed to handle open goal-oriented dynamic work tasks such as coastal navigation.…”

Section: Current State Of Automated Assessment Of Operator Performancementioning

confidence: 99%

“…those that are not competent). However, since a human subject matter expert (SME) is the assessment tool of the traineesʹ performance within the simulator, the assessment is affected by the biases that follows from such a subjective evaluation (Manca et al, 2012;. A bias in assessment involves the tendency to systematically shift the evaluation away from a consistent score (Kahneman, 2011;Allen & Yen, 1979).…”

Section: Introductionmentioning

confidence: 99%