Implementation of performance indicators for automatic assessment

Marcano, Laura; Komulainen, Tiina M.; Haugen, Finn

doi:10.1016/b978-0-444-63965-3.50497-9

Cited by 5 publications

(17 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our previous studies, simulator training data relative to time (in seconds) and precision (in pixels) of image-guided pick-and-place task performance were recorded from different study populations, including absolute beginners, novice surgeons without specific experience in image-guided simulator training, and expert surgeons with variable experience in image-guided surgery [2,3,5]. These previous studies were aimed at investigating the effects of different camera views, monitor positions and levels of expertise on simulator skill statistics.…”

Section: Resultsmentioning

confidence: 99%

“…The way feedback is delivered to the operator, and the amount of feedback he/she has to process and integrate, represents an important challenge in the development of automatic systems for simulator training assistance [2,3]. If relevant information is not delivered effectively, either by drowning essential feedback parameters in a large amount of unnecessary ones, or by not providing truly useful feedback that analysis, a principle of automatic expert benchmarking is brought forward to conceptualize key properties of an automatic procedure that (1) knows the expert surgeon's statistics (in-built benchmarks) relative to task execution time and task precision, (2) detects, and if necessary (3) controls for individual speed-precision strategies by comparing a trainee's performance statistics to the expert's benchmarks and, finally, (3) is able to provide appropriate user feedback when necessary. It is argued that such a system will enable any surgical trainee, without the intervention of a tutor and at the earliest stages of training, to attain the highest level of task precision he/she is capable of on the simulator.…”

Section: Introductionmentioning

confidence: 99%

“…How expert benchmark statistics may be exploited for automatic performance control is explained.will help the trainee gain proficiency at the task at hand, then operators will not be able to understand the message given by the system, and the latter will have failed its purpose altogether. Guidelines for user interface design and feedback procedures in simulator training contexts do not yet exist, but they can and should be worked out and tested [2,3].Relevant performance metrics [4][5][6][7][8][9][10] are essential to the development of surgical simulator systems for optimal independent training. The presentation of such metrics to the user, in a way that boosts independent learning by producing a measurable skill improvement, is the most important aspect of an effective training system [11].…”

mentioning

confidence: 99%

“…will help the trainee gain proficiency at the task at hand, then operators will not be able to understand the message given by the system, and the latter will have failed its purpose altogether. Guidelines for user interface design and feedback procedures in simulator training contexts do not yet exist, but they can and should be worked out and tested [2,3].…”

mentioning

confidence: 99%

See 3 more Smart Citations

Towards Expert-Based Speed–Precision Control in Early Simulator Training for Novice Surgeons

Dresp-Langley

2018

Information

View full text Add to dashboard Cite

Simulator training for image-guided surgical interventions would benefit from intelligent systems that detect the evolution of task performance, and take control of individual speed-precision strategies by providing effective automatic performance feedback. At the earliest training stages, novices frequently focus on getting faster at the task. This may, as shown here, compromise the evolution of their precision scores, sometimes irreparably, if it is not controlled for as early as possible. Artificial intelligence could help make sure that a trainee reaches her/his optimal individual speed-accuracy trade-off by monitoring individual performance criteria, detecting critical trends at any given moment in time, and alerting the trainee as early as necessary when to slow down and focus on precision, or when to focus on getting faster. It is suggested that, for effective benchmarking, individual training statistics of novices are compared with the statistics of an expert surgeon. The speed-accuracy functions of novices trained in a large number of experimental sessions reveal differences in individual speed-precision strategies, and clarify why such strategies should be automatically detected and controlled for before further training on specific surgical task models, or clinical models, may be envisaged. How expert benchmark statistics may be exploited for automatic performance control is explained.will help the trainee gain proficiency at the task at hand, then operators will not be able to understand the message given by the system, and the latter will have failed its purpose altogether. Guidelines for user interface design and feedback procedures in simulator training contexts do not yet exist, but they can and should be worked out and tested [2,3].Relevant performance metrics [4][5][6][7][8][9][10] are essential to the development of surgical simulator systems for optimal independent training. The presentation of such metrics to the user, in a way that boosts independent learning by producing a measurable skill improvement, is the most important aspect of an effective training system [11]. Metric-based simulation ensures that training sessions are more than just simulated clinical procedures and gets rid of subjectivity in evaluating skill evolution; there is no ambiguity about the progress of training. Benchmarking individual levels of proficiency on the performance levels of experts in a validated, metric-based simulation system has well-established intrinsic face validity [11], and appears a better approach than benchmarking on performance concepts based on expert consensus, for example. Building expert performance in terms of benchmark metrics into simulator training programs provides a sound basis for automatic skill assessment. Benchmarking ensures that the "pass" level is defined by realistic criteria, set directly by the proficiency levels of individuals who are highly experienced at performing the clinical procedures that simulator training is aimed at preparing novices for [11][12][13][14][15].Whether ...

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Towards Expert-Based Speed–Precision Control in Early Simulator Training for Novice Surgeons

Dresp-Langley

2018

Information

View full text Add to dashboard Cite

show abstract

“…In our latest research (Marcano et al, 2017a) an automatic assessment tool was tested, and even though the trainees found it helpful, they still considered that the feedback and direction given by the instructor was necessary. Hitherto, this indicated that the relevance of the instructor lies in the feedback and Proceedings of the 58th SIMS September 25th -27th, Reykjavik, Iceland guidance s/he gives to the trainees.…”

Section: Introductionmentioning

confidence: 99%

Towards Effective Automatic Feedback for Simulator Training

Marcano

Yazidi

Ferati

et al. 2017

Linköping Electronic Conference Proceedings

Self Cite

View full text Add to dashboard Cite

The presence of an expert instructor during simulator training is of great importance for the trainees, if not indispensable. The instructor's role is to give feedback and guide the trainees to help them make the right decisions on time as effectively as possible. The instructor starts or pauses the training scenarios when needed and facilitates reflection during and after the scenarios. However, the fact that simulator-training sessions are very dependent on the participation of a guiding instructor can be a drawback, since there are not always sufficient expert instructors to fulfil the training demands. In this work, an intelligent tutoring system (ITS) is proposed as an automatic feedback solution for simulator training. It is based on effective assessment of the system conditions using a clustering based anomaly detection technique as a core component. Furthermore, we provide insights into the design of a proper interface for our ITS. The article presents the methodology for developing such a system which consists of three stages: data collection, data analysis and delivering feedback.

show abstract