Validation of the updated ArthroS simulator: face and construct validity of a passive haptic virtual reality simulator with novel performance metrics

Roberts, Philip; Guyver, P.M.; Baldwin, Martin; Akhtar, Kash; Alvand, Abtin; Price, Andrew; Rees, Jonathan

doi:10.1007/s00167-016-4114-1

Cited by 55 publications

(37 citation statements)

References 38 publications

(71 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The studies were broadly split into 3 categories: studies measuring the impact of a simulation training intervention (26 studies 6 - 31 ), studies assessing the construct validity of a simulator designed for training surgeons (42 studies 32 - 73 ), and studies validating an assessment tool (37 studies 74 - 110 ) (see Appendix Tables 1 and 2, column 1). Of the included studies, 60% assessed arthroscopic skill involving the knee (34 studies) 6 , 8 , 9 , 13 , 15 , 17 , 19 , 31 - 33 , 36 , 38 , 39 , 41 , 42 , 47 , 48 , 54 - 56 , 63 , 74 , 75 , 77 - 83 , 86 , 87 , 89 , 91 , the shoulder (25 studies) 7 , 12 , 15 , 16 , 18 , 20 , 37 , 40 , 44 , 45 , 49 , 51 , 53 , 54 , 56 , 63 , 76 , 77 , 81 , 82 , 88 , 90 - 92 , 110 , the hip (3 studies) 43 , 50 , 85 , the ankle (1 study) 14 , and basic general arthroscopic skills (6 studies)…”

Section: Resultsmentioning

confidence: 99%

“…Hand-motion analysis can generate a sophisticated data profile that can detect subtle improvement in surgical performance, and may be able to measure the attainment of mastery. Other motion parameters, such as gaze tracking 6 , triangulation time 74 , instrument path length 12 , 15 , 40 , 48 , 49 , 51 , 55 , 56 , 63 , 110 , and collisions 38 , 55 , have demonstrated construct validity and feasibility in the simulated environment but are unlikely to be useful in the live operative theater, as most of these measurements are derived from the simulator itself.…”

Section: Discussionmentioning

confidence: 99%

“…This is potentially problematic, as extrinsic patient and staff factors beyond surgeons’ immediate control could influence procedure time, and it gives no indication of quality of performance; procedure time may be measured as fast because the surgeon was a masterfully efficient operator, but alternatively they may have rushed the procedure and been careless. The evidence for construct and concurrent validity for procedure time is mixed, with many studies showing it can discriminate between experience levels 6 , 11 , 12 , 30 , 31 , 33 , 34 , 40 , 43 - 45 , 47 , 48 , 50 , 51 , 53 - 56 , 61 , 63 , 64 , 67 , 78 , 86 , 110 , and performs well against other types of assessment 6 , 18 , 47 , 86 , with others showing it cannot 18 , 20 , 23 , 57 , 60 , 62 , 72 , 73 , 99 , 102 . Both final-product analysis and procedure time are therefore unlikely to be useful in isolation, but rather could be used as adjunctive measures of technical proficiency.…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training

et al. 2020

View full text Add to dashboard Cite

Background: Robust assessment of skills acquisition and surgical performance during training is vital to ensuring operative competence among orthopaedic surgeons. A move to competency-based surgical training requires the use of tools that can assess surgical skills objectively and systematically. The aim of this systematic review was to describe the evidence for the utility of assessment tools used in evaluating operative performance in trauma and orthopaedic surgical training. Methods: We performed a comprehensive literature search of MEDLINE, Embase, and Google Scholar databases to June 2019. From eligible studies we abstracted data on study aim, assessment format (live theater or simulated setting), skills assessed, and tools or metrics used to assess surgical performance. The strengths, limitations, and psychometric properties of the assessment tools are reported on the basis of previously defined utility criteria. Results: One hundred and five studies published between 1990 and 2019 were included. Forty-two studies involved open orthopaedic surgical procedures, and 63 involved arthroscopy. The majority (85%) were used in the simulated environment. There was wide variation in the type of assessment tools in used, the strengths and weaknesses of which are assessor and setting-dependent. Conclusions: Current technical skills-assessment tools in trauma and orthopaedic surgery are largely procedure-specific and limited to research use in the simulated environment. An objective technical skills-assessment tool that is suitable for use in the live operative theater requires development and validation, to ensure proper competency-based assessment of surgical performance and readiness for unsupervised clinical practice. Clinical Relevance: Trainers and trainees can gain further insight into the technical skills assessment tools that they use in practice through the utility evidence provided.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Collecting participant feedback regarding face validity is a commonly used approach in the field of surgical simulation. In this context, the opinions of expert surgeons are often sought about how the simulation looks and feels and is an important part of simulation validation (Sankaranarayanan et al, 2016;Roberts et al, 2017). A similar approach is also used in the development and testing of aircraft simulations with expert pilots (Perfect et al, 2014).…”

Section: Face Validitymentioning

confidence: 99%

A Framework for the Testing and Validation of Simulated Environments in Experimentation and Training

et al. 2020

View full text Add to dashboard Cite

New computer technologies, like virtual reality (VR), have created opportunities to study human behavior and train skills in novel ways. VR holds significant promise for maximizing the efficiency and effectiveness of skill learning in a variety of settings (e.g., sport, medicine, safety-critical industries) through immersive learning and augmentation of existing training methods. In many cases the adoption of VR for training has, however, preceded rigorous testing and validation of the simulation tool. In order for VR to be implemented successfully for both training and psychological experimentation it is necessary to first establish whether the simulation captures fundamental features of the real task and environment, and elicits realistic behaviors. Unfortunately evaluation of VR environments too often confuses presentation and function, and relies on superficial visual features that are not the key determinants of successful training outcomes. Therefore evidence-based methods of establishing the fidelity and validity of VR environments are required. To this end, we outline a taxonomy of the subtypes of fidelity and validity, and propose a variety of practical methods for testing and validating VR training simulations. Ultimately, a successful VR environment is one that enables transfer of learning to the real-world. We propose that key elements of psychological, affective and ergonomic fidelity, are the real determinants of successful transfer. By adopting an evidence-based approach to VR simulation design and testing it is possible to develop valid environments that allow the potential of VR training to be maximized.

show abstract

“…Surgical simulators are now being used for training and assessment purposes in various surgical fields including arthroscopy. The advantages of using these simulators in training programs consist of unrestricted practice time, lower cost compared to cadaver models, the opportunity for independent learning, and decreasing the risk to patients in the operating room [ 1 ]. The suitability of these simulators for training and assessment purposes not only depends on a realistic design and efficient use of the simulator, but also on the assessment method that is incorporated into the simulator to evaluate the proficiency levels of users.…”

Section: Introductionmentioning

confidence: 99%

Energy-Based Metrics for Arthroscopic Skills Assessment

Poursartip

LeBel

McCracken³

et al. 2017

Sensors

View full text Add to dashboard Cite

Abstract:Minimally invasive skills assessment methods are essential in developing efficient surgical simulators and implementing consistent skills evaluation. Although numerous methods have been investigated in the literature, there is still a need to further improve the accuracy of surgical skills assessment. Energy expenditure can be an indication of motor skills proficiency. The goals of this study are to develop objective metrics based on energy expenditure, normalize these metrics, and investigate classifying trainees using these metrics. To this end, different forms of energy consisting of mechanical energy and work were considered and their values were divided by the related value of an ideal performance to develop normalized metrics. These metrics were used as inputs for various machine learning algorithms including support vector machines (SVM) and neural networks (NNs) for classification. The accuracy of the combination of the normalized energy-based metrics with these classifiers was evaluated through a leave-one-subject-out cross-validation. The proposed method was validated using 26 subjects at two experience levels (novices and experts) in three arthroscopic tasks. The results showed that there are statistically significant differences between novices and experts for almost all of the normalized energy-based metrics. The accuracy of classification using SVM and NN methods was between 70% and 95% for the various tasks. The results show that the normalized energy-based metrics and their combination with SVM and NN classifiers are capable of providing accurate classification of trainees. The assessment method proposed in this study can enhance surgical training by providing appropriate feedback to trainees about their level of expertise and can be used in the evaluation of proficiency.

show abstract

Validation of the updated ArthroS simulator: face and construct validity of a passive haptic virtual reality simulator with novel performance metrics

Cited by 55 publications

References 38 publications

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training

A Framework for the Testing and Validation of Simulated Environments in Experimentation and Training

Energy-Based Metrics for Arthroscopic Skills Assessment

Contact Info

Product

Resources

About