Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-assisted Radical Prostatectomy

Ghani, Khurshid R.; Miller, David C.; Linsell, Susan; Brachulis, Andrew; Lane, Brian R.; Sarle, Richard; Dalela, Deepansh; Menon, Mani; Comstock, Bryan A.; Lendvay, Thomas S.; Montie, James E.; Peabody, James O.

doi:10.1016/j.eururo.2015.11.028

Cited by 108 publications

(76 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, there may be unmeasured patient factors that we have been unable to assess. Finally, while crowd-sourced GEARS assessment is a valid method of assessing global robotic surgical technical skill in urology, 8,25 in our study, crowd-sourced reviewers could not differentiate between UIS and control videos. While platforms like C-SATS have a role in generating high-volume assessment and feedback, crowd-sourcing has yet to demonstrate the ability to discern between clinically relevant patient outcomes, unlike peer-review.…”

Section: Discussionmentioning

confidence: 88%

“…These assessments have been demonstrated to correlate with the assessments of content experts. 8 Crowd workers are paid a nominal fee for each video clip they rate. The raters enlisted by C-SATS are trained in use of the Global Evaluative Assessment of Robotic Skill (GEARS) assessment tool, which consists of Likert scale scoring across six domains of robotic skill.…”

Section: Crowd-sourcingmentioning

confidence: 99%

“…[3][4][5] Recently, evaluation of video footage by laypeople trained in the use of validated assessment metrics, termed "crowdsourcing," has been shown to provide efficient and reliable feedback that correlates well with expert ratings. 6,7 This methodology has been translated to differentiating surgeon skill 8 in robotic prostatectomy. In order to improve care provided to patients, one aspect of surgical quality improvement seeks to identify both intraoperative steps that contribute to clinically relevant outcomes.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Feasibility of expert and crowd-sourced review of intraoperative video for quality improvement of intracorporeal urinary diversion during robotic radical cystectomy

Goldenberg

Nabhani

Wallis

et al. 2017

CUAJ

View full text Add to dashboard Cite

Introduction: Development of uretero-ileal stricture (UIS) after robotic-assisted radical cystectomy (RARC) may be dependent on surgical technique. Video review of intraoperative technique is an emerging paradigm for surgical quality improvement. We examined whether surgeon-perceived risk of UIS or crowd-sourced assessment of robotic skill are associated with the development of UIS. Methods: We conducted a case-control study comparing the operative technique of uretero-ileal anastomoses resulting in clinically significant UIS with the contralateral anastomosis for the same patient. De-identified videos were analyzed by 1) five high-volume surgeons; and 2) crowd workers (Crowd-Sourced Assessment of Technical Skill, C-SATS) to determine Global Evaluative Assessment of Robotic Skill (GEARS) score. Mantel-Haenszel common odds ratio (OR) estimates were calculated to assess the association between surgeon performance and the development of UIS. Logistic regression models were used to examine the association between GEARS scores and the development of UIS. Results: A total of 10 UIS videos were compared to eight control videos by five surgeons and 2142 crowd workers. Expert surgeons systematically evaluated intraoperative footage, however, no association between the expert mode response and UIS (OR 0.42; 95% confidence interval [CI] 0.05-3.45; p=0.91) was identified. Crowd-sourced assessment was not predictive of UIS (p=0.62). Conclusions: We used video review to systematically analyze procedure-specific content and technique. The inability of surgeons to predict UIS may reflect the questionnaire, uncontrolled patient factors, or a lack of power. Crowd-sourced GEARS score was unsuccessful in predicting UIS after RARC.

show abstract

Section: Discussionmentioning

confidence: 88%

Section: Crowd-sourcingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Feasibility of expert and crowd-sourced review of intraoperative video for quality improvement of intracorporeal urinary diversion during robotic radical cystectomy

Goldenberg

Nabhani

Wallis

et al. 2017

CUAJ

View full text Add to dashboard Cite

show abstract

“…A recent study showed that medically trained reviewers were able to identify surgeons with higher complication rates by watching videos of their technique in the context of laparoscopic bariatric surgery [14]. Surprisingly, healthcare consumers (via crowdsourcing among the general population) were also able to identify surgeons with higher complication rates by watching videos of their operative technique in the context of robotic radical prostatectomy [15]. Based on this data, patients may be justified in choosing surgeons who post videos online, since they appear to be able discern good from bad surgeons by viewing examples of their best work.…”

Section: Discussionmentioning

confidence: 99%

Association of Social Media Presence with Online Physician Ratings and Surgical Volume Among California Urologists: Observational Study (Preprint)

Houman¹,

Weinberger²,

Caron³

et al. 2018

Preprint

View full text Add to dashboard Cite

BACKGROUND Urologists are increasingly using various forms of social media to promote their professional practice and attract patients. Currently, the association of social media on a urologists’ practice is unknown. OBJECTIVES We aimed to determine whether social media presence is associated with higher online physician ratings and surgical volume among California urologists. METHODS We sampled 195 California urologists who were rated on the ProPublica Surgeon Scorecard website. We obtained information on professional use of online social media (Facebook, Instagram, Twitter, blog, and YouTube) in 2014 and defined social media presence as a binary variable (yes/no) for use of an individual platform or any platform. We collected data on online physician ratings across websites (Yelp, Healthgrades, Vitals, RateMD, and UCompareHealthcare) and calculated the mean physician ratings across all websites as an average weighted by the number of reviews. We then collected data on surgical volume for radical prostatectomy from the ProPublica Surgeon Scorecard website. We used multivariable linear regression to determine the association of social media presence with physician ratings and surgical volume. RESULTS Among our sample of 195 urologists, 62 (32%) were active on some form of social media. Social media presence on any platform was associated with a slightly higher mean physician rating (β coefficient: .3; 95% CI 0.03-0.5; P=.05). However, only YouTube was associated with higher physician ratings (β coefficient: .3; 95% CI 0.2-0.5; P=.04). Social media presence on YouTube was strongly associated with increased radical prostatectomy volume (β coefficient: 7.4; 95% CI 0.3-14.5; P=.04). Social media presence on any platform was associated with increased radical prostatectomy volume (β coefficient: 7.1; 95% CI –0.7 to 14.2; P=.05). CONCLUSIONS Urologists’ use of social media, especially YouTube, is associated with a modest increase in physician ratings and prostatectomy volume. Although a majority of urologists are not currently active on social media, patients may be more inclined to endorse and choose subspecialist urologists who post videos of their surgical technique.

show abstract

“…The authors compared assessments by expert raters (robotic and open surgeons) and those from lay people from the Crowd-Sourced Assessment of Technical Skills (C-SATS) 1,2 group with regard to experienced surgeons' technical ability during a robotic-assisted radical cystectomy. The authors prepared short video segments (60 seconds) showing mobilization of the ureter, ureteral preparation for the anastomosis, and the ureteral-ileal anastomosis from nine (out of a potential 102) cases that resulted in clinically significant postoperative uretero-ileal strictures (UIS) (10 strictures in total).…”

mentioning

confidence: 99%

Video replay in surgery: Can we make the “right call” in predicting outcomes?

Matsumoto

2017

CUAJ

View full text Add to dashboard Cite

Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-assisted Radical Prostatectomy

Cited by 108 publications

References 10 publications

Feasibility of expert and crowd-sourced review of intraoperative video for quality improvement of intracorporeal urinary diversion during robotic radical cystectomy

Feasibility of expert and crowd-sourced review of intraoperative video for quality improvement of intracorporeal urinary diversion during robotic radical cystectomy

Association of Social Media Presence with Online Physician Ratings and Surgical Volume Among California Urologists: Observational Study (Preprint)

Video replay in surgery: Can we make the “right call” in predicting outcomes?

Contact Info

Product

Resources

About