Analysis of Communication, Team Situational Awareness, and Feedback in a Three-Person Intelligent Team Tutoring System

Abstract

This research assessed how the performance and team skills of three-person teams working with an Intelligent Team Tutoring System (ITTS) on a virtual military surveillance task were affected by feedback privacy, participant role, task experience, prior team experience, and teammate familiarity. Previous work in Intelligent Tutoring Systems (ITSs) has focused on outcomes for task skill training for individual learners. As research extends into intelligent tutoring for teams, both task skills and team skills are necessary for good team performance. This work includes a brief review of previous research on ITTSs, feedback, teams, and teamwork, including the recounting of two categories of a framework of teamwork performance, Communication and Cognition, which are relevant to the present study. This research examines the effects of an intelligent agent, as well as features of the team, its members, and the task being undertaken, on team communication (measured by relevant key-presses) and team situation awareness (as measured by scores on a quiz). Thirty-seven teams of three participants, each at their own computer running a multiplayer surveillance simulation, were given just-in-time private (individually delivered) or public (team-delivered) performance feedback during four 5-min trials. In the fourth trial, two of the three participants switched roles. Feedback type, teamwork experience, and teammate familiarity had no statistically significant effect on communication or team situation awareness. However, higher levels of role experience and task experience showed significant and medium-sized effects on communication performance. Results, based on performance data and structured interview responses, also revealed areas of improvement in future feedback design and a potential benchmark for feedback frequency in an action-oriented serious game-based ITTS. Among the conclusions are six design objectives for future ITTSs, establishing a foundation for future research on designing effective ITTSs that train interpersonal skills to nascent teams.

Enhancing shared mental models: A systematic review and meta-analysis of randomized controlled trials.

“…Two effects within two studies produced residuals exceeding three standard deviations (Andres, 2013; Burke, 2000); their removal reduced the overall effect of interventions on SMM by 0.04 ( g = .57, SE = .09, 95% CI [.38, .75]). Eight effects within five studies had a Cook’s distance over three times the mean (Andres, 2012, 2013; Burke, 2000; Crespin, 1996; Dalenberg et al, 2009; Ouverson, 2019); exclusion of these effects resulted in a reduction in the overall pooled effect by 0.12 ( g = .49, SE = .08, 95% CI [.32, .65]). Collectively, these exploratory analyses indicated that the influence of outliers or influential studies was small-to-moderate in nature, though the overall conclusion regarding the effectiveness of interventions remained the same.…”