Evaluation of an intelligent team tutoring system for a collaborative two-person problem: Surveillance

Abstract

This paper describes the development and evaluation of an Intelligent Team Tutoring System (ITTS) for pairs of learners working collaboratively to monitor an area. In the Surveillance Team Tutor (STT), learners performed a surveillance task in a virtual environment, communicating to track hostile moving soldiers. This collaborative problem solving task required significant communication to achieve the common goal of perfect surveillance. In a pilot evaluation, 16 twoperson teams performed the task within one of three feedback conditions (Individual, Team, or None) across four trials each. The STT used a unique approach to filtering feedback so that teams in both individual and team conditions received a similar amount of feedback. In one performance measure, Team condition participants made fewer errors in one task than those in other conditions, though at a potential cost of mental workload. Feedback condition also significantly affected participants' subjective rating of both their own performance and their teammate's. This ITTS is one of the first automated team tutoring systems that provided real-time feedback during task execution. Recommendations are offered for the design of the optimal team task for future ITTSs that offer tutoring for small teams performing collaborative problem solving.

Analysis of Communication, Team Situational Awareness, and Feedback in a Three-Person Intelligent Team Tutoring System

“…Feedback can enable each team member to identify the goals of their respective roles within the team, in essence helping them to create a shared mental model of an "expert team." While performance on task work has been shown to be most affected by privately given feedback, teamwork has been affected by publicly given feedback (Geister et al, 2006;Mumm and Mutlu, 2011), although feedback-effectiveness results in other studies have been mixed (Peñarroja et al, 2015;Ostrander et al, 2019).…”

“…Starting with systems that kept human trainers better apprised of the team's whole performance (Zachary et al, 1999), work investigating ITTSs has branched from simple algorithms to social tutors giving team-and individual-level assessment (Kumar et al, 2010;Walton et al, 2015b;Ostrander et al, 2019). Within the environments of ITTSs, behavioral markers are used to identify team metrics (Salas et al, 2007b;Sottilare et al, 2018).…”

“…Within the environments of ITTSs, behavioral markers are used to identify team metrics (Salas et al, 2007b;Sottilare et al, 2018). From these metrics, ITTSs use feedback to present information about team and individual performance on task and team skills (Sinatra et al, 2018;Walton et al, 2018;Ostrander et al, 2019). Feedback from the tutor, in turn, influences teammates' actions, interactions, and team SA.…”

“…Since the late 1990s, ITTSs have been developed to facilitate training on a variety of team tasks, including Naval air defense training ( Zachary et al, 1999 ), mechanical engineering ( Kumar et al, 2010 ), group coordination ( Walton et al, 2015b ), collaborative problem solving ( Fiore et al, 2017 ), and team-coordinated surveillance ( Ostrander et al, 2019 ). One of the first ITTS-like systems, the Advanced Embedded Training System (AETS), facilitated Naval air defense training ( Zachary et al, 1999 ).…”

“…Unlike Avis and the PISA systems, the surveillance team tutor (STT; Ostrander et al, 2019 ), similarly to AETS ( Zachary et al, 1999 ) and TMET ( Walton et al, 2015b ), contained a fast-paced, high-cognitive load psychomotor-performance task that required steady, focused attention and did not typically allow for conversational dialog. The STT was designed to train two-person teams on a military task using just-in-time feedback tailored to individual player actions.…”