Multidimensional Team Communication Modeling for Adaptive Team Training: A Hybrid Deep Learning and Graphical Modeling Framework

Min, Wookhee; Spain, Randall D.; Saville, Jason D.; Mott, Bradford W.; Brawner, Keith; Johnston, Joan; Lester, James C.

doi:10.1007/978-3-030-78292-4_24

Cited by 3 publications

(11 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These schemes enable researchers to classify individual utterances according to their intent. In recent years, there has been an expansion of research investigating NLP-driven approaches for dialogue act classi cation labeling (Min et al, 2021;Pande et al, 2023). When integrated into team training systems, automatically recognized dialogue acts can provide insights that support additional downstream tasks essential for adaptive team training.…”

Section: Team Communicationmentioning

confidence: 99%

“…In our previous research, we explored an approach that combines deep neural networks with probabilistic graphical models to recognize dialogue acts in team communication, using a hybrid NLP framework (Min et al, 2021). This framework combined ELMo word embeddings with CRF to facilitate dialogue act recognition and information ow classi cation (i.e., the directional ow of communication within the team) with team training transcripts.…”

Section: Introductionmentioning

confidence: 99%

“…For RQ2, we compare the accuracy of the highest-performing LLMs against our framework's baseline models. Speci cally, we evaluate LLMs' performance compared to models investigated in prior work, such as CRF-based probabilistic graphical models (Min et al, 2021) and T5-based models (Pande et al, 2023) for the dialogue act recognition task. Finally, for RQ3, we present an error analysis of our team communication models.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Applying Large Language Models to Enhance Dialogue and Communication Analysis for Adaptive Team Training

Spain,

Min,

Kumaran

et al. 2024

Preprint

View full text Add to dashboard Cite

Adaptive training environments that can analyze team communication content and provide remediation to facilitate team coordination offer great potential for enhancing adaptive training systems for teams. Developing computational models that can perform robust team communication analytics based on small datasets is challenging. Large language models (LLMs) offer significant potential to address these challenges and enhance dialogue act classification performance using zero-shot and few-shot learning. This paper evaluates LLMs against previous state-of-the-art methods, with an emphasis on dialogue act recognition performance and error analysis for identifying frequently misclassified instances. Results from a small team communication dataset indicate that zero-shot LLMs, particularly those utilizing GPT-4 and refined through robust prompt engineering, achieve significant classification performance improvements in dialogue act recognition compared to previous state-of-the-art transformer-based models fine-tuned with team communication data. Error analysis shows that the prompt refinements, especially those aimed at clarifying confusion between dialogue acts, result in superior recall rates for challenging dialogue act labels by effectively handling complex dialogue scenarios and ambiguities within communication data. Our transformer-based framework demonstrates its effectiveness in achieving high accuracy rates in dialogue act recognition with minimal training data, underscoring its potential to enhance team training programs by providing adaptive feedback. This approach paves the way for developing AI-enabled training systems that can adapt to the dynamic communication styles of different teams.

show abstract

Section: Team Communicationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Applying Large Language Models to Enhance Dialogue and Communication Analysis for Adaptive Team Training

Spain,

Min,

Kumaran

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…A key capability of dialogue modeling is recognizing when speakers’ utterances contain certain dialogue acts, which represent the meaning of an utterance in terms of its underlying intention such as statement, question, and agreement (Stolcke et al, 2000). Recent work has investigated various NLP and machine learning techniques for dialogue act recognition, including probabilistic graphical models leveraging contextual language models (Min et al, 2021) and deep bi-directional recurrent neural network-based multi-task models (Firdaus et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

“…Advances in natural language processing (NLP) methods are affording researchers new opportunities to expedite the coding process using models that can analyze team discourse and automatically assign informative labels to each utterance (Foltz et al, 2006, Min et al, 2021Sandoval et al, 2022, Spain et al, 2020. Automated coding of teams' spoken communication can provide insight about the content of team communication (Foltz et al, 2006) and team communication patterns during high and low periods of workload (Gontar et al, 2017), and support the prediction of team performance (Foltz et al, 2006;Sandoval et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

Evaluating the Classification Performance of Natural Language Processing-Driven Team Communication Analysis Models

Paul,

Spain,

Min

et al. 2023

Proceedings of the Human Factors and Ergonomics Society Annual

Self Cite

View full text Add to dashboard Cite

In this paper, we evaluate the performance of transformer-based natural language processing models in analyzing team communication captured during a live training event. We use a multi-class confusion matrix technique to identify patterns in the performance of two models which recognize dialogue acts and classify how information flows between team members. The dialogue act recognition model was particularly accurate on utterances related to acknowledgement, commanding, and providing information. For information flow, the model showed good performance on classifying utterances labeled as commands from the bottom and middle of the chain of command, although the error analysis revealed a high number of misclassifications related to providing information down and up the chain of command. Results of the multi-class confusion matrix technique provide insight into performance at a more granular level that may lead to model improvements and a better understanding of how the models can be applied to new datasets.

show abstract

Multimodal Predictive Student Modeling with Multi-Task Transfer Learning

Emerson

Min

Rowe

et al. 2023

LAK23: 13th International Learning Analytics and Knowledge Conference

View full text Add to dashboard Cite

Multidimensional Team Communication Modeling for Adaptive Team Training: A Hybrid Deep Learning and Graphical Modeling Framework

Cited by 3 publications

References 36 publications

Applying Large Language Models to Enhance Dialogue and Communication Analysis for Adaptive Team Training

Applying Large Language Models to Enhance Dialogue and Communication Analysis for Adaptive Team Training

Evaluating the Classification Performance of Natural Language Processing-Driven Team Communication Analysis Models

Multimodal Predictive Student Modeling with Multi-Task Transfer Learning

Contact Info

Product

Resources

About