Multimodal Models for Contextual Affect Assessment in Real-Time

Vice, Jordan; Khan, Masood Mehmood; Yanushkevich, Svetlana

doi:10.1109/cogmi48466.2019.00020

Cited by 10 publications

(14 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first AI system assessed for this work was designed to have the proposed AXAI framework built into it. It is a multimodal definite AI system (Vice, Khan & Yanushkevich, 2019) implemented to be a portable affective state assessment module (ASAM). The current version of the ASAM is an enhanced and improved form of our previously published system (Vice, Khan & Yanushkevich, 2019).…”

Section: Assessment Of the Affective State Assessment Modulementioning

confidence: 99%

“…It is a multimodal definite AI system (Vice, Khan & Yanushkevich, 2019) implemented to be a portable affective state assessment module (ASAM). The current version of the ASAM is an enhanced and improved form of our previously published system (Vice, Khan & Yanushkevich, 2019). The ASAM was developed for real time multimodal analysis of facial expressions and speech for assessing affective states.…”

Section: Assessment Of the Affective State Assessment Modulementioning

confidence: 99%

See 1 more Smart Citation

Toward Accountable and Explainable Artificial Intelligence Part one: Theory and Examples

Khan¹,

Vice²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

<div>After reviewing the current state of explainable Artificial Intelligence (XAI) capabilities in artificial Intelligence (AI) systems developed for critical domains like criminology, engineering, governance, health, law and psychology, this paper proposes a domain-independent Accountable explainable Artificial Intelligence (AXAI) capability framework. The proposed AXAI framework extends the XAI capability to let AI systems share their decisions and adequately explain the underlying reasoning processes. The idea is to help AI system developers overcome algorithmic biases and system limitations through incorporation of domain independent AXAI capabilities. Moreover, existing XAI methods would neither separate nor quantify measures of comprehensibility, accuracy and accountability so incorporating and assessing XAI capabilities remains difficult. Assessment of the AXAI capabilities of two AI systems in this paper demonstrates that the proposed AXAI framework facilitates separation and measurement of comprehensibility, predictive accuracy and accountability. The AXAI framework allows for the delineation of AI systems in a three-dimensional AXAI space. It measures comprehensibility as the readiness of a human to apply the acquired knowledge. The system accuracy is measured as the ratio of the test and training data, training data size and the observed number of false-positive inferences. Finally, the AXAI framework measures accountability in terms of the inspect ability of the input cues, the processed data and the output information, for addressing any legal and ethical issues.</div>

show abstract

Section: Assessment Of the Affective State Assessment Modulementioning

confidence: 99%

Section: Assessment Of the Affective State Assessment Modulementioning

confidence: 99%

Toward Accountable and Explainable Artificial Intelligence Part one: Theory and Examples

Khan¹,

Vice²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The first assessed ML system was designed to have the proposed AXAI framework built into it. The Affective State Assessment Module (ASAM) is a multimodal definite system [53] implemented as a portable affective state assessment Output information is descriptive and easy to understand/interpret sub-system for integration into robotic systems. The tested version of the ASAM is an improved system of our previously developed and published system [53].…”

Section: A An Affective State Assessment Module (Asam)mentioning

confidence: 99%

Toward Accountable and Explainable Artificial Intelligence Part One: Theory and Examples

Khan

Vice

2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

Like other Artificial Intelligence (AI) systems, Machine Learning (ML) applications cannot explain decisions, are marred with training-caused biases, and suffer from algorithmic limitations. Their eXplainable Artificial Intelligence (XAI) capabilities are typically measured in a two-dimensional space of explainability and accuracy ignoring the accountability aspects. During system evaluations, measures of comprehensibility, predictive accuracy and accountability remain inseparable. We propose an Accountable eXplainable Artificial Intelligence (AXAI) capability framework for facilitating separation and measurement of predictive accuracy, comprehensibility and accountability. The proposed framework, in its current form, allows assessing embedded levels of AXAI for delineating ML systems in a three-dimensional space. The AXAI framework quantifies comprehensibility in terms of the readiness of users to apply the acquired knowledge and assesses predictive accuracy in terms of the ratio of test and training data, training data size and the number of false-positive inferences. For establishing a chain of responsibility, accountability is measured in terms of the inspectability of input cues, data being processed and the output information. We demonstrate applying the framework for assessing the AXAI capabilities of three ML systems. The reported work provides bases for building AXAI capability frameworks for other genres of AI systems.

show abstract

“…To imitate the human model of ASA, algorithm-based learning and classification approaches rely on one or more of visual, vocal, psychophysiological and neural cues [13,[17][18][19]. Emerging sophisticated ASA systems use different combinations of affective state models, learning techniques and classification methods [20][21][22] as their capabilities and accuracies progressively improve [23][24][25][26]. Modern ASA capabilities include the dynamic assessment of affect-arousal and, multimodal and contextual assessment of affective states [18,[26][27][28].…”

Section: Introductionmentioning

confidence: 99%

“…Emerging sophisticated ASA systems use different combinations of affective state models, learning techniques and classification methods [20][21][22] as their capabilities and accuracies progressively improve [23][24][25][26]. Modern ASA capabilities include the dynamic assessment of affect-arousal and, multimodal and contextual assessment of affective states [18,[26][27][28]. Since ASA systems rely on algorithm-based analysis, they inherit algorithmic biases while making inferences.…”

Section: Introductionmentioning

confidence: 99%

Toward Accountable and Explainable Artificial Intelligence Part Two: The Framework Implementation

Khan¹,

Vice²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

<div>This paper builds upon the theoretical foundations of the Accountable explainable Artificial Intelligence (AXAI) capability framework presented in part one of this paper. This part demonstrates the incorporation of the AXAI capability in the real time Affective State Assessment Module (ASAM) of a robotic system. The paper argues that adhering to the extreme Programming (XP) practices would help in understanding user behavior while systematically incorporating the AXAI capability in AI systems. Issues pertaining to identification of ethical, technical, functional, and domain-specific system requirements were resolved using an appropriate software design process, thus establishing confidence in the system. The presented ASAM synthesizes discrete and continuous models of affective state expressions and classifies them in real-time. Input, processed and output data are continuously shared with users via a graphical user interface (GUI) for providing reasons behind decisions. The GUI also disseminates information about local reasoning, data handling and decision-making. We hope this work will initiate further investigations on selection of suitable software design practices for incorporating the AXAI capability in AI systems, enhancing AI system acceptability and utility and, establishing a chain of responsibility.</div>

show abstract

Multimodal Models for Contextual Affect Assessment in Real-Time

Cited by 10 publications

References 15 publications

Toward Accountable and Explainable Artificial Intelligence Part one: Theory and Examples

Toward Accountable and Explainable Artificial Intelligence Part one: Theory and Examples

Toward Accountable and Explainable Artificial Intelligence Part One: Theory and Examples

Toward Accountable and Explainable Artificial Intelligence Part Two: The Framework Implementation

Contact Info

Product

Resources

About