An Information-Theoretic Predictive Model for the Accuracy of AI Agents Adapted from Psychometrics

Chmait, Nader; Dowe, David L.; Li, Yuan Fang; Green, David G.

doi:10.1007/978-3-319-63703-7_21

Cited by 9 publications

(7 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An accurate AI system can deal with errors and problems and yields reproducible output with the same input under the example, a feature squeeze recommended various methods to ensure the input space-making items. For instance, a feature that squeezes problems reduces the complexity of the input space, making it less likely to be subjected to system problems [142]. Another proposed method involves incorporating problematic case examples into the system's training data [143,144].…”

Section: Acceptance Of Aimentioning

confidence: 99%

Towards Risk-Free Trustworthy Artificial Intelligence: Significance and Requirements

Alzubaidi,

Al-Sabaawi,

Bai

et al. 2023

International Journal of Intelligent Systems

View full text Add to dashboard Cite

Given the tremendous potential and influence of artificial intelligence (AI) and algorithmic decision-making (DM), these systems have found wide-ranging applications across diverse fields, including education, business, healthcare industries, government, and justice sectors. While AI and DM offer significant benefits, they also carry the risk of unfavourable outcomes for users and society. As a result, ensuring the safety, reliability, and trustworthiness of these systems becomes crucial. This article aims to provide a comprehensive review of the synergy between AI and DM, focussing on the importance of trustworthiness. The review addresses the following four key questions, guiding readers towards a deeper understanding of this topic: (i) why do we need trustworthy AI? (ii) what are the requirements for trustworthy AI? In line with this second question, the key requirements that establish the trustworthiness of these systems have been explained, including explainability, accountability, robustness, fairness, acceptance of AI, privacy, accuracy, reproducibility, and human agency, and oversight. (iii) how can we have trustworthy data? and (iv) what are the priorities in terms of trustworthy requirements for challenging applications? Regarding this last question, six different applications have been discussed, including trustworthy AI in education, environmental science, 5G-based IoT networks, robotics for architecture, engineering and construction, financial technology, and healthcare. The review emphasises the need to address trustworthiness in AI systems before their deployment in order to achieve the AI goal for good. An example is provided that demonstrates how trustworthy AI can be employed to eliminate bias in human resources management systems. The insights and recommendations presented in this paper will serve as a valuable guide for AI researchers seeking to achieve trustworthiness in their applications.

show abstract

Section: Acceptance Of Aimentioning

confidence: 99%

Towards Risk-Free Trustworthy Artificial Intelligence: Significance and Requirements

Alzubaidi,

Al-Sabaawi,

Bai

et al. 2023

International Journal of Intelligent Systems

View full text Add to dashboard Cite

show abstract

“…A completely different approach is Item Response Theory, a well-developed subdiscipline in psychometrics (Embretson and Reise 2000), only recently brought to AI (Martínez-Plumed et al 2016;Lalor 2020). IRT has been used in several areas of AI, where the AI systems are treated as respondents and the tasks as items, including classification (Martínez-Plumed et al 2016;Martínez-Plumed et al 2019;Chen and Ahn 2020), regression (Moraes et al 2020), multi-agent scenarios (Chmait et al 2017), XAI (Kline et al 2020) and other AI benchmarks (Martínez-Plumed and Hernández-Orallo 2017;Hernández-Orallo 2018, 2020).…”

Section: Related Workmentioning

confidence: 99%

When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty

Martínez-Plumed

Castellano

Monserrat-Aranda

et al. 2022

AAAI

View full text Add to dashboard Cite

One of challenges of artificial intelligence as a whole is robustness. Many issues such as adversarial examples, out of distribution performance, Clever Hans phenomena, and the wider areas of AI evaluation and explainable AI, have to do with the following question: Did the system fail because it is a hard instance or because something else? In this paper we address this question with a generic method for estimating IRT-based instance difficulty for a wide range of AI domains covering several areas, from supervised feature-based classification to automated reasoning. We show how to estimate difficulty systematically using off-the-shelf machine learning regression models. We illustrate the usefulness of this estimation for a range of applications.

show abstract

“…They considered a series of factors hindering and influencing the effectiveness of interactive cognitive systems [19][20][21].…”

Section: Introductionmentioning

confidence: 99%

Evaluation Mechanism of Collective Intelligence for Heterogeneous Agents Group

Dai

Zhao

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Collective intelligence is manifested when multiple agents coherently work in observation, interaction, decision-making and action. In this paper, we define and quantify the intelligence level of heterogeneous agents group with the improved Anytime Universal Intelligence Test(AUIT), based on an extension of the existing evaluation of homogeneous agents group. The relationship of intelligence level with agents composition, group size, spatial complexity and testing time is analyzed. The intelligence level of heterogeneous agents groups is compared with the homogeneous ones to analyze the effects of heterogeneity on collective intelligence. Our work will help to understand the essence of collective intelligence more deeply and reveal the effect of various key factors on group intelligence level.

show abstract

An Information-Theoretic Predictive Model for the Accuracy of AI Agents Adapted from Psychometrics

Cited by 9 publications

References 16 publications

Towards Risk-Free Trustworthy Artificial Intelligence: Significance and Requirements

Towards Risk-Free Trustworthy Artificial Intelligence: Significance and Requirements

When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty

Evaluation Mechanism of Collective Intelligence for Heterogeneous Agents Group

Contact Info

Product

Resources

About