Towards Flexible Task Environments for Comprehensive Evaluation of Artificial Intelligent Systems and Automatic Learners

Þórisson, Kristinn R.; Bieger, Jordi; Schiffel, Stephan; Garrett, Deon

doi:10.1007/978-3-319-21365-1_20

Cited by 11 publications

(12 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several researchers believe that it would be beneficial if one used the term "artificial intelligence" only when referring to "artificial general intelligence" (AGI), that is, the intelligence of a machine that can understand or learn any intellectual task that a human being can (Goertzel, 2015;Thórisson, Bieger, Schiffel, & Garrett, 2015).…”

Section: The Most Restrictive Definitionmentioning

confidence: 99%

Artificial Intelligence and Marketing: Pitfalls and Opportunities

Bruyn

Viswanathan

Beh

et al. 2020

Journal of Interactive Marketing

236

100

View full text Add to dashboard Cite

This article discusses the pitfalls and opportunities of AI in marketing through the lenses of knowledge creation and knowledge transfer. First, we discuss the notion of "higher-order learning" that distinguishes AI applications from traditional modeling approaches, and while focusing on recent advances in deep neural networks, we cover its underlying methodologies (multilayer perceptron, convolutional, and recurrent neural networks) and learning paradigms (supervised, unsupervised, and reinforcement learning). Second, we discuss the technological pitfalls and dangers marketing managers need to be aware of when implementing AI in their organizations, including the concepts of badly defined objective functions, unsafe or unrealistic learning environments, biased AI, explainable AI, and controllable AI. Third, AI will have a deep impact on predictive tasks that can be automated and require little explainability, we predict that AI will fall short of its promises in many marketing domains if we do not solve the challenges of tacit knowledge transfer between AI models and marketing organizations.

show abstract

Section: The Most Restrictive Definitionmentioning

confidence: 99%

Artificial Intelligence and Marketing: Pitfalls and Opportunities

Bruyn

Viswanathan

Beh

et al. 2020

Journal of Interactive Marketing

236

100

View full text Add to dashboard Cite

show abstract

“…A task theory should similarly allow us to relate the features of a task to measurable physical and/or conceptual aspects, enabling comparison of similar and dissimilar tasks, and facilitate the construction of task-environments and variations on known tasks without changing their nature, so that we may select or design tasks capable of measuring various aspects of AIs.We would like to be able to compose and decompose tasks and environments, and scale them up or down in size or complexity in accordance with robust and well-understood principles. In order to provide a characterization of task-environments, measures ought to be defined for properties like determinism, ergodicity, continuity, asynchronicity, dynamism, observability, controllability,periodicity, and repeatability [11].…”

Section: What We Might Want From a Task Theorymentioning

confidence: 99%

“…We've argued before that a good A(G)I evaluation framework should enable the easy manual and automatic construction of task-environments and their variants as well as facilitate the analysis of parameters of interest [11]. A task theory should similarly allow us to relate the features of a task to measurable physical and/or conceptual aspects, enabling comparison of similar and dissimilar tasks, and facilitate the construction of task-environments and variations on known tasks without changing their nature, so that we may select or design tasks capable of measuring various aspects of AIs.We would like to be able to compose and decompose tasks and environments, and scale them up or down in size or complexity in accordance with robust and well-understood principles.…”

Section: What We Might Want From a Task Theorymentioning

confidence: 99%

“…procedural generation of almost completely random tasks [2,7]. 2 To adequately evaluate AGI-aspiring systems, rigorous tests profiling cognitive abilities such as transfer of training, knowledge retention, attentional control, and knowledge acquisition rate, would be highly valuable [11]. A deep and general understanding of tasks and the ability to construct and compare related tasks would greatly facilitate the design of an evaluation tool with these capabilities.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Why Artificial Intelligence Needs a Task Theory

Þórisson

Bieger

Thorarensen

et al. 2016

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

The concept of "task" is at the core of artificial intelligence (AI): Tasks are used for training and evaluating AI systems, which are built in order to perform and automatize tasks we deem useful. In other fields of engineering theoretical foundations allow thorough evaluation of designs by methodical manipulation of well understood parameters with a known role and importance; this allows an aeronautics engineer, for instance, to systematically assess the effects of wind speed on an airplane's performance and stability. No framework exists in AI that allows this kind of methodical manipulation: Performance results on the few tasks in current use (cf. board games, question-answering) cannot be easily compared, however similar or different. The issue is even more acute with respect to artificial general intelligence systems, which must handle unanticipated tasks whose specifics cannot be known beforehand. A task theory would enable addressing tasks at the class level, bypassing their specifics, providing the appropriate formalization and classification of tasks, environments, and their parameters, resulting in more rigorous ways of measuring, comparing, and evaluating intelligent behavior. Even modest improvements in this direction would surpass the current ad-hoc nature of machine learning and AI evaluation. Here we discuss the main elements of the argument for a task theory and present an outline of what it might look like for physical tasks.

show abstract

“…Thus, in this study, rather than discussing particular approaches, which are indeed very diverse in the issues they attempt to tackle, we consider beneficial to organise the discussion based on four major directions of research in machine education, which already attracted sufficient interest and generated a body of literature which, while still small, is sufficient to create a momentum that moves forward the whole field. These directions account for the educational goal, or in other words, the learning outcome that the educator wants to maximise, and are as follows: improving the learning speed [2], improving the task proficiency [3], achieve trustworthy AI [4], and achieve strong AI [5].…”

Section: Introductionmentioning

confidence: 99%

On educating machines

Leu,

Tang

2019

Preprint

View full text Add to dashboard Cite

Machine education is an emerging research field that focuses on the problem which is inverse to machine learning. To date, the literature on educating machines is still in its infancy. A fairly low number of methodology and method papers are scattered throughout various formal and informal publication avenues, mainly because the field is not yet well coalesced (with no well established discussion forums or investigation pathways), but also due to the breadth of its potential ramifications and research directions. In this study we bring together the existing literature and organise the discussion into a small number of research directions (out of many) which are to date sufficiently explored to form a minimal critical mass that can push the machine education concept further towards a standalone research field status.

show abstract

Towards Flexible Task Environments for Comprehensive Evaluation of Artificial Intelligent Systems and Automatic Learners

Cited by 11 publications

References 13 publications

Artificial Intelligence and Marketing: Pitfalls and Opportunities

Artificial Intelligence and Marketing: Pitfalls and Opportunities

Why Artificial Intelligence Needs a Task Theory

On educating machines

Contact Info

Product

Resources

About