Human Strategic Steering Improves Performance of Interactive Optimization

Abstract

A central concern in an interactive intelligent system is optimization of its actions, to be maximally helpful to its human user. In recommender systems for instance, the action is to choose what to recommend, and the optimization task is to recommend items the user prefers. The optimization is done based on earlier user's feedback (e.g. "likes" and "dislikes"), and the algorithms assume the feedback to be faithful. That is, when the user clicks like, they actually prefer the item. We argue that this fundamental assumption can be extensively violated by human users, who are not passive feedback sources. Instead, they are in control, actively steering the system towards their goal. To verify this hypothesis, that humans steer and are able to improve performance by steering, we designed a function optimization task where a human and an optimization algorithm collaborate to find the maximum of a 1-dimensional function. At each iteration, the optimization algorithm queries the user for the value of a hidden function f at a point x, and the user, who sees the hidden function, provides an answer about f (x). Our study on 21 participants shows that users who understand how the optimization works, strategically provide biased answers (answers not equal to f (x)), which results in the algorithm finding the optimum significantly faster. Our work highlights that next-generation intelligent systems will need user models capable of helping users who steer systems to pursue their goals.

Bayesian Optimization Augmented with Actively Elicited Expert Knowledge

“…This would bring this work closer to the multi-fidelity BO line of research (Kandasamy et al, 2016;Takeno et al, 2020;Li et al, 2020). In that scenario, as humans are not passive sources of information, but rather active planners, we would need to build models able to anticipate behaviours such as steering (Colella et al, 2020).…”

A Workflow for building Computationally Rational Models of Human Behavior

“…This model is geared for strategically steering the system towards user-desired behavior during interaction. While systems that comply with this steering may exhibit stronger interaction performance (Colella et al, 2020), a more advanced system could aim to identify and learn from users' mental models of AI, their refinement over the course of the interaction, and the influence of mutable user goals on interaction behavior. Developing such mental models of AI systems is currently a research challenge, even more so in the context of learning these online during interaction (Howes et al, 2023;Steyvers and Kumar, 2022;Bansal et al, 2019).…”

The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop Failures

“…Although prior research [11,21,38,42,64,66] that involves HITL strategies have shown human knowledge to be helpful for a machine to learn, previous literature rarely discusses the circumstances under which HITL could shine. Especially when users intentionally or unintentionally provide defective or uncertain inputs, it is unclear whether the system can continue to process it effectively and whether other cascading effects will be triggered.…”

“…With the increasing interest in human-AI interaction, human-in-theloop (HITL) [45] systems have been applied to a wide range of domains, such as material design [6], animation design [5], photo color enhancement [39], image restoration [64], and more [11,21,38,67]. These systems actively exploit human choices for optimizing machine results.…”