A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges

Cruz, Christian Arzate; Igarashi, Takeo

doi:10.48550/arxiv.2105.12949

Cited by 2 publications

(3 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Task-Dependency -We can generally assume that the feedback quality is dependent on the task, e.g., a tasks observation space-dimensionality, complexity, diversity of possible (close-to)-optimal policies, among other characteristics. Progress-Dependency -Feedback can differ in quality at different training phases, e.g., at the beginning or with a converged model policy (Arzate Cruz & Igarashi, 2020;Knox et al, 2012).…”

Section: Dependencies Influencing Feedback Qualitymentioning

confidence: 99%

“…Additionally, there are additional dependencies that may influence feedback but are only partially measurable because they cannot be fully disentangled from other dependencies: Personality-Dependency -Feedback can depend on a human's personality or background, which might influence the quality under different criteria. This might encompass things such as directness or frustration tolerance (Arzate Cruz & Igarashi, 2020;Lindner & El-Assady, 2022). Knowledge-Dependency -A human's ability to give highquality feedback can depend on their knowledge, i.e., if the task is of high complexity or requires domain expertise.…”

Section: Dependencies Influencing Feedback Qualitymentioning

confidence: 99%

“…F1: Evaluative Feedback -We define this type of feedback for cases in which a human gives a numerical or otherwise quantifiable judgment of a target, e.g., binary feedback or a numeric score (Arzate Cruz & Igarashi, 2020). It is generally defined for a single target (simultaneous feedback for a group of targets can be split into single-target feedback by assuming conditional inde-pendence).…”

Section: Feedback Typesmentioning

confidence: 99%

See 2 more Smart Citations

VISITOR: Visual Interactive State Sequence Exploration for Reinforcement Learning

Metz

Bykovets

Joos

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Understanding the behavior of deep reinforcement learning agents is a crucial requirement throughout their development. Existing work has addressed the identification of observable behavioral patterns in state sequences or analysis of isolated internal representations; however, the overall decision‐making of deep‐learning RL agents remains opaque. To tackle this, we present VISITOR, a visual analytics system enabling the analysis of entire state sequences, the diagnosis of singular predictions, and the comparison between agents. A sequence embedding view enables the multiscale analysis of state sequences, utilizing custom embedding techniques for a stable spatialization of the observations and internal states. We provide multiple layers: (1) a state space embedding, highlighting different groups of states inside the state‐action sequences, (2) a trajectory view, emphasizing decision points, (3) a network activation mapping, visualizing the relationship between observations and network activations, (4) a transition embedding, enabling the analysis of state‐to‐state transitions. The embedding view is accompanied by an interactive reward view that captures the temporal development of metrics, which can be linked directly to states in the embedding. Lastly, a model list allows for the quick comparison of models across multiple metrics. Annotations can be exported to communicate results to different audiences. Our two‐stage evaluation with eight experts confirms the effectiveness in identifying states of interest, comparing the quality of policies, and reasoning about the internal decision‐making processes.

show abstract

Section: Dependencies Influencing Feedback Qualitymentioning

confidence: 99%

Section: Dependencies Influencing Feedback Qualitymentioning

confidence: 99%

Section: Feedback Typesmentioning

confidence: 99%

See 1 more Smart Citation

VISITOR: Visual Interactive State Sequence Exploration for Reinforcement Learning

Metz

Bykovets

Joos

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

PrISM-Tracker

Arakawa

Yakura

Mollyn

et al. 2022

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

A user often needs training and guidance while performing several daily life procedures, e.g., cooking, setting up a new appliance, or doing a COVID test. Watch-based human activity recognition (HAR) can track users' actions during these procedures. However, out of the box, state-of-the-art HAR struggles from noisy data and less-expressive actions that are often part of daily life tasks. This paper proposes PrISM-Tracker, a procedure-tracking framework that augments existing HAR models with (1) graph-based procedure representation and (2) a user-interaction module to handle model uncertainty. Specifically, PrISM-Tracker extends a Viterbi algorithm to update state probabilities based on time-series HAR outputs by leveraging the graph representation that embeds time information as prior. Moreover, the model identifies moments or classes of uncertainty and asks the user for guidance to improve tracking accuracy. We tested PrISM-Tracker in two procedures: latte-making in an engineering lab study and wound care for skin cancer patients at a clinic. The results showed the effectiveness of the proposed algorithm utilizing transition graphs in tracking steps and the efficacy of using simulated human input to enhance performance. This work is the first step toward human-in-the-loop intelligent systems for guiding users while performing new and complicated procedural tasks.

show abstract

A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges

Cited by 2 publications

References 80 publications

VISITOR: Visual Interactive State Sequence Exploration for Reinforcement Learning

VISITOR: Visual Interactive State Sequence Exploration for Reinforcement Learning

PrISM-Tracker

Contact Info

Product

Resources

About