Complex Interaction as Emergent Behaviour: Simulating Mid-Air Virtual Keyboard Typing using Reinforcement Learning

Hetzel, Lorenz; Dudley, John J.; Feit, Anna Maria; Kristensson, Per Ola

doi:10.1109/tvcg.2021.3106494

Cited by 12 publications

(6 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, we used the data from our user study, which was explicitly recorded for the considered interaction task. However, the obtained torque ranges should be appropriate for related interaction techniques and tasks as well, as was recently shown for the case of mid-air keyboard typing [29]. If major changes to the user model are made, such as modifying the physiology, running CFAT on new reference data is recommended.…”

Section: Cfat: a Methods To Compute Maximum Voluntary Torques Formentioning

confidence: 97%

See 1 more Smart Citation

Simulating Interaction Movements via Model Predictive Control

Klar¹,

Fischer²,

Fleig³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a method to simulate movement in interaction with computers, using Model Predictive Control (MPC). The method starts from understanding interaction from an Optimal Feedback Control (OFC) perspective. We assume that users aim to minimize an internalized cost function, subject to the constraints imposed by the human body and the interactive system. In contrast to previous linear approaches used in HCI, MPC can compute optimal controls for nonlinear systems. This allows us to use state-of-the-art biomechanical models and handle nonlinearities that occur in almost any interactive system. Instead of torque actuation, our model employs second-order muscles acting directly at the joints. We compare three different cost functions and evaluate the simulated trajectories against user movements in a Fitts' Law type pointing study with four different interaction techniques. Our results show that the combination of distance, control, and joint acceleration cost matches individual users' movements best, and predicts movements with an accuracy that is within the between-user variance. To aid HCI researchers and designers, we introduce CFAT, a novel method to identify maximum voluntary torques in joint-actuated models based on experimental data, and give practical advice on how to simulate human movement for different users, interaction techniques, and tasks.CCS Concepts: • Human-centered computing → HCI theory, concepts and models.

show abstract

Section: Cfat: a Methods To Compute Maximum Voluntary Torques Formentioning

confidence: 97%

“…The resulting end-effector trajectories follow both Fitts' Law [19] and the 2 /3 Power Law [35]. Lately, Hetzel et al have extended the model from [16] to simulate mid-air keyboard typing, using the same RL method [29].…”

Section: Deep Learning and Muscle Controlmentioning

confidence: 99%

Simulating Interaction Movements via Model Predictive Control

Klar¹,

Fischer²,

Fleig³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The proposed method utilizes deep RL for collision avoidance and is enabled through AR. In addition to providing a framework for safe symbiotic HRI, Hetzel et al [24] demonstrated the potential of RL to improve the performance of mid-air typing. Many studies have been conducted on enhancing the service experience by providing ideal feedback.…”

Section: The Use Of Reinforcement Learning With Extended Realitymentioning

confidence: 99%

Augmented Reality-Assisted Deep Reinforcement Learning-Based Model towards Industrial Training and Maintenance for NanoDrop Spectrophotometer

Alatawi,

Albalawi,

Shahata

et al. 2023

Sensors

View full text Add to dashboard Cite

The use of augmented reality (AR) technology is growing in the maintenance industry because it can improve efficiency and reduce costs by providing real-time guidance and instruction to workers during repairs and maintenance tasks. AR can also assist with equipment training and visualization, allowing users to explore the equipment’s internal structure and size. The adoption of AR in maintenance is expected to increase as hardware options expand and development costs decrease. To implement AR for job aids in mobile applications, 3D spatial information and equipment details must be addressed, and calibrated using image-based or object-based tracking, which is essential for integrating 3D models with physical components. The present paper suggests a system using AR-assisted deep reinforcement learning (RL)-based model for NanoDrop Spectrophotometer training and maintenance purposes that can be used for rapid repair procedures in the Industry 4.0 (I4.0) setting. The system uses a camera to detect the target asset via feature matching, tracking techniques, and 3D modeling. Once the detection is completed, AR technologies generate clear and easily understandable instructions for the maintenance operator’s device. According to the research findings, the model’s target technique resulted in a mean reward of 1.000 and a standard deviation of 0.000. This means that all the rewards that were obtained in the given task or environment were exactly the same. The fact that the reward standard deviation is 0.000 shows that there is no variability in the outcomes.

show abstract

“…Deep RL has been used to learn control policies in increasingly complex state-action spaces -such as torque-actuated humanoids [7,18,53,77] and musculoskeletal systems [37,51], as well as eye-hand coordination in typing [31]. Hetzel et al [25] presented an RL agent for simulating joint-controlled movement of hands in typing, however lamenting that while muscle control would have been preferable, they were not able to train a model that had muscles. Moveover, their control problem was not perceptual like ours; their agent state was a vector describing joint kinematics and the position of next target key.…”

Section: Reinforcement Learning -Based User Modellingmentioning

confidence: 99%