Robot-Assisted Feeding: Generalizing Skewering Strategies Across Food Items on a Plate

Feng, Ryan; Kim, Youngsun; Lee, Gilwoo; Gordon, Ethan K.; Schmittle, Matt; Kumar, Shivaum; Bhattacharjee, Tapomayukh; Srinivasa, Siddhartha S.

doi:10.1007/978-3-030-95459-8_26

Cited by 8 publications

(8 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first step was to ensure that our post hoc context, the haptic data, was descriptive enough to potentially benefit the visual context model. To this end, we collected 115 samples of the robot skewering 3 food items, chosen to be representative of different haptic categories and optimal action (as determined in [7]): grape is classified as "hard skin" and has the optimal action TV90, strawberry is "medium" and prefers VS0 or TV0, and banana is "soft" and prefers TA0 or TA90. For each sample, we recorded the visual context c t , post hoc context p t , action taken a t , loss l t [a t ], and food type name (e.g.…”

Section: B Offline Results and Tuningmentioning

confidence: 99%

“…Research labs have also explored meal preparation [50,51], baking cookies [52], making pancakes [53], separating Oreos [54], and preparing meals [55] with robots. Most of these studies either interacted with a specific food item with a fixed manipulation strategy [52,53] or with an unchanging set of food items and manipulation strategies [7,8,56,57]. Some of these studies have looked at using multi-modal data [55] or online learning [9], but not a combination of the two.…”

Section: B Robot-assisted Feeding: Food Manipulationmentioning

confidence: 99%

“…Our visual context is generated using the Skewering Position Action Network (SPANet) from [7] while our action space and our haptic context specification are derived from human data [6].…”

Section: B Robot-assisted Feeding: Food Manipulationmentioning

confidence: 99%

“…Previous work has identified useful manipulation strategies for a variety of food items [6] and ways to generalize those strategies to similar-looking food items [7,8], but one challenge is figuring out how to select a manipulation strategy for previously unseen food. Recent work has suggested modeling bite acquisition as a contextual bandit [9].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding

Gordon¹,

Roychowdhury²,

Bhattacharjee³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Autonomous robot-assisted feeding requires the ability to acquire a wide variety of food items. However, it is impossible for such a system to be trained on all types of food in existence. Therefore, a key challenge is choosing a manipulation strategy for a previously unseen food item. Previous work showed that the problem can be represented as a linear contextual bandit on visual information. However, food has a wide variety of multi-modal properties relevant to manipulation that can be hard to distinguish visually. Our key insight is that we can leverage the haptic information we collect during manipulation to learn some of these properties and more quickly adapt our visual model to previously unseen food. In general, we propose a modified linear contextual bandit framework augmented with post hoc context observed after action selection to empirically increase learning speed (as measured by cross-validation mean square error) and reduce cumulative regret. Experiments on synthetic data demonstrate that this effect is more pronounced when the dimensionality of the context is large relative to the post hoc context or when the post hoc context model is particularly easy to learn. Finally, we apply this framework to the bite acquisition problem and demonstrate the acquisition of 8 previously unseen types of food with 21% fewer failures across 64 attempts.

show abstract

Section: B Offline Results and Tuningmentioning

confidence: 99%

Section: B Robot-assisted Feeding: Food Manipulationmentioning

confidence: 99%

“…Our visual context is generated using the Skewering Position Action Network (SPANet) from [7] while our action space and our haptic context specification are derived from human data [6].…”

Section: B Robot-assisted Feeding: Food Manipulationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding

Gordon¹,

Roychowdhury²,

Bhattacharjee³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Among the challenges of eating is the ability to prepare and transport food [14], a skill that assistive robotics has tried to enable. Researchers have developed robot policies that can autonomously manipulate and deliver different types of food to users with disabilities [10,25]. However, designing a fully autonomous system to handle a task as variable and personalized as eating is exceedingly challenging: one size does not fit all.…”

Section: Related Workmentioning

confidence: 99%

Shared Autonomy with Learned Latent Actions

Jeon¹,

Losey²,

Sadigh³

2020

Preprint

View full text Add to dashboard Cite

Assistive robots enable people with disabilities to conduct everyday tasks on their own. However, these tasks can be complex, containing both coarse reaching motions and fine-grained manipulation. For example, when eating, not only does one need to move to the correct food item, but they must also precisely manipulate the food in different ways (e.g., cutting, stabbing, scooping). Shared autonomy methods make robot teleoperation safer and more precise by arbitrating user inputs with robot controls. However, these works have focused mainly on the high-level task of reaching a goal from a discrete set, while largely ignoring manipulation of objects at that goal. Meanwhile, dimensionality reduction techniques for teleoperation map useful high-dimensional robot actions into an intuitive lowdimensional controller, but it is unclear if these methods can achieve the requisite precision for tasks like eating. Our insight is that-by combining intuitive embeddings from learned latent actions with robotic assistance from shared autonomy-we can enable precise assistive manipulation. In this work, we adopt learned latent actions for shared autonomy by proposing a new model structure that changes the meaning of the human's input based on the robot's confidence of the goal. We show convergence bounds on the robot's distance to the most likely goal, and develop a training procedure to learn a controller that is able to move between goals even in the presence of shared autonomy. We evaluate our method in simulations and an eating user study. See videos of our experiments here: https://youtu.be/7BouKojzVyk.

show abstract

Towards practical robotic chef: Review of relevant work and future challenges

Sochacki,

Zhang,

Abdulali

et al. 2024

Journal of Field Robotics

View full text Add to dashboard Cite

Robotic chefs are a promising technology that can improve the availability of quality food by reducing the time required for cooking, therefore decreasing food's overall cost. This paper clarifies and structures design and benchmarking rules in this new area of research, and provides a comprehensive review of technologies suitable for the construction of cooking robots. The diner is an ultimate judge of the cooking outcome, therefore we put focus on explaining human food preferences and perception of taste and ways to use them for control. Mechanical design of robotic chefs at a practically low cost remains the challenge, but some recently published gripper designs as well as whole robotic systems show the use of cheap materials or off‐the‐shelf components. Moreover, technologies like taste sensing, machine learning, and computer vision are making their way into robotic cooking enabling smart sensing and therefore improving controllability and autonomy. Furthermore, objective assessment of taste and food palatability is a challenge even for trained humans, therefore the paper provides a list of procedures for benchmarking the robot's tasting and cooking abilities. The paper is written from the point of view of a researcher or engineer building a practical robotic system, therefore there is a strong priority for solutions and technologies that are proven, robust and self‐contained enough to be a part of a larger system.

show abstract

Robot-Assisted Feeding: Generalizing Skewering Strategies Across Food Items on a Plate

Cited by 8 publications

References 23 publications

Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding

Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding

Shared Autonomy with Learned Latent Actions

Towards practical robotic chef: Review of relevant work and future challenges

Contact Info

Product

Resources

About