VIRDO: Visio-tactile Implicit Representations of Deformable Objects

Wi, Youngsun; Florence, Pete; Zeng, Andy; Fazeli, Nima

doi:10.1109/icra46639.2022.9812097

Cited by 15 publications

(9 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Control is achieved by predicting a flow field from action, which utilizes the correspondence field to drive to the target state. VIRDO [WFZF22] pretrains an object module to learn the nominal shape of the object, i.e. the shape before deformation and utilizes an encoder to incorporate measured external forces and contact points in learning the deformation field.…”

Section: State‐of‐the‐art Methodsmentioning

confidence: 99%

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Yunus,

Lenssen,

Niemeyer

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

Reconstructing models of the real world, including 3D geometry, appearance, and motion of real scenes, is essential for computer graphics and computer vision. It enables the synthesizing of photorealistic novel views, useful for the movie industry and AR/VR applications. It also facilitates the content creation necessary in computer games and AR/VR by avoiding laborious manual design processes. Further, such models are fundamental for intelligent computing systems that need to interpret real‐world scenes and actions to act and interact safely with the human world. Notably, the world surrounding us is dynamic, and reconstructing models of dynamic, non‐rigidly moving scenes is a severely underconstrained and challenging problem. This state‐of‐the‐art report (STAR) offers the reader a comprehensive summary of state‐of‐the‐art techniques with monocular and multi‐view inputs such as data from RGB and RGB‐D sensors, among others, conveying an understanding of different approaches, their potential applications, and promising further research directions. The report covers 3D reconstruction of general non‐rigid scenes and further addresses the techniques for scene decomposition, editing and controlling, and generalizable and generative modeling. More specifically, we first review the common and fundamental concepts necessary to understand and navigate the field and then discuss the state‐of‐the‐art techniques by reviewing recent approaches that use traditional and machine‐learning‐based neural representations, including a discussion on the newly enabled applications. The STAR is concluded with a discussion of the remaining limitations and open challenges.

show abstract

Section: State‐of‐the‐art Methodsmentioning

confidence: 99%

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Yunus,

Lenssen,

Niemeyer

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…However, learning a full 3D representation in an endto-end framework for realistic deformable objects remains an open question. A line of concurrent work (Wi et al 2022a(Wi et al , 2022b studies volume metric deformable objects using a combination of explicit and implicit 3D representations, with a focus on bendable tools like spatulas.…”

Section: Learning Deformable Dynamics Models For Manipulationmentioning

confidence: 99%

Action-conditional implicit visual dynamics for deformable object manipulation

Shen

Jiang

Choy

et al. 2023

The International Journal of Robotics Research

View full text Add to dashboard Cite

Manipulating volumetric deformable objects in the real world, like plush toys and pizza dough, brings substantial challenges due to infinite shape variations, non-rigid motions, and partial observability. We introduce ACID, an action-conditional visual dynamics model for volumetric deformable objects based on structured implicit neural representations. ACID integrates two new techniques: implicit representations for action-conditional dynamics and geodesics-based contrastive learning. To represent deformable dynamics from partial RGB-D observations, we learn implicit representations of occupancy and flow-based forward dynamics. To accurately identify state change under large non-rigid deformations, we learn a correspondence embedding field through a novel geodesics-based contrastive loss. To evaluate our approach, we develop a simulation framework for manipulating complex deformable shapes in realistic scenes and a benchmark containing over 17,000 action trajectories with six types of plush toys and 78 variants. Our model achieves the best performance in geometry, correspondence, and dynamics predictions over existing approaches. The ACID dynamics models are successfully employed for goal-conditioned deformable manipulation tasks, resulting in a 30% increase in task success rate over the strongest baseline. Furthermore, we apply the simulation-trained ACID model directly to real-world objects and show success in manipulating them into target configurations. https://b0ku1.github.io/acid/

show abstract

“…The tactile information acquired from the physical interaction with an object can include a range of different features describing the objects' (i) geometry, such as size [171,185], edges [186,187], curvatures [188], shape [189,190], or texture [191][192][193][194]; (ii) dynamics, such as inertia [195], stiffness [196][197][198], friction [199,200], deformation [201], or contained fluid [202]; and (iii) properties related to the robot controllers, such as grasping/picking pose [129,203], slippage [134,204], contact point [205], or in-hand object pose [206]. We will focus on the tactile features, which are related to the agri-food domain and present the challenges and shortcomings of the reviewed approaches.…”

Section: Tactile Feature Extractionmentioning

confidence: 99%

Tactile-Sensing Technologies: Trends, Challenges and Outlook in Agri-Food Manipulation

Mandil,

Rajendran,

Nazari

et al. 2023

Sensors

View full text Add to dashboard Cite

Tactile sensing plays a pivotal role in achieving precise physical manipulation tasks and extracting vital physical features. This comprehensive review paper presents an in-depth overview of the growing research on tactile-sensing technologies, encompassing state-of-the-art techniques, future prospects, and current limitations. The paper focuses on tactile hardware, algorithmic complexities, and the distinct features offered by each sensor. This paper has a special emphasis on agri-food manipulation and relevant tactile-sensing technologies. It highlights key areas in agri-food manipulation, including robotic harvesting, food item manipulation, and feature evaluation, such as fruit ripeness assessment, along with the emerging field of kitchen robotics. Through this interdisciplinary exploration, we aim to inspire researchers, engineers, and practitioners to harness the power of tactile-sensing technology for transformative advancements in agri-food robotics. By providing a comprehensive understanding of the current landscape and future prospects, this review paper serves as a valuable resource for driving progress in the field of tactile sensing and its application in agri-food systems.

show abstract

VIRDO: Visio-tactile Implicit Representations of Deformable Objects

Cited by 15 publications

References 63 publications

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Action-conditional implicit visual dynamics for deformable object manipulation

Tactile-Sensing Technologies: Trends, Challenges and Outlook in Agri-Food Manipulation

Contact Info

Product

Resources

About