Explore and Explain: Self-supervised Navigation and Recounting

Bigazzi, Roberto; Landi, Federico; Cornia, Marcella; Cascianelli, Silvia; Baraldi, Lorenzo; Cucchiara, Rita

doi:10.1109/icpr48806.2021.9412628

Cited by 12 publications

(8 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The resulting learned language groundings are often tightly coupled to the agent's observations and actions, making them inflexible to new objects or concepts. Researchers have explored numerous methods to mitigate this, including data augmentation (Blukis et al, 2020), dual-coding memory (Hill et al, 2021), auxiliary language generation objectives (Yan et al, 2022;Bigazzi et al, 2021), or interactive supervision (Kulick et al, 2013;Mohan & Laird, 2014;She et al, 2014;Thomason et al, 2017;Co-Reyes et al, 2018;Chai et al, 2018;Nguyen et al, 2021). Recent approaches instead leverage internet-scale language models to achieve generalization.…”

Section: Learning From Human Interactionsmentioning

confidence: 99%

Distilling Internet-Scale Vision-Language Models into Embodied Agents

Sumers¹,

Marino²,

Ahuja³

et al. 2023

Preprint

View full text Add to dashboard Cite

Instruction-following agents must ground language into their observation and action spaces. Learning to ground language is challenging, typically requiring domain-specific engineering or large quantities of human interaction data. To address this challenge, we propose using pretrained vision-language models (VLMs) to supervise embodied agents. We combine ideas from model distillation and hindsight experience replay (HER), using a VLM to retroactively generate language describing the agent's behavior. Simple prompting allows us to control the supervision signal, teaching an agent to interact with novel objects based on their names (e.g., planes) or their features (e.g., colors) in a 3D rendered environment. Fewshot prompting lets us teach abstract category membership, including pre-existing categories (food vs toys) and ad-hoc ones (arbitrary preferences over objects). Our work outlines a new and effective way to use internet-scale VLMs, repurposing the generic language grounding acquired by such models to teach task-relevant groundings to embodied agents.

show abstract

Section: Learning From Human Interactionsmentioning

confidence: 99%

Distilling Internet-Scale Vision-Language Models into Embodied Agents

Sumers¹,

Marino²,

Ahuja³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, many state-of-the-art architectures are still considered black boxes, as their behavior lacks explainability [2]. In this respect, some attempts have been made to make the robot navigation and decision-making processes more interpretable for the end-user by letting it produce natural language descriptions of what it observes [5,12,3]. In this work, we consider a curiositydriven exploration agent [30] and equip it with the ability to produce natural language descriptions of what it observes while navigating the environment, also exploiting explainable maps to enhance the interpretability of the descriptions.…”

Section: Related Workmentioning

confidence: 99%

Towards Explainable Navigation and Recounting

Poppi,

Bigazzi,

Rawal

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Both autonomous robotics [5,14] and embodied AI [8,11,18,6,12,21] have recently witnessed a boost of interest, which has been enabled by the release of photorealistic 3D simulated environments. In such environments, algorithms for intelligent exploration and navigation can be developed safely and more quickly than in the real-world, before being easily deployed on real robotic platforms [15,7,2].…”

Section: Related Workmentioning

confidence: 99%

Embodied Navigation at the Art Gallery

Bigazzi¹,

Landi²,

Cascianelli³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Embodied agents, trained to explore and navigate indoor photorealistic environments, have achieved impressive results on standard datasets and benchmarks. So far, experiments and evaluations have involved domestic and working scenes like offices, flats, and houses. In this paper, we build and release a new 3D space with unique characteristics: the one of a complete art museum. We name this environment ArtGallery3D (AG3D). Compared with existing 3D scenes, the collected space is ampler, richer in visual features, and provides very sparse occupancy information. This feature is challenging for occupancy-based agents which are usually trained in crowded domestic environments with plenty of occupancy information. Additionally, we annotate the coordinates of the main points of interest inside the museum, such as paintings, statues, and other items. Thanks to this manual process, we deliver a new benchmark for PointGoal navigation inside this new space. Trajectories in this dataset are far more complex and lengthy than existing groundtruth paths for navigation in Gibson and Matterport3D. We carry on extensive experimental evaluation using our new space for evaluation and prove that existing methods hardly adapt to this scenario. As such, we believe that the availability of this 3D model will foster future research and help improve existing solutions.

show abstract

Explore and Explain: Self-supervised Navigation and Recounting

Cited by 12 publications

References 21 publications

Distilling Internet-Scale Vision-Language Models into Embodied Agents

Distilling Internet-Scale Vision-Language Models into Embodied Agents

Towards Explainable Navigation and Recounting

Embodied Navigation at the Art Gallery

Contact Info

Product

Resources

About