A Simple Framework for Open-Vocabulary Segmentation and Detection

Zhang, Hao; Li, Feng; Zou, Xueyan; Liu, Shilong; Li, Chunyuan; Gao, Jianfeng; Yang, Jun; Zhang, Lei

doi:10.48550/arxiv.2303.08131

Cited by 5 publications

(5 citation statements)

References 47 publications

(83 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zero-shot learning is a way to generalize on unseen labels, without having specifically trained to classify them. We compare SAM with zero-shot segmentation models from the state of the art, OpenSeed [60,56] and CLIPSeg [37], using the mean Intersection-Over-Union (mIoU) metric. The IoU is defined as the area of overlap between the predicted segmentation and the ground truth divided by the area of union between the predicted segmentation and the ground truth.…”

Section: Segmentation Using Sammentioning

confidence: 99%

Semantic Segmentation using Foundation Models for Cultural Heritage: an Experimental Study on Notre-Dame de Paris

Réby,

Guilhelm,

De Luca

2023

2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

The zero-shot performance of foundation models has captured a lot of attention. Specifically, the Segment Anything Model (SAM) has gained popularity in computer vision due to its label-free segmentation capabilities. Our study proposes using SAM on cultural heritage data, specifically images of Notre-Dame de Paris, with a controlled vocabulary. SAM can successfully identify objects within the cathedral. To further improve segmentation, we utilized Grounding DINO to detect objects and CLIP to automatically add labels from the segmentation masks generated by SAM. Our study demonstrates the usefulness of foundation models for zero-shot semantic segmentation of cultural heritage data.

show abstract

Section: Segmentation Using Sammentioning

confidence: 99%

Semantic Segmentation using Foundation Models for Cultural Heritage: an Experimental Study on Notre-Dame de Paris

Réby,

Guilhelm,

De Luca

2023

2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

show abstract

“…Examples of such neural networks are the classic Mask R-CNN model [34], the fast YOLACT-EDGE model [35], which takes into account a sequence of images, and the most recent YOLOv8 model [36]. Existing transformer models for object segmentation with a fixed number of objects, such as OneFormer [37], or models with an open vocabulary, such as OpenSeed [38], are not considered due to their limited computational efficiency. Semantic segmentation methods are also not considered, as they do not separate closely located objects of the same class.…”

Section: ) Neural Network-based Object Segmentationmentioning

confidence: 99%

Fine-Tuning Multimodal Transformer Models for Generating Actions in Virtual and Real Environments

Staroverov,

Gorodetsky,

Krishtopik

et al. 2023

IEEE Access

View full text Add to dashboard Cite

In this work, we propose and investigate an original approach to using a pre-trained multimodal transformer of a specialized architecture for controlling a robotic agent in an object manipulation task based on language instruction, which we refer to as RozumFormer. Our model is based on a bimodal (text-image) transformer architecture originally trained for solving tasks that use one or both modalities, such as language modeling, visual question answering, image captioning, text recognition, text-to-image generation, etc. The discussed model was adapted for robotic manipulation tasks by organizing the input sequence of tokens in a particular way, consisting of tokens for text, images, and actions. We demonstrated that such a model adapts well to new tasks and shows better results with fine-tuning than complete training in simulation and real environments. To transfer the model from the simulator to a real robot, new datasets were collected and annotated. In addition, experiments controlling the agent in a visual environment using reinforcement learning have shown that fine-tuning the model with a mixed dataset that includes examples from the initial visual-linguistic tasks only slightly decreases performance on these tasks, simplifying the addition of tasks from another domain.INDEX TERMS Action generation, bimodal transformer models, intelligent agent, robotic manipulator arm control.

show abstract

“…Improved depth estimation of pedestrians and vehicles through a LiDAR data-like data representation Pseudo-LiDAR is studied in [238,239] Finally, 3D reconstruction methods have developed greatly with more learning integrated into the 3D reconstruction pipeline [240][241][242][243][244], from learned monocular depthmaps [245], to learned 3D reconstruction features [246] to 3d matching [247] and the visually pleasing NERF-based methods [248,249]. Semantic segmentation, tracking, and object detection methods are also becoming less supervised utilizing learned matching and language model based labels [144,[250][251][252][253][254][255][256][257]. Combining different visual tasks like object detection semantics segmentation, tracking with 3D modeling has seen success in [144,258].…”

Section: Developments In the Fieldmentioning

confidence: 99%

Semantic Synthesis of Pedestrian Locomotion

Priisalu

Păduraru

Pirinen

et al. 2021

Computer Vision – ACCV 2020

View full text Add to dashboard Cite

It is difficult to perform 3D reconstruction from on-vehicle gathered video due to the large forward motion of the vehicle. Even object detection and human sensing models perform significantly worse on onboard videos when compared to standard benchmarks because objects often appear far away from the camera compared to the standard object detection benchmarks, image quality is often decreased by motion blur and occlusions occur often. This has led to the popularisation of traffic data-specific benchmarks. Recently Light Detection And Ranging (LiDAR) sensors have become popular to directly estimate depths without the need to perform 3D reconstructions. However, LiDAR-based methods still lack in articulated human detection at a distance when compared to image-based methods. We hypothesize that benchmarks targeted at articulated human sensing from LiDAR data could bring about increased research in human sensing and prediction in traffic and could lead to improved traffic safety for pedestrians.

show abstract

A Simple Framework for Open-Vocabulary Segmentation and Detection

Cited by 5 publications

References 47 publications

Semantic Segmentation using Foundation Models for Cultural Heritage: an Experimental Study on Notre-Dame de Paris

Semantic Segmentation using Foundation Models for Cultural Heritage: an Experimental Study on Notre-Dame de Paris

Fine-Tuning Multimodal Transformer Models for Generating Actions in Virtual and Real Environments

Semantic Synthesis of Pedestrian Locomotion

Contact Info

Product

Resources

About