Shot Contrastive Self-Supervised Learning for Scene Boundary Detection

Chen, Shixing; Nie, Xiaohan; Fan, David D.; Zhang, Dongqing; Bhat, Vimal; Hamid, Raffay

doi:10.48550/arxiv.2104.13537

Cited by 2 publications

(3 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, the experiments are conducted on both SS task and SSC task. For SS task, LGSS [18] and ShotCoL [7] are currently the SOTA algorithms, so we use them as our strong baselines. Other methods such as Siamese [2], StoryGraph [30], Grouping [21] and etc., are not fresh enough to compare.…”

Section: Compared Methodsmentioning

confidence: 99%

“…LGSS [18] raises B-Net to identify boundaries and uses multimodal features, and ShotCoL [7] leverages contrastive learning. However, their work performed a 2-classes classification which refers to whether the shot should be segmented or not, ignoring richer relation information between shots within a scene.…”

Section: Related Work 21 Video Segmentationmentioning

confidence: 99%

“…In previous works [7,18], mean average precision (mAP) and recall are used for evaluate SS. However, the metrics are not consistent and they need enumerating thresholds to calculate while the thresholds are not common in various algorithms.…”

Section: Evaluation Criteriamentioning

confidence: 99%

See 2 more Smart Citations

OS-MSL: One Stage Multimodal Sequential Link Framework for Scene Segmentation and Classification

Liu¹,

Qiao²,

Yin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Scene segmentation and classification (SSC) serve as a critical step towards the field of video structuring analysis. Intuitively, jointly learning of these two tasks can promote each other by sharing common information. However, scene segmentation concerns more on the local difference between adjacent shots while classification needs the global representation of scene segments, which probably leads to the model dominated by one of the two tasks in the training phase. In this paper, from an alternate perspective to overcome the above challenges, we unite these two tasks into one task by a new form of predicting shots link: a link connects two adjacent shots, indicating that they belong to the same scene or category. To the end, we propose a general One Stage Multimodal Sequential Link Framework (OS-MSL) to both distinguish and leverage the twofold semantics by reforming the two learning tasks into a unified one. Furthermore, we tailor a specific module called DiffCorrNet to explicitly extract the information of differences and correlations among shots. Extensive experiments on a brand-new large scale dataset collected from real-world applications, and MovieScenes are conducted. Both the results demonstrate the effectiveness of our proposed method against strong baselines. CCS CONCEPTS• Computing methodologies → Computer vision tasks.

show abstract

Section: Compared Methodsmentioning

confidence: 99%

Section: Related Work 21 Video Segmentationmentioning

confidence: 99%

See 1 more Smart Citation

OS-MSL: One Stage Multimodal Sequential Link Framework for Scene Segmentation and Classification

Liu¹,

Qiao²,

Yin³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection

Kang

Kim

et al. 2021

Preprint

View full text Add to dashboard Cite

Generic Event Boundary Detection (GEBD) is a newly suggested video understanding task that aims to find one level deeper semantic boundaries of events. Bridging the gap between natural human perception and video understanding, it has various potential applications, including interpretable and semantically valid video parsing. Still at an early development stage, existing GEBD solvers are simple extensions of relevant video understanding tasks, disregarding GEBD's distinctive characteristics. In this paper, we propose a novel framework for unsupervised/supervised GEBD, by using the Temporal Self-similarity Matrix (TSM) as the video representation. The new Recursive TSM Parsing (RTP) algorithm exploits local diagonal patterns in TSM to detect boundaries, and it is combined with the Boundary Contrastive (BoCo) loss to train our encoder to generate more informative TSMs. Our framework can be applied to both unsupervised and supervised settings, with both achieving state-of-the-art performance by a huge margin in GEBD benchmark. Especially, our unsupervised method outperforms the previous state-of-the-art "supervised" model, implying its exceptional efficacy.

show abstract

Shot Contrastive Self-Supervised Learning for Scene Boundary Detection

Cited by 2 publications

References 29 publications

OS-MSL: One Stage Multimodal Sequential Link Framework for Scene Segmentation and Classification

OS-MSL: One Stage Multimodal Sequential Link Framework for Scene Segmentation and Classification

UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection

Contact Info

Product

Resources

About