Temporally stable video segmentation without video annotations

Azulay, Aharon; Halperin, Tavi; Vantzos, Orestis; Bornstein, Nadav; Bibi, Ofir

doi:10.1109/wacv51458.2022.00198

Cited by 4 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This brilliant performance mainly thanks to the weak rectangle supervision that we interactively obtain, which has more instructional information than a completely unsupervised approach. Owing to the full-duplex strategy, the very recent method FSNet [10] obtains the state-of-the-art performance of , which is lower than the proposed Rect-VOS by a margin of , which demonstrates the effectiveness of the user interaction proposed by the method. Also, the learned EMD and cross-squeeze based temporal modulation plays significant role in the object representations, especially in the fast-moving video scenarios.…”

Section: 5%mentioning

confidence: 96%

“…al. [10 ] studied a new network architecture to combine the features of appearance and movement. Wang et.…”

Section: Interactive Vos Uvos and Svosmentioning

confidence: 99%

“…On the other hand, unsupervised (also namely zero-shot) VOS, given no initial guidance, the model is easy to misunderstand user's intention and obtains inaccurate initial mask, yielding inferior subsequent VOS results (see the first two rows in Fig. 1 [10]). As illustrated in Fig.…”

Section: Introductionmentioning

confidence: 99%

“…Method LSMO AGS AnDiff[51] COSNet[52] MATNet[46] WCS[53] DFNet[54] F2Net[50] RTNet[3] FSNet[10] RectVOS(ours)…”

unclassified

See 3 more Smart Citations

Unsupervised Video Object Segmentation via Weak User Interaction and Temporal Modulation

Fan

Zhang

Zhao³

et al. 2023

Chinese J. Elect.

View full text Add to dashboard Cite

In unsupervised video object segmentation (UVOS), the whole video might segment the wrong target due to the lack of initial prior information. Also, in semi-supervised video object segmentation (SVOS), the initial video frame with a fine-grained pixel-level mask is essential to good segmentation accuracy. It is expensive and laborious to provide the accurate pixel-level masks for each training sequence. To address this issue, We present a weak user interactive UVOS approach guided by a simple human-made rectangle annotation in the initial frame. We first interactively draw the region of interest by a rectangle, and then we leverage the mask RCNN (region-based convolutional neural networks) method to generate a set of coarse reference labels for subsequent mask propagations. To establish the temporal correspondence between the coherent frames, we further design two novel temporal modulation modules to enhance the target representations. We compute the earth mover's distance (EMD)-based similarity between coherent frames to mine the co-occurrent objects in the two images, which is used to modulate the target representation to highlight the foreground target. We design a cross-squeeze temporal modulation module to emphasize the co-occurrent features across frames, which further helps to enhance the foreground target representation. We augment the temporally modulated representations with the original representation and obtain the compositive spatio-temporal information, producing a more accurate video object segmentation (VOS) model. The experimental results on both UVOS and SVOS datasets including Davis2016, FBMS, Youtube-VOS, and Davis2017, show that our method yields favorable accuracy and complexity. The related code is available.

show abstract

Section: 5%mentioning

confidence: 96%

“…al. [10 ] studied a new network architecture to combine the features of appearance and movement. Wang et.…”

Section: Interactive Vos Uvos and Svosmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…Method LSMO AGS AnDiff[51] COSNet[52] MATNet[46] WCS[53] DFNet[54] F2Net[50] RTNet[3] FSNet[10] RectVOS(ours)…”

unclassified

See 2 more Smart Citations

Unsupervised Video Object Segmentation via Weak User Interaction and Temporal Modulation

Fan

Zhang

Zhao³

et al. 2023

Chinese J. Elect.

View full text Add to dashboard Cite

show abstract

“…Pre-computed frame-wise features using optical flow have much cheaper computational costs compared to learning models (learning video features) [18], [19]. A method was introduced for the optical flow-based still image to video segmentation models, resulting in increased stability [20]. Optical flow has emerged as an effective tool for motion analysis in video processing.…”

Section: Introductionmentioning

confidence: 99%

Temporal Action Segmentation in Sign Language System for Bahasa Indonesia (SIBI) Videos Using Optical Flow-Based Approach

I Dewa Made Bayu Atmaja Darmawan,

Linawati,

Sukadarmika

et al. 2024

Jurnal Ilmu Komputer dan Informasi

View full text Add to dashboard Cite

Sign language (SL) is vital in fostering communication for the deaf and hard-of-hearing communities. Continuous Sign Language Translation (CSLT) is a work that translates sign language into spoken language. CSLT translation is done by changing continuous forms into isolated signs. Segmenting morpheme signs from phrase signs has several challenges, such as the availability of annotated datasets and the complexity of continuous gesture movements. The Indonesian Sign Language (SIBI) system follows Indonesian grammatical norms, including word formation, in contrast to other sign languages with rules derived from their spoken language. In SIBI, a word can consist of a root word and an affix word. Therefore, temporal action segmentation in SIBI is important to reconstruct the results of translating each sign into spoken Indonesian sentences. This research uses an optical flow approach to segment temporal actions in SIBI videos. Optical flow methods that calculate changes in intensity between adjacent frames can be used to determine the occurrence of sign movement or vice versa to determine the delay between sign movements. The absence of intensity differences between the two frames indicates the boundary between sign gestures. This study tested the use of dense optical flow on videos containing SIBI sentences taken from 3 signers. Evaluation is done on several parameters in the dense optical flow algorithm, such as threshold size, PyrScale, and WinSize, to obtain the best accuracy. This paper shows that the optical flow algorithm successfully performs segmentation, as measured by Perf and F1r. The experimental results showed that the highest Perf and F1r yields were 0.8298 and 0.8524, respectively.

show abstract