Self-Supervised Learning of Interpretable Keypoints From Unlabelled Videos

Jakab, Tomas; Gupta, Ankush; Bilen, Hakan; Vedaldi, Andrea

doi:10.1109/cvpr42600.2020.00881

Cited by 77 publications

(80 citation statements)

References 65 publications

(108 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…quences, as unlabeled videos can be acquired without effort while providing rich information. The general idea of using videos for self-supervision is not new [13,28,36,40,63,65]. Here, we consider video sequences to automatically generate masks for the objects visible in their frames.…”

Section: Input Imagementioning

confidence: 99%

Learning to Better Segment Objects from Unseen Classes with Unlabeled Videos

Du¹,

Xiao²,

Lepetit³

2021

Preprint

View full text Add to dashboard Cite

The ability to localize and segment objects from unseen classes would open the door to new applications, such as autonomous object learning in active vision. Nonetheless, improving the performance on unseen classes requires additional training data, while manually annotating the objects of the unseen classes can be labor-extensive and expensive. In this paper, we explore the use of unlabeled video sequences to automatically generate training data for objects of unseen classes. It is in principle possible to apply existing video segmentation methods to unlabeled videos and automatically obtain object masks, which can then be used as a training set even for classes with no manual labels available. However, our experiments show that these methods do not perform well enough for this purpose. We therefore introduce a Bayesian method that is specifically designed to automatically create such a training set: Our method starts from a set of object proposals and relies on (non-realistic) analysis-by-synthesis to select the correct ones by performing an efficient optimization over all the frames simultaneously. Through extensive experiments, we show that our method can generate a high-quality training set which significantly boosts the performance of segmenting objects of unseen classes. We thus believe that our method could open the door for open-world instance segmentation using abundant Internet videos.

show abstract

Section: Input Imagementioning

confidence: 99%

Learning to Better Segment Objects from Unseen Classes with Unlabeled Videos

Du¹,

Xiao²,

Lepetit³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, unlike our method, their method does not contain any temporal constraint, intuitive editing, nor does it demonstrate any result over videos. Jakab et al [16] use unpaired pose prior to train a keypoint extractor, but their work is also limited to humans. Similar to ours, Siarohin et al [34] learn keypoint representations in an unsupervised fashion, however, as we demonstrate (see Section 4), their method cannot handle cross-domain videos well.…”

Section: Shared Geometric Representationmentioning

confidence: 99%

JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting

Mokady,

Tzaban,

Benaim

et al. 2021

Preprint

View full text Add to dashboard Cite

The task of unsupervised motion retargeting in videos has seen substantial advancements through the use of deep neural networks. While early works concentrated on specific object priors such as a human face or body, recent work considered the unsupervised case. When the source and target videos, however, are of different shapes, current methods fail. To alleviate this problem, we introduce JOKRa JOint Keypoint Representation that captures the motion common to both the source and target videos, without requiring any object prior or data collection. By employing a domain confusion term, we enforce the unsupervised keypoint representations of both videos to be indistinguishable. This encourages disentanglement between the parts of the motion that are common to the two domains, and their distinctive appearance and motion, enabling the generation of videos that capture the motion of the one while depicting the style of the other. To enable cases where the objects are of different proportions or orientations, we apply a learned affine transformation between the JOKRs. This augments the representation to be affine invariant, and in practice broadens the variety of possible retargeting pairs. This geometry-driven representation enables further intuitive control, such as temporal coherence and manual editing. Through comprehensive experimentation, we demonstrate the applicability of our method to different challenging cross-domain video pairs. We evaluate our method both qualitatively and quantitatively, and demonstrate that our method handles various cross-domain scenarios, such as different animals, different flowers, and humans. We also demonstrate superior temporal coherency and visual quality compared to state-of-the-art alternatives, through statistical metrics and a user study. Source code and videos can be found at: https://rmokady.github.io/JOKR/.Preprint. Under review.

show abstract

“…These methods implicitly take in equivariant constraints by modeling objects as deformation (or flow). For example, Jakab et al [11] proposed a method for recognizing poses of objects, which is trained by learning auxiliary tasks without manually labeled data, where a network is deployed to reconstruct the original image. Wiles et al [12] developed an approach by leveraging information from multiple source frames and predicting confidence masks for each frame.…”

Section: Related Workmentioning

confidence: 99%

“…To further utilize the pixel-wise information, other video-based methods tries to disentangle pictorial representations from appearances. [11] distills a dual representation of pose from the target image, which reconstructs the objective with the appearance information extracted from the source frame. Though, progress has been achieved in terms of performances, these methods heavily depend on a preliminary that the predicting object must be shown in the video with stable motion.…”

Section: Introductionmentioning

confidence: 99%

Exploiting Invariance of Mining Facial Landmarks

Shi

Gao

Liu

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

In this paper, we propose an invariant learning method for facial landmark mining in a self-supervised manner. The conventional methods mostly train with raw data of paired facial appearances and landmarks, assuming that they are evenly distributed. However, assumptions like this tend to lead to failures in challenging cases even undergo costly training since they usually don't hold in real-world scenarios. To address this issue, our model achieves to be invariant to facial biases by learning through the landmark-anchored distributions. Specifically, we generate faces from these distributions, then group them based on the appearance sources and the probe facial landmarks into intra-identities and intra-landmarks classes, respectively. Thus, we construct intra-class invariance losses to

show abstract

Self-Supervised Learning of Interpretable Keypoints From Unlabelled Videos

Cited by 77 publications

References 65 publications

Learning to Better Segment Objects from Unseen Classes with Unlabeled Videos

Learning to Better Segment Objects from Unseen Classes with Unlabeled Videos

JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting

Exploiting Invariance of Mining Facial Landmarks

Contact Info

Product

Resources

About