Wildlife recognition in nature documentaries with weak supervision from subtitles and external data

Venkitasubramanian, Aparna Nurani; Tuytelaars, Tinne; Moens, Marie‐Francine

doi:10.1016/j.patrec.2016.01.025

Cited by 7 publications

(29 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The problem of aligning animals from videos with their mentions in subtitles has been studied in (Dusart et al, 2013) and (Venkitasubramanian et al, 2016). The former relies on hand-annotated bounding boxes to localize the animals in a frame, which are difficult to acquire.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning to Recognize Animals by Watching Documentaries: Using Subtitles as Weak Supervision

Venkitasubramanian

Tuytelaars²,

Moens³

2017

Proceedings of the Sixth Workshop on Vision and Language

Self Cite

View full text Add to dashboard Cite

We investigate animal recognition models learned from wildlife video documentaries by using the weak supervision of the textual subtitles. This is a challenging setting, since i) the animals occur in their natural habitat and are often largely occluded and ii) subtitles are to a great degree complementary to the visual content, providing a very weak supervisory signal. This is in contrast to most work on integrated vision and language in the literature, where textual descriptions are tightly linked to the image content, and often generated in a curated fashion for the task at hand. We investigate different image representations and models, in particular a support vector machine on top of activations of a pretrained convolutional neural network, as well as a Naive Bayes framework on a 'bag-of-activations' image representation, where each element of the bag is considered separately. This representation allows key components in the image to be isolated, in spite of vastly varying backgrounds and image clutter, without an object detection or image segmentation step. The methods are evaluated based on how well they transfer to unseen camera-trap images captured across diverse topographical regions under different environmental conditions and illumination settings, involving a large domain shift.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Discretization can also be applied to the global representation used by the SVM, but as shown in(Venkitasubramanian et al, 2016), it is particularly useful in conjunction with a naive Bayes classifier.…”

mentioning

confidence: 99%

Learning to Recognize Animals by Watching Documentaries: Using Subtitles as Weak Supervision

Venkitasubramanian

Tuytelaars²,

Moens³

2017

Proceedings of the Sixth Workshop on Vision and Language

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the huge wildlife monitoring images with invalid images included, traditional method such as manual sorting is disadvantageous in terms of high labor intensity, low efficiency and unstable the recognition accuracy. Therefore automatic wildlife recognition [22] is demanded to improve the efficiency of wildlife monitoring. In recent years, various novel models such as AlexNet [23], VGG Net [24], Google Net [25], Deep Residual Net [26] and Dense Net [27] were proposed to improve wildlife recognition capability and achieve the automatic recognition of wildlife accurately.…”

Section: Introductionmentioning

confidence: 99%

High-Efficiency Progressive Transmission and Automatic Recognition of Wildlife Monitoring Images With WISNs

Feng

Ju²,

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Wireless image sensor networks (WISNs) are widely applied in wildlife monitoring, as they present a better performance in remote, real-time monitoring. However, traditional WISNs suffer from the limitations of low processing capability, power consumption restrictions and narrow transmission bandwidth. For the contradiction between the above limitations of WISNs and the wildlife monitoring images with high resolution and complex background, we propose a novel wildlife intelligent monitoring system. On the foundation of saliency object detection, the convolutional encoder-decoder network is utilized to realize the progressive compression transmission and restoration for wildlife monitoring images, which guarantees the transmission efficiency and quality of wildlife part. Moreover, to deal with the problems of high labor intensity, low efficiency and low recognition accuracy in classical manual sorting method, an improved Faster RCNN algorithm is proposed on the automatic recognition of wildlife images. The experimental results on our own wildlife dataset, show that the peak signal to noise ratio (PSNR) and structural similarity index (SSIM) are improved by 7.93%, 18.15% and 7.01%, 12.67% respectively on reconstruction image, when compared with the set partitioned in hierarchical tree (SPIHT) and embedded zerotree (EZW) algorithms. Compared with the traditional Faster RCNN algorithm, the recognition accuracy of six species wildlife is respectively improved by 1%, 18%, 5%, 17%, 2% and 19%, and the final mAP value reaches to 92.2% in test set increased by 10.9%, which demonstrates the proposed algorithm can ideally achieve the wildlife intelligent monitoring with WISNs.

show abstract

“…Particularly in videos, language and vision are complementary to each other and it is essential to look at them in unison. Vision tasks often benefit from the associated text [42] while Natural Language Processing (NLP) tasks benefit from the vision. As an example, consider Figure 1.…”

Section: Introductionmentioning

confidence: 99%

Entity linking across vision and language

Venkitasubramanian

Tuytelaars

Moens

2017

Multimed Tools Appl

Self Cite

View full text Add to dashboard Cite

We propose a novel weakly supervised framework that jointly tackles entity analysis tasks in vision and language. Given a video with subtitles, we jointly address the questions: a) What do the textual entity mentions refer to? and b) What/ who are in the video key frames? We use a Markov Random Field (MRF) to encode the dependencies within and across the two modalities. This MRF model incorporates beliefs using independent methods for the textual and visual entities. These beliefs are propagated across the modalities to jointly derive the entity labels. We apply the framework to a challenging dataset of wildlife documentaries with subtitles and show that this integrated modelling yields significantly better performance over text-based and vision-based approaches. We show that textual mentions that cannot be resolved using text-only methods are resolved correctly using our method. The approaches described here bring us closer to automated multimedia indexing.

show abstract

Wildlife recognition in nature documentaries with weak supervision from subtitles and external data

Cited by 7 publications

References 25 publications

Learning to Recognize Animals by Watching Documentaries: Using Subtitles as Weak Supervision

Learning to Recognize Animals by Watching Documentaries: Using Subtitles as Weak Supervision

High-Efficiency Progressive Transmission and Automatic Recognition of Wildlife Monitoring Images With WISNs

Entity linking across vision and language

Contact Info

Product

Resources

About