Web-scale computer vision using MapReduce for multimedia data mining

White, Brandyn; Yeh, Tom; Lin, Jimmy; Davis, Larry S.

doi:10.1145/1814245.1814254

Cited by 70 publications

(35 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As a result, for the same physical storage250 times additional memory space will be required, compared to the previous example. As a matter of fact, there is many fieldsthat produce tremendous numbers of small files continuouslysuch as analysis for multimedia data mining [6], astronomy [7], meteorology [8], signal recognition [9], climatology [10,11], energy, and E learning [12] where numbers of small files are in the ranges of millions to billions. For instance, Facebook has stored more than 260billion images [13].…”

Section: T R a N S A C T I O N S O N M A C H I N E L E A R N I N G A mentioning

confidence: 99%

Optimizing Hadoop for Small File Management

Elmahouti

Achandair

Khoulji

et al. 2017

TMLAI

View full text Add to dashboard Cite

HDFS is one of the most used distributed file systems, that offer a high availability and scalability on low cost hardware. HDFS is delivered as the storage component of Hadoop framework. Coupled with map reduce, which is the processing component, HDFS and MapReduce become the de facto platform for managing big data nowadays. However, HDFS was designed to handle specifically a huge number of large files, while when it comes to a large number of small files, Hadoop deployments may be not efficient. In this paper, we proposed a new strategy to manage small files. Our approach consists of two principal phases. The first phase is about consolidating more than only one client's small files input, and store the inputs continuously in the first allocated block, in a SequenceFile format, and so on into the next blocks. That way we avoid multiple block allocations for different streams, to reduce calls for available blocks and to reduce the metadata memory on the NameNode. This is because groups of small files packaged in a SequenceFile on the same block will require one entry instead of one for each small file. The second phase consists of analyzing attributes of stored small files to distribute them in such a way that the most called files will be referenced by an additional index as a MapFile format to reduce the read throughput during random access.

show abstract

Section: T R a N S A C T I O N S O N M A C H I N E L E A R N I N G A mentioning

confidence: 99%

Optimizing Hadoop for Small File Management

Elmahouti

Achandair

Khoulji

et al. 2017

TMLAI

View full text Add to dashboard Cite

show abstract

“…White et.al [16] presents a case study of classifying and clustering billions of regular images using MapReduce. It describes an image pre-processing technique for use in a sliding-window approach for object recognition.…”

Section: Prior Workmentioning

confidence: 99%

Hadoop Image Processing Framework

Vemula

Crick

2015

2015 IEEE International Congress on Big Data

View full text Add to dashboard Cite

With the rapid growth of social media, the number of images being uploaded to the internet is exploding. Massive quantities of images are shared through multi-platform services such as Snapchat, Instagram, Facebook and WhatsApp; recent studies estimate that over 1.8 billion photos are uploaded every day. However, for the most part, applications that make use of this vast data have yet to emerge. Most current image processing applications, designed for small-scale, local computation, do not scale well to web-sized problems with their large requirements for computational resources and storage. The emergence of processing frameworks such as the Hadoop MapReduce[3] platform addresses the problem of providing a system for computationally intensive data processing and distributed storage. However, to learn the technical complexities of developing useful applications using Hadoop requires a large investment of time and experience on the part of the developer. As such, the pool of researchers and programmers with the varied skills to develop applications that can use large sets of images has been limited. To address this we have developed the Hadoop Image Processing Framework, which provides a Hadoop-based library to support large-scale image processing. The main aim of the framework is to allow developers of image processing applications to leverage the Hadoop MapReduce framework without having to master its technical details and introduce an additional source of complexity and error into their programs.

show abstract

“…The algorithm estimates the computational time of each task a priori, which effectively compresses system idle time. White et al [157] discussed MapReduce implementations of several popular algorithms in computer vision and multimedia problems (e.g., classifier training, clustering, and bag-of-features).…”

Section: Scalability and Efficiencymentioning

confidence: 99%

High-level event recognition in unconstrained videos

Jiang

Bhattacharya

Chang

et al. 2012

Int J Multimed Info Retr

160

108

View full text Add to dashboard Cite

The goal of high-level event recognition is to automatically detect complex high-level events in a given video sequence. This is a difficult task especially when videos are captured under unconstrained conditions by nonprofessionals. Such videos depicting complex events have limited quality control, and therefore, may include severe camera motion, poor lighting, heavy background clutter, and occlusion. However, due to the fast growing popularity of such videos, especially on the Web, solutions to this problem are in high demands and have attracted great interest from researchers. In this paper, we review current technologies for complex event recognition in unconstrained videos. While the existing solutions vary, we identify common key modules and provide detailed descriptions along with some insights for each of them, including extraction and representation of low-level features across different modalities, classification strategies, fusion techniques, etc. Publicly available benchmark datasets, performance metrics, and related research forums are also described. Finally, we discuss promising directions for future research.

show abstract

Web-scale computer vision using MapReduce for multimedia data mining

Cited by 70 publications

References 22 publications

Optimizing Hadoop for Small File Management

Optimizing Hadoop for Small File Management

Hadoop Image Processing Framework

High-level event recognition in unconstrained videos

Contact Info

Product

Resources

About