Large-Scale Coarse-to-Fine Object Retrieval Ontology and Deep Local Multitask Learning

Abstract

Object retrieval plays an increasingly important role in video surveillance, digital marketing, e-commerce, etc. It is facing challenges such as large-scale datasets, imbalanced data, viewpoint, cluster background, and fine-grained details (attributes). This paper has proposed a model to integrate object ontology, a local multitask deep neural network (local MDNN), and an imbalanced data solver to take advantages and overcome the shortcomings of deep learning network models to improve the performance of the large-scale object retrieval system from the coarse-grained level (categories) to the fine-grained level (attributes). Our proposed coarse-to-fine object retrieval (CFOR) system can be robust and resistant to the challenges listed above. To the best of our knowledge, the new main point of our CFOR system is the power of mutual support of object ontology, a local MDNN, and an imbalanced data solver in a unified system. Object ontology supports the exploitation of the inner-group correlations to improve the system performance in category classification, attribute classification, and conducting training flow and retrieval flow to save computational costs in the training stage and retrieval stage on large-scale datasets, respectively. A local MDNN supports linking object ontology to the raw data, and an imbalanced data solver based on Matthews’ correlation coefficient (MCC) addresses that the imbalance of data has contributed effectively to increasing the quality of object ontology realization without adjusting network architecture and data augmentation. In order to evaluate the performance of the CFOR system, we experimented on the DeepFashion dataset. This paper has shown that our local MDNN framework based on the pretrained NASNet architecture has achieved better performance (14.2% higher in recall rate) compared to single-task learning (STL) in the attribute learning task; it has also shown that our model with an imbalanced data solver has achieved better performance (5.14% higher in recall rate for fewer data attributes) compared to models that do not take this into account. Moreover, MAP@30 hovers 0.815 in retrieval on an average of 35 imbalanced fashion attributes.

An Overview of Facial Attribute Learning

“…The attributes will be predicted more accurately, the predicted attributes do not tend to favor the majority attribute group. Some works performed Imbalanced Data Solver on fashion image (Ly et al, 2019), on face recognition (P. . In the FAL, typical works are MOON (Rudd et al, 2016), LMLE-KNN (Loy et al, 2017), CRL (Dong et al, 2017), Selective Learning (Hand et al, 2018), CLMLE (Huang et al, 2018), and DCL .…”

“…Finally, using ontology in AL to support semantic of images is also common such as in a study by Nguyen et al (2018), Ly et al (2019), Ly et al (2020).…”

Person Re-Identification System at Semantic Level based on Pedestrian Attributes Ontology

“…Our PAO is inspired by our Face Attribute Ontology (FaAO) and our Fashion Attribute Ontology (FasAO) [39]. PAO helps us to exploit inner group and inter group correlations between attributes, it is then very useful for training the Local MDCNN.…”

“…Therefore, our Local MDCNN learns attributes from local features instead of global ones. Follow [39], we employ a clothing attribute dataset named DeepFashion [40] to build a general FasAO, and our prior knowledge to build a general FaAO. In experiment, we re-build specific PAO on another dataset that has both Re-ID and attribute label.…”

Discriminative Codebook Hashing for Supervised Video Retrieval

“…YouTube receives a total of 100 hours of videos per minute [4][5][6]. Due to the economic storage and efficiency of binary codes, hash-based methods have been widely applied to visual retrieval tasks [7][8][9][10][11][12][13].…”