Scene Classification in Indoor Environments for Robots using Context Based Word Embeddings

Abstract

Scene Classification has been addressed with numerous techniques in the computer vision literature. However with the increasing number of scene classes in datasets in the field, it has become difficult to achieve high accuracy in the context of robotics. In this paper, we implement an approach which combines traditional deep learning techniques with natural language processing methods to generate a word embedding based Scene Classification algorithm. We use the key idea that context (objects in the scene) of an image should be representative of the scene label meaning a group of objects could assist to predict the scene class. Objects present in the scene are represented by vectors and the images are re-classified based on the objects present in the scene to refine the initial classification by a Convolutional Neural Network (CNN). In our approach we address indoor Scene Classification task using a model trained with a reduced pre-processed version of the Places365 dataset and an empirical analysis is done on a real world dataset that we built by capturing image sequences using a GoPro camera. We also report results obtained on a subset of the Places365 dataset using our approach and additionally show a deployment of our approach on a robot operating in a real world environment.

Indoor Scene Recognition via Object Detection and TF-IDF

“…Furthermore, the room categories were confined to rooms typically found in residential homes. Chen and colleagues [ 13 ] use the Top-5 predictions and refine their predictions based on word embeddings. Our results are comparable to theirs as they use the same dataset with more room categories represented in their “home” data ( n = 14).…”

“…More recently, techniques have focused on scene classification by leveraging object-level semantic information [ 18 , 19 , 20 , 21 ]. In 2019, Chen and colleagues [ 13 ] investigated scene classification by combining traditional scene classification techniques with NLP methods. Using a convolutional neural network (CNN) module, they generated an ordered top-5 prediction for a given image and segmented the scene using a scene parser module.…”

“…Our approach is closely related to both methods [ 12 , 13 ]. While both incorporate object-level semantic information in scene recognition, our approach performs scene recognition solely through object-level representations.…”

“…Object co-occurrence has been shown to improve classification models when performing object-level scene classification [ 12 ]. Other NLP approaches have also been used to facilitate scene recognition, such as word embeddings [ 13 ].…”

Indoor Scene Recognition via Object Detection and TF-IDF

“…More recently, techniques have focused on scene classification by leveraging object-95 level semantic information [18][19][20][21]. In 2019, Chen and colleagues [13] investigated scene 96 classification by combining traditional scene classification techniques with NLP methods.…”