Segmenting Sky Pixels in Images: Analysis and Comparison

Abstract

Outdoor scene parsing models are often trained on ideal datasets and produce quality results. However, this leads to a discrepancy when applied to the real world. The quality of scene parsing, particularly sky classification, decreases in night time images, images involving varying weather conditions, and scene changes due to seasonal weather. This project focuses on approaching these challenges by using a state-of-the-art model in conjunction with non-ideal datasets: SkyFinder and a subset from the SUN database containing the Sky object. We focus specifically on sky segmentation, the task of determining sky and not-sky pixels, and improving upon an existing state-of-the-art model: Re-fineNet. As a result of our efforts, we have seen an improvement of 10-15% in the average MCR compared to the prior methods on the SkyFinder dataset. We have also improved from an off-the-shelf model in terms of average mIOU by nearly 35%. Further, we analyze our trained models on images w.r.t two aspects: times of day and weather, and find that in spite of facing the same challenges as prior methods, our trained models significantly outperform them.

Toward Texturing for Immersive Modeling of Environment Reconstructed from 360 Multi-Camera

“…Alternative methods are possible by following the current trend of deep learning. However such methods have drawbacks: the increase of complexity, the choice of a training dataset, and time consistency assumptions (single image [12] or video [6] queries) which are not ours (a series of KFs).…”

Clear Skies Ahead: Towards Real‐Time Automatic Sky Replacement in Video

“…The location for shooting a video may be chosen carefully, yet the sky which often covers a large portion of the frame is subject to uncontrollable weather and lighting conditions. To fix this, methods for sky segmentation and replacement in still images have been studied [TSL * 16, LUB17]. We build upon these works and extend them to video.…”

Sky Detection in Hazy Image

“…To make up the shortcoming that hand-engineered features adapt poorly to the variational appearance of the sky, two more powerful sky labeling models [ 15 , 16 ], based on deep neural networks, were designed and tested on the SkyFinder dataset. Mihail et al trained an rCNNmodel by adding the output of three baseline methods [ 6 , 8 , 9 ] to the training set and achieved a lower MCR (MisClassificationRate) than the three baseline methods.…”

“…Mihail et al trained an rCNNmodel by adding the output of three baseline methods [ 6 , 8 , 9 ] to the training set and achieved a lower MCR (MisClassificationRate) than the three baseline methods. Place et al [ 16 ] trained a RefineNet model on the SkyFinder dataset and achieved a lower MCR than Mihail et al across their own testing split.…”

“…In contrast, traditional methods (hand-engineered features + classifier) are more flexible, less computing resource consuming, have easier convergence when training and fewer sample requirements, therefore being more suitable for practical applications (e.g., the embedded system application). Place et al’s [ 16 ] experiments also demonstrated the fact that off-the-shelf models, even trained on a huge dataset, would produce poor results when the application environment changes. Therefore, existing methods are still effective so long as they have been modified to suit the task’s need.…”