$$N^4$$-Fields: Neural Network Nearest Neighbor Fields for Image Transforms

Abstract

We propose a new architecture for difficult image processing operations, such as natural edge detection or thin object segmentation. The architecture is based on a simple combination of convolutional neural networks with the nearest neighbor search. We focus our attention on the situations when the desired image transformation is too hard for a neural network to learn explicitly. We show that in such situations, the use of the nearest neighbor search on top of the network output allows to improve the results considerably and to account for the underfitting effect during the neural network training. The approach is validated on three challenging benchmarks, where the performance of the proposed architecture matches or exceeds the state-of-the-art.

Richer Convolutional Features for Edge Detection

“…Ganin et al [20] proposed N 4 -Fields that combines CNNs with the nearest neighbor search. Shen et al [49] partitioned contour data into subclasses and fit each subclass by learning model parameters.…”

Contour detection from deep patch-level boundary prediction

“…They encode an input image using the generic dictionaries and then reconstruct using the transfer function. N4 fields [9] rely on dictionary learning and the use of the Nearest Neighbor algorithm within a CNN framework. Both [16] and [9] predict the boundaries of a whole patch.…”

“…N4 fields [9] rely on dictionary learning and the use of the Nearest Neighbor algorithm within a CNN framework. Both [16] and [9] predict the boundaries of a whole patch. Kivinen et al uses a two-stream CNN architecture to compute edges at each pixel given its surrounding patch.…”

“…As shown in [14,7], edges in a local patch are actually highly interdependent. Recently, [7,16,9] proposed to predict patches' structured labels. Such methods require the learning of edge dictionary.…”

Holistically-Nested Edge Detection

“…HED shows a clear advantage in consistency over Canny. ing, one may categorize works into a few groups such as I: early pioneering methods like the Sobel detector [20], zerocrossing [27,37], and the widely adopted Canny detector [4]; methods driven by II: information theory on top of features arrived at through careful manual design, such as Statistical Edges [22], Pb [28], and gPb [1]; and III: learningbased methods that remain reliant on features of human design, such as BEL [5], Multi-scale [30], Sketch Tokens [24], and Structured Edges [6]. In addition, there has been a recent wave of development using Convolutional Neural Networks that emphasize the importance of automatic hierarchical feature learning, including N 4 -Fields [10], DeepContour [34], DeepEdge [2], and CSCNN [19]. Prior to this explosive development in deep learning, the Structured Edges method (typically abbreviated SE) [6] emerged as one of the most celebrated systems for edge detection, thanks to its state-of-the-art performance on the BSD500 dataset [28] (with, e.g., F-score of .746) and its practically significant speed of 2.5 frames per second.…”