A Simple Fix for Convolutional Neural Network via Coordinate Embedding

Abstract

Convolutional Neural Networks (CNN) has been widely applied in the realm of computer vision. However, given the fact that CNN models are translation invariant, they are not aware of the coordinate information of each pixel. Thus the generalization ability of CNN will be limited since the coordinate information is crucial for a model to learn affine transformations which directly operate on the coordinate of each pixel. In this project, we proposed a simple approach to incorporate the coordinate information to the CNN model through coordinate embedding. Our approach does not change the downstream model architecture and can be easily applied to the pre-trained models for the task like object detection. Our experiments on the German Traffic Sign Detection Benchmark show that our approach not only significantly improve the model performance but also have better robustness with respect to the affine transformation.

A retrospective study of 3D deep learning approach incorporating coordinate information to improve the segmentation of pre- and post-operative abdominal aortic aneurysm

“…Incorporation of coordinate information is an alternative approach that improves performance by providing more information for training. Recently, the incorporation of coordinate information has been proposed to improve object detection in 2D images ( Liu et al, 2018 ; Ren & Hao, 2020 ). Incorporation of coordinate information has the advantage of being able to work with existing CNN models without modification to their architecture.…”

“…However, the performance of these two approaches has nearly similar results ( Vaswani et al, 2017 ). In the recent implementations of image processing, researchers still used a linear function to generate a simple index sequence of coordinate information ( Liu et al, 2018 ; Ren & Hao, 2020 ). The coordinate information tends to be of more benefit on the NCCT dataset than the CECT dataset because the NCCT dataset has less contrast resolution.…”

“…The positional encoding was implemented by a linear function ( Gehring et al, 2017 ) and sinusoidal function ( Vaswani et al, 2017 ). In the image processing domain, the coordinate information encoding was also proposed in recent literature ( Liu et al, 2018 ; Ren & Hao, 2020 ) by adding extra channels in input images. Liu et al (2018) proposed incorporating coordinate information in two extra channels for the x, y axes of 2D images using a continuous sequence of integers starting with zero in row and column.…”

“…They demonstrated an improvement of CNNs in image classification, objection detection and generative models. Ren & Hao (2020) proposed a similar coordinate information embedding in the extra channels of images as the input of a downstream CNN. They demonstrated an improvement of object detection in traffic sign images.…”

“…Recent work on use of CNNs for medical image segmentation has explored various network architectures to improve performance ( Shen, Wu & Suk, 2017 ; Kim et al, 2019 ). With this line of work seeming to have reached a plateau, a promising approach to achieve further improvement is to incorporate additional data, such as coordinate information ( Liu et al, 2018 ; Ren & Hao, 2020 ). Segmentation of AAA is a good candidate for this approach since the AAA is a tubular structure almost always oriented from head to toe.…”