Learning Diverse Image Colorization

Abstract

Colorization is an ambiguous problem, with multiple viable colorizations for a single grey-level image. However, previous methods only produce the single most probable colorization. Our goal is to model the diversity intrinsic to the problem of colorization and produce multiple colorizations that display long-scale spatial co-ordination. We learn a low dimensional embedding of color fields using a variational autoencoder (VAE). We construct loss terms for the VAE decoder that avoid blurry outputs and take into account the uneven distribution of pixel colors. Finally, we build a conditional model for the multi-modal distribution between grey-level image and the color field embeddings. Samples from this conditional model result in diverse colorization. We demonstrate that our method obtains better diverse colorizations than a standard conditional variational autoencoder (CVAE) model, as well as a recently proposed conditional generative adversarial network (cGAN).

A Learned Representation of Artist-Specific Colourisation

“…The loss at each pixel is re-weighted based on a weighting factor determined by the rarity of the target colour. This approach prevents the loss function from being dominated by highly common colours and is similar to the approach described in [3].…”

“…Recent work on fully automatic image colourisation has shown that Convolutional Neural Networks (CNN) are capable of producing visually appealing colourisation results [15,28,3,8,10]. CNN-based fully automatic approaches can be categorised into two groups: (1) per-pixel descriptor approaches [2,15] and (2) encoder-decoder type architectures [8,28,3,10].…”

“…CNN-based fully automatic approaches can be categorised into two groups: (1) per-pixel descriptor approaches [2,15] and (2) encoder-decoder type architectures [8,28,3,10]. The per-pixel descriptor approach consists of passing the input image through a (pretrained) CNN and extracting a hypercolumn descriptor [7] for each pixel.…”

“…Followed by a decoder which either upsamples using interpolation (e.g., nearestneighbour, bilinear, or bicubic), or deconvolution (i.e., fractional strided convolution) [27]. Encoder-decoder architectures are trained in either a Generative Adversial setting [10], or with a pixel-wise loss [8,28,3].…”

“…If we are able to recognise what is depicted, we may be able to suggest a plausible colourisation. Recent work has shown that Convolutional Neural Networks (CNN) can obtain sufficient visual understanding to perform automatic image colourisation [15,28,3,8,10]. Depending on the type of image other factors than the image semantics might play a role in determining the likelihood of colours.…”