Network Bending: Expressive Manipulation of Generative Models in Multiple Domains

Abstract

This paper presents the network bending framework, a new approach for manipulating and interacting with deep generative models. We present a comprehensive set of deterministic transformations that can be inserted as distinct layers into the computational graph of a trained generative neural network and applied during inference. In addition, we present a novel algorithm for analysing the deep generative model and clustering features based on their spatial activation maps. This allows features to be grouped together based on spatial similarity in an unsupervised fashion. This results in the meaningful manipulation of sets of features that correspond to the generation of a broad array of semantically significant features of the generated results. We outline this framework, demonstrating our results on deep generative models for both image and audio domains. We show how it allows for the direct manipulation of semantically meaningful aspects of the generative process as well as allowing for a broad range of expressive outcomes.

El error como recurso visual y conceptual en la creación artística con inteligencias artificiales

“…Sin embargo, son precisamente estos errores los que crean un contrapunto con la aparente veracidad de la obra. La exploración del antropomorfismo en el arte de la IA a menudo implica la extraña apariencia de entidades artificiales, siendo lo siniestro la experiencia de percibir un objeto o evento familiar como inquietante, espeluznante o tabú (Broad et al, 2020).…”

PandA: Unsupervised Learning of Parts and Appearances in the Feature Maps of GANs

“…However, whilst several methods identify ways of manipulating the latent space of GANs to bring about global semantic changes-either in a supervised [16,44,48,47] or unsupervised [53,49,20,52,42] manner-many of them struggle to apply local changes to regions of interest in the image. In this framework of local image editing, one can swap certain parts between images [11,24,8,50,32,2], or modify the style at particular regions [54,55,5,63,62,39,26]. This is achieved with techniques such as clustering [11,62,5,26], manipulating the AdaIN [23] parameters [55,54], or/and operating on the feature maps themselves [54,5,62] to aid the locality of the edit.…”

“…In this framework of local image editing, one can swap certain parts between images [11,24,8,50,32,2], or modify the style at particular regions [54,55,5,63,62,39,26]. This is achieved with techniques such as clustering [11,62,5,26], manipulating the AdaIN [23] parameters [55,54], or/and operating on the feature maps themselves [54,5,62] to aid the locality of the edit. Other approaches employ additional latent spaces or architectures [32,39], require the computation of expensive gradient maps [54,55] and semantic segmentation masks/networks [55,64,39], or require manually specified regions of interest [63].…”

“…One important question in this line of research is how latent representations are combined to form the appearance at a particular local region of the image. Whilst some recent methods attempt to tackle this problem [54,55,5,63,62,39,26], the current state-of-the-art methods come with a number of important drawbacks and limitations. In particular, existing techniques require prohibitively long training times [55,63], costly Jacobian-based optimization [63], and the requirement of semantic masks [55] or manually specified regions of interest [63].…”