From data to functa: Your data point is a function and you can treat it like one

Abstract

It is common practice in deep learning to represent a measurement of the world on a discrete grid, e.g. a 2D grid of pixels. However, the underlying signal represented by these measurements is often continuous, e.g. the scene depicted in an image. A powerful continuous alternative is then to represent these measurements using an implicit neural representation, a neural function trained to output the appropriate measurement value for any input spatial location. In this paper, we take this idea to its next level: what would it take to perform deep learning on these functions instead, treating them as data? In this context we refer to the data as functa, and propose a framework for deep learning on functa. This view presents a number of challenges around efficient conversion from data to functa, compact representation of functa, and effectively solving downstream tasks on functa. We outline a recipe to overcome these challenges and apply it to a wide range of data modalities including images, 3D shapes, neural radiance fields (NeRF) and data on manifolds. We demonstrate that this approach has various compelling properties across data modalities, in particular on the canonical tasks of generative modeling, data imputation, novel view synthesis and classification.

Meta-Learning Sparse Compression Networks

“…Of particular importance to the remainder of our discussion around compression is the recent observation that signals can be accurately learned using merely data-item specific modulations to a shared base network (Perez et al, 2018;Mehta et al, 2021;Dupont et al, 2022a). Specifically, in the forward pass of a network, each layer l represents the transformation x → f (W (l) x + b (l) + m (l) ), where {W (l) , b (l) } are weights and biases shared between signal with only modulations m (l) being specific to each signal.…”

“…It is worth pointing out that the minimisation of Equation ( 1) is extraordinarily expensive: Learning a single NeRF ( ) scene can take up to an entire day on a single GPU (Dupont et al, 2022a); even the compression of a single low-dimensional image requires thousands of iterative optimisation steps. Fortunately, we need not resort to tabula-rasa optimisation for each data item in turn: In recent years, developments in Meta-Learning (e.g.…”

“…This has interesting consequences: While we previously relied solely on the inner-loop adaptation procedure to "pick out" an appropriate network for each signal, we have now in effect reserved a sub-network that provides a particularly suitable initialisation for the set of signals at hand. This sub-network can either be taken to be fixed (such as when the adopted networks are used to learn a prior or generative model (Dupont et al, 2022a)) or adopted within an acceptable budget of inner steps, providing a mostly overlapping but yet signal-specific set of gates.…”

“…The recently proposed Functa (Dupont et al, 2022a) is a key baseline in our work as it also works on the insights of the modulation-based approach. Rather than using sparsity, the authors introduce a second network which maps a low dimensional latent vector to the full set of modulations.…”

“…Secondly, in order to tackle the computational cost of learning INRs, we follow recent work (e.g. Lee et al, 2021;Tancik et al, 2021;Dupont et al, 2022a) by adopting Meta-Learning techniques such as MAML (Finn et al, 2017) which allow learning an INR representing a single signal by finetuning from a learned initialisation using only a handful of optimisation steps. Crucially, our sparsity procedure allow efficient Backpropagation through this learning procedure, resulting in an initialisation that is specifically optimised for sparse signals.…”