Explicit Correlation Amplifiers for Finding Outlier Correlations in Deterministic Subquadratic Time

Abstract

We derandomize Valiant’s (J ACM 62, Article 13, 2015) subquadratic-time algorithm for finding outlier correlations in binary data. This demonstrates that it is possible to perform a deterministic subquadratic-time similarity join of high dimensionality. Our derandomized algorithm gives deterministic subquadratic scaling essentially for the same parameter range as Valiant’s randomized algorithm, but the precise constants we save over quadratic scaling are more modest. Our main technical tool for derandomization is an explicit family of correlation amplifiers built via a family of zigzag-product expanders by Reingold et al. (Ann Math 155(1):157–187, 2002). We say that a function $$f:\{-1,1\}^d\rightarrow \{-1,1\}^D$$ f : { - 1 , 1 } d → { - 1 , 1 } D is a correlation amplifier with threshold $$0\le \tau \le 1$$ 0 ≤ τ ≤ 1 , error $$\gamma \ge 1$$ γ ≥ 1 , and strength p an even positive integer if for all pairs of vectors $$x,y\in \{-1,1\}^d$$ x , y ∈ { - 1 , 1 } d it holds that (i) $$|\langle x,y\rangle |<\tau d$$ | ⟨ x , y ⟩ | < τ d implies $$|\langle f(x),f(y)\rangle |\le (\tau \gamma )^pD$$ | ⟨ f ( x ) , f ( y ) ⟩ | ≤ ( τ γ ) p D ; and (ii) $$|\langle x,y\rangle |\ge \tau d$$ | ⟨ x , y ⟩ | ≥ τ d implies $$\left (\frac{\langle x,y\rangle }{\gamma d}\right )^pD \le \langle f(x),f(y)\rangle \le \left (\frac{\gamma \langle x,y\rangle }{d}\right )^pD$$ ⟨ x , y ⟩ γ d p D ≤ ⟨ f ( x ) , f ( y ) ⟩ ≤ γ ⟨ x , y ⟩ d p D .

An Illuminating Algorithm for the Light Bulb Problem

“…Our runtime of O(n 2ω/3+ε ) ≤ O(n 1.582 ) is faster than the runtime of Karppa et al [KKKÓC16], which is at best O(n 1.996 ). Our algorithm also uses more straightforward and elementary techniques.…”

“…Our algorithm combines techniques from past work on subquadratic algorithms for the Light Bulb Problem [Val15,KKK16,KKKÓC16] with techniques for batch nearest neighbor algorithms using the 'polynomial method in algorithm design' [AW15,ACW16]. Our algorithm begins in a way common to both methods: we partition S into m = n 2/3 groups S = S 1 ∪ • • • ∪ S m .…”

“…We differ from past work in how the A i and B i vectors are constructed. In [Val15,KKK16], a sophisticated random sampling technique is used, including an involved probabilistic analysis to keep t low, and in [KKKÓC16], that technique is derandomized. We instead use the polynomial method: we first design a polynomial (see (3) below) which, it is not hard to show, has the desired properties.…”

“…One option is to design an algorithm which correctly solves almost all instances. This is the notion which was introduced and used in the past work by Karppa et al [KKKÓC16] on deterministic algorithms for the Light Bulb Problem. We say that an algorithm is correct on almost all instances if the probability of drawing an instance where the algorithm fails is 1/poly(n).…”

“…Although the algorithm of [KKKÓC16] is presented as working on almost all instances, it implicitly works in a stronger regime. It solves a promise version of the Light Bulb Problem, which we introduce here, in which we are guaranteed that no pair of random vectors is too correlated: Problem 1.2 (Promise Light Bulb Problem with parameter w).…”