Near-Optimal Procedures for Model Discrimination with Non-Disclosure Properties

Abstract

Let θ 0 , θ 1 ∈ R d be the population risk minimizers associated to some loss : R d × Z → R and two distributions P 0 , P 1 on Z. Our work is motivated by the following question: Given i.i.d. samples from P 0 and P 1 , what sample sizes are sufficient and necessary to distinguish between the two hypotheses θ * = θ 0 and θ * = θ 1 for given θ * ∈ {θ 0 , θ 1 }?Making the first steps towards answering this question in full generality, we first consider the case of a well-specified linear model with squared loss. Here we provide matching upper and lower bounds on the sample complexity, showing it to be min{1/∆ 2 , √ r/∆} up to a constant factor, where ∆ is a measure of separation between P 0 and P 1 , and r is the rank of the design covariance matrix. This bound is dimension-independent, and rank-independent for large enough separation. We then extend this result in two directions: (i) for the general parametric setup in asymptotic regime; (ii) for generalized linear models in the small-sample regime n r and under weak moment assumptions. In both cases, we derive sample complexity bounds of a similar form, even under misspecification. In fact, our testing procedures only access θ * through a certain functional of empirical risk. In addition, the number of observations that allows to reach statistical confidence in our tests does not allow to "resolve" the two models -that is, recover θ 0 , θ 1 up to O(∆) prediction accuracy. These two properties allow to use our framework in applied tasks where one would like to identify a prediction model, which can be proprietary, while guaranteeing that the model cannot be actually inferred by the agent performing identification. * Equal contribution of the first two authors. 1 For this expository discussion, we define the sample complexity of a (binary) testing problem as the size of an i.i.d. sample for which there exists a test with testing errors of both types at most 0.05.

Nonasymptotic one-and two-sample tests in high dimension with unknown covariance structure

“…Since this model has bounded data, our analysis could be applied in that setting, however it concerns separation in ℓ 2 distance (the separation in ℓ 1 distance exhibits considerably more involved behavior). Ostrovskii et al (2020) consider a different type of two-sample testing problem, in a regression context, where the goal is to determine which one of the two distributions has a given (known to the user) regression vector. They give a sharp bound on the minimum separation distance between the two regression vectors including the role of the dimension, also exhibiting a difference with estimation rates.…”