Bounds for the Distance to the Nearest Correlation Matrix

Abstract

Abstract. In a wide range of practical problems correlation matrices are formed in such a way that, while symmetry and a unit diagonal are ensured, they may lack semidefiniteness. We derive a variety of new upper bounds for the distance from an arbitrary symmetric matrix to the nearest correlation matrix. The bounds are of two main classes: those based on the eigensystem and those based on a modified Cholesky factorization. Bounds from both classes have a computational cost of O(n 3 ) flops for a matrix of order n but are much less expensive to evaluate than the nearest correlation matrix itself. For unit diagonal A with |a ij | ≤ 1 for all i = j the eigensystem bounds are shown to overestimate the distance by a factor at most 1 + n √ n. We show that for a collection of matrices from the literature and from practical applications the eigensystem-based bounds are often good order of magnitude estimates of the actual distance; indeed the best upper bound is never more than a factor 5 larger than a related lower bound. The modified Cholesky bounds are less sharp but also less expensive, and they provide an efficient way to test for definiteness of the putative correlation matrix. Both classes of bounds enable a user to identify an invalid correlation matrix relatively cheaply and to decide whether to revisit its construction or to compute a replacement, such as the nearest correlation matrix.

Anymatrix: an extensible MATLAB matrix collection

“…• https://github.com/higham/matrices-correlation-invalid: invalid correlation matrices collected by Higham and Strabić [36,37]. These are matrices that are intended to be correlation matrices but for various reasons relating to their construction have a negative eigenvalue and so are not positive semidefinite.…”

Simulation-based capital models: testing, justifying and communicating choices. A report from the life aggregation and simulation techniques working party

“…These techniques can be modified to apply weights to particular columns and rows, which may be useful if one has strong views that those assumptions are appropriate, or constrain an existing PSD sub-matrix to remain unchanged. See for example, Higham (2013).…”

On the R-superlinear convergence of the KKT residues generated by the augmented Lagrangian method for convex composite conic programming

“…We can apply the ALM in (28) to solve (31) with the subproblems being solved by the semismooth Newton-CG method. One nice feature of the nearest correlation matrix problem is that the constraints in (30) are always nondegenerate, making the semismooth Newton-CG method converging at least Q-superlinearly [43].…”

“…In our numerical experiments, we take the matrix G to be the indefinite symmetric matrices constructed from stock data by the investment company Orbis [30] (one with the matrix dimensions 1399 1399 and the other with dimensions 3210 ˆ3210), which are available at https://github.com/ higham/matrices-correlation-invalid. We randomly set some entries H i j " 0 (the corresponding G i j are thus treated as "missing" from the the observations) and the other entries H i j " 1.…”

“…Fig. 3 The KKT residues of the ALM for solving the dual problem (31) of the H-weighted nearest correlation matrix problem (30) with missing data.…”