The lower spectral radius, or joint spectral subradius, of a set of real d × d matrices is defined to be the smallest possible exponential growth rate of long products of matrices drawn from that set. The lower spectral radius arises naturally in connection with a number of topics including combinatorics on words, the stability of linear inclusions in control theory, and the study of random Cantor sets. In this article we apply some ideas originating in the study of dominated splittings of linear cocycles over a dynamical system to characterise the points of continuity of the lower spectral radius on the set of all compact sets of invertible d × d matrices. As an application we exhibit open sets of pairs of 2 × 2 matrices within which the analogue of the Lagarias-Wang finiteness property for the lower spectral radius fails on a residual set, and discuss some implications of this result for the computation of the lower spectral radius.