Theory and Algorithms for Weighted Total Least-Squares Fitting of Lines, Planes, and Parallel Planes to Support Tolerancing Standards

Abstract

We present the theory and algorithms for fitting a line, a plane, two parallel planes (corresponding to a slot or a slab), or many parallel planes in a total (orthogonal) least-squares sense to coordinate data that is weighted. Each of these problems is reduced to a simple 3 × 3 matrix eigenvalue/eigenvector problem or an equivalent singular value decomposition problem, which can be solved using reliable and readily available commercial software. These methods were numerically verified by comparing them with brute-force minimization searches. We demonstrate the need for such weighted total least-squares fitting in coordinate metrology to support new and emerging tolerancing standards, for instance, ISO 14405-1:2010. The widespread practice of unweighted fitting works well enough when point sampling is controlled and can be made uniform (e.g., using a discrete point contact coordinate measuring machine). However, we show by example that nonuniformly sampled points (arising from many new measurement technologies) coupled with unweighted least-squares fitting can lead to erroneous results. When needed, the algorithms presented also solve the unweighted cases simply by assigning the value one to each weight. We additionally prove convergence from the discrete to continuous cases of least-squares fitting as the point sampling becomes dense.

Datum Planes Based on a Constrained L1 Norm

“…But for completeness, we define * as the direction normal to the least-squares plane of the original surface and pointing into (as opposed to outside) the material. See [11] for an appropriate least-squares algorithm, if needed.…”

“…A check can be added for the case that the points form a perfect plane. This check (which is not needed for the 2D case) can be performed by simply looking at the residuals from a least-squares plane fit as described in [11]. If the residuals are essentially zero, then the least-squares plane is the datum plane.…”

“…3) 2 unconstrained: Find the plane fit to the planar feature in a least-squares sense (see [11]). 4) 2 shifted: Find the plane as in (3) above, but translate the plane such that it contacts the datum feature but does not pass through any material.…”

“…See Table 5. [8,11]). An algorithm that generates its own weights typically would need part information (though there are workarounds with varying success depending on the sampling density).…”

“…An algorithm that generates its own weights typically would need part information (though there are workarounds with varying success depending on the sampling density). The references [8,11] show why the 1 constrained and all the 2 based definitions need weighting information. Since the ∞ fits are determined by a few extreme points, the effect of the other points do not alter the fit, making weighting not important.…”