An analysis of the effect of rounding errors on the flow of control in numerical processes

Abstract

Abstract.Rounding errors may change the flow of control in numerical computing processes by leading to changes in some branching decisions of the process. In this paper some general topological and measure theoretical results associated with this effect of rounding errors are derived. The approach is based on a model of numerical computation related to program schemes. Each computing process specified by the model computes .a partial function R~ R" using rational operations and simple tests on real numbers. The topological structure of input point sets in R n on which the computation follows the same execution path is studied. We also investigate input points, called sensitive, on which rounding errors may change the execution path followed. Conditions concerning computing processes are given which guarantee that the Lebesgue measure of sensitive points approaches zero (i.e. the probability of a branching error gets arbitrarily small) as the precision of the arithmetic increases. Most numerical processes used in practice are easily seen to satisfy these conditions.