Automatically improving accuracy for floating point expressions

Abstract

Scientific and engineering applications depend on floating point arithmetic to approximate real arithmetic. This approximation introduces rounding error, which can accumulate to produce unacceptable results. While the numerical methods literature provides techniques to mitigate rounding error, applying these techniques requires manually rearranging expressions and understanding the finer details of floating point arithmetic.We introduce Herbie, a tool which automatically discovers the rewrites experts perform to improve accuracy. Herbie's heuristic search estimates and localizes rounding error using sampled points (rather than static error analysis), applies a database of rules to generate improvements, takes series expansions, and combines improvements for different input regions. We evaluated Herbie on examples from a classic numerical methods textbook, and found that Herbie was able to improve accuracy on each example, some by up to 60 bits, while imposing a median performance overhead of 40%. Colleagues in machine learning have used Herbie to significantly improve the results of a clustering algorithm, and a mathematical library has accepted two patches generated using Herbie.

Eliminating Unstable Tests in Floating-Point Programs

“…Additionally, an extensive experimental evaluation is needed in order to assess the quality of the approach and its applicability to real-world applications. Another interesting future direction is the integration of the proposed approach with tools such as Salsa [4] and Herbie [17]. This integration will improve the accuracy of the mathematical expressions used inside a program and, at the same time, prevent unstable tests that may cause unexpected behaviors.…”

“…Program optimization tools aim at improving the accuracy of floating-point programs by rewriting arithmetic expressions in equivalent ones with a lower accumulated round-off error. Herbie [17] is a tool that automatically improves the accuracy of floating-point programs though a heuristic search. Herbie detects the expressions where rounding-errors occur and it applies a series of rewriting and simplification rules.…”

Automatic source‐to‐source error compensation of floating‐point programs: code synthesis to optimize accuracy and time

“…A set of equivalent programs is generated over the real numbers, and the one with the smallest rounding error is chosen. Panchekha et al [20] propose the Herbie tool, which improves the accuracy of expressions by randomly sampling inputs, localizing error, generating candidate rewrites, and merging rewrites with complementary effects. Herbie's results show that it can effectively discover transformations that substantially improve accuracy while imposing a median runtime overhead of 40% on their 3 of 24 numerous selected case studies.…”

Combining Tools for Optimization and Analysis of Floating-Point Computations

“…Other such tools use dynamic techniques to find inputs that suffer from large rounding errors [3,22]. Yet other tools perform rewriting-based optimization [4,16,19,7] and mixed-precision tuning [2,6] to improve the accuracy and performance of floating-point programs.…”

“…In this paper we report on our experience implementing the first combination of two complementary floating-point analysis tools using FPBench: Herbie [16] and Daisy [7]. Herbie optimizes the accuracy of straight-line floating-point expressions, but employs a dynamic roundoff error analysis and thus cannot provide sound guarantees on the results.…”