Chronological aging is uniform, but biological aging is heterogeneous. Clinically, this heterogeneity manifests itself in health status and mortality, and it distinguishes healthy from unhealthy aging. Clinical frailty indexes (FIs) serve as an important tool in gerontology to capture health status. FIs have been adapted for use in mice and are an effective predictor of mortality risk. To accelerate our understanding of biological aging, high-throughput approaches to pre-clinical studies are necessary. Currently, however, mouse frailty indexing is manual and relies on trained scorers, which imposes limits on scalability and reliability. Here, we introduce a machine learning based visual frailty index (vFI) for mice that operates on video data from an open field assay. We generate a large mouse FI dataset of both male and female mice. From video data on these same mice, we use neural networks to extract morphometric, gait, and other behavioral features that correlate with manual FI score and age. We use these features to train a regression model that accurately predicts frailty within 0.04 ± 0.002 (3.7% ± 0.01%) of the pre-normalized FI score in terms of mean absolute error. We show that features of biological aging are encoded in open-field video data and can be used to construct a vFI that can complement or replace current manual FI methods. We use the vFI data to examine sex-specific aspects of aging in mice. This vFI provides increased accuracy, reproducibility, and scalability, that will enable large scale mechanistic and interventional studies of aging in mice.