Inference of action potentials (‘spikes’) from neuronal calcium signals is complicated by the scarcity of simultaneous measurements of action potentials and calcium signals (‘ground truth’). We compiled a large, diverse ground-truth database from publicly available and newly performed recordings in zebrafish and mice covering a broad range of calcium indicators, cell types, and signal-to-noise ratios, comprising a total of >35 recording hours from 298 neurons. We developed an algorithm for spike inference (CASCADE) that is based on supervised deep networks, takes advantage of the ground-truth database, infers absolute spike rates, and outperforms existing model-based algorithms. To optimize performance for unseen imaging data, CASCADE retrains itself by resampling ground-truth data to match the respective sampling rate and noise level; therefore, no parameters need to be adjusted by the user. In addition, we developed systematic performance assessments for unseen data, openly release a resource toolbox, and provide a user-friendly cloud-based implementation.