The general principle of Cyclotron Radiation Emission Spectroscopy (CRES) experiments is to build an energy spectrum by reconstructing the start frequencies of charged particle trajectories (called tracks) which leave quasilinear profiles in the time-frequency plane when exposed to a magnetic field. The Project 8 collaboration is developing the CRES technique in order to extract the unknown absolute neutrino mass value with a final sensitivity 0.04 eV/$c^2$ from the $\beta$-decay energy spectrum of tritium. Due to the small number of events in the spectrum's endpoint region and the need for excellent instrumental energy resolution, efficient and highly accurate reconstruction methods are desired. Deep learning convolutional neural networks (CNNs) - particularly suited to deal with information-sparse data and which offer precise foreground localization - may be utilized to extract track properties from basic CRES signals with relative computational ease. In this work, we develop a novel machine learning based model which operates a CNN and a support vector machine in tandem to reconstruct simulated CRES tracks and events. As applied to simulated CRES data, we show comparable performance in accuracy of track parameter reconstruction and a relative gain of 24.1\% in event reconstruction efficiency when compared to a traditional point-clustering based approach (baseline).