This paper aims to study the influence of basalt fiber (BF) and polypropylene fiber (PPF) in crumb rubber (CR) mortar made of two different types of cement, including ordinary Portland cement (OPC) and calcium aluminate cement (CAC). CR was used to partially (5%, 10%, 15%, and 20% by volume) replace the fine aggregate in OPC and CAC mortars. BF and PPF were added (0.1%, 0.3%, and 0.5% by total volume) in the CR mortars. The consistency, density, compressive, and flexural strength of cement mortars were investigated. The use of CAC cement slightly increased the consistency; however, the results showed that the CR replacement and the addition of both fiber types tend to reduce the consistency in OPC and CAC mortars. Significant reduction in the density of fiber-added CR mortar was found with increasing CR content, whereas the influence of both PPF and BF was minimal. The fiber-added CR mortar made of both binder and fiber types in general exhibited a reducing trend in the 28 days compressive strength when increasing CR and fiber contents. Nevertheless, an enhancement in the compressive strength of CAC mortar with 20% CR was found with the addition of 0.1% of both fibers. The use of CR and addition of the fibers generally decreased the flexural strength of mortar made of both binder types; however, the addition of 0.3% BF in mortars containing 15–20% CR positively affected the flexural performance. Finally, the artificial neural network (ANN) approach demonstrated the ability to predict the compressive strength of fiber-added CR mortars. The model showed a considerably insignificant mean square error (MSE) of 1.4–1.5 and high plot regression (R) results of 0.97–0.98.