Topologically non-trivial electronic structures can give rise to a range of unusual physical phenomena, and the interplay of band topology with other effects such as electronic correlations and magnetism requires further exploration. The rare earth monopnictides X(Sb,Bi) (X = lanthanide) are a large family of semimetals where these different effects may be tuned by the substitution of rare-earth elements. Here we observe anomalous behavior in the quantum oscillations of one member of this family, antiferromagnetic SmSb. The analysis of Shubnikov-de Haas (SdH) oscillations provides evidence for a non-zero Berry phase, indicating a non-trivial topology of the α-band. Furthermore, striking differences are found between the temperature dependence of the amplitudes of de Haas-van Alphen effect oscillations, which are well fitted by the Lifshitz-Kosevich (LK) formula across the measured temperature range, and those from SdH measurements which show a significant disagreement with LK behavior at low temperatures. Our findings of unusual quantum oscillations in an antiferromagnetic, mixed valence semimetal with a possible non-trivial band topology can provide an opportunity for studying the interplay between topology, electronic correlations and magnetism.