The highlights of this paper are the following ones.• We propose a threat model to identify various leak-age channels, both in software and hardware layers, to demonstrate possible threats and vulnerabilities. We identify leakages at different levels of the Intel x86 cache hierarchy that help narrow down major attack possibilities in caches. • We provide a taxonomy of leakage channels, their classification, and the type of threat associated with each for different implementations of RSA cryptosys-tems in particular.• We investigate the timing channels on various cryp-tographic implementations. In the last decade, dif-ferent implementations of RSA cryptosystem have been attacked. We provide a detailed analysis of these attacks on different implementations of RSA and explain the problem of leakage and threat level involved in these implementations.• We analyze and list software and hardware counter-measure techniques proposed so far against known attacks on RSA. We also identify significant mitiga tion techniques that are effective at various cache levels and that address different threat levels with respect to our proposed threat model. We analyze the efficiency of the proposed countermeasures and explain their efficacy against our proposed threat model. • We identify existing auditing/detection techniques against cache-based side-channel attacks (CSCAs) using hardware performance monitoring counters (HPCs) to detect stealthy attacks. We underline the importance of detection mechanisms as a new angle of research to provide need-based mitigation toward CSCAs. • We discuss various open threat areas in cache hier-archy that have not been properly addressed by the proposed mitigation techniques so far. We also dis-cuss the challenges associated with hardware mitiga-tion solutions and argue in favor of strong software countermeasures against threats in cache hierarchy in contemporary processors (Intel x86).