The imaging performance of clinical positron emission tomography (PET) systems has evolved impressively during the last ∼15 years. A main driver of these improvements has been the introduction of time-of-flight (TOF) detectors with high spatial resolution and detection efficiency, initially based on photomultiplier tubes, later silicon photomultipliers. This review aims to offer insight into the challenges encountered, solutions developed, and lessons learned during this period. Detectors based on fast, bright, inorganic scintillators form the scope of this work, as these are used in essentially all clinical TOF-PET systems today. The improvement of the coincidence resolving time (CRT) requires the optimization of the entire detection chain and a sound understanding of the physics involved facilitates this effort greatly. Therefore, the theory of scintillation detector timing is reviewed first. Once the fundamentals have been set forth, the principal detector components are discussed: the scintillator and the photosensor. The parameters that influence the CRT are examined and the history, state-of-the-art, and ongoing developments are reviewed. Finally, the interplay between these components and the optimization of the overall detector design are considered. Based on the knowledge gained to date, it appears feasible to improve the CRT from the values of 200-400 ps achieved by current state-of-the-art TOF-PET systems to about 100 ps or less, even though this may require the implementation of advanced methods such as time resolution recovery. At the same time, it appears unlikely that a system-level CRT in the order of ∼10 ps can be reached with conventional scintillation detectors. Such a CRT could eliminate the need for conventional tomographic image reconstruction and a search for new approaches to timestamp annihilation photons with ultra-high precision is therefore warranted. While the focus of this review is on timing performance, it attempts to approach the topic from a clinically driven perspective, i.e. bearing in mind that the ultimate goal is to optimize the value of PET in research and (personalized) medicine. Selected abbreviations and symbols BSR Backside readout C d Diode capacitance CFD Constant-fraction discriminator CRLB Cramér-Rao lower bound CRT Coincidence resolving time C q Parallel capacitance of quench resistor DCR Dark count rate DOI Depth of interaction dSiPM Digital silicon photomultiplier DSR Dual-sided readout