In this study we sought to collect evidence regarding the validity of the Comprehensive Inventory of Thriving (CIT), systematically reviewing studies that tested its psychometric properties (Study 1) and trying to replicate validity evidence collected across previous validation studies (Study 2). We found five studies that tested the validity of CIT scores through the collection of different kinds of evidence (score structure validity, convergent validity, discriminant validity, criterion-related validity, incremental validity, internal consistency, test-retest reliability). Results were often inconsistent across studies (especially for the score structure validity evidence). Using a sample of 483 Italian participants (63.0% female; aged 18–71 years), we replicated the tests performed in the previous validation studies. Findings suggest that the best fitting model is the one that (1) adds the overarching latent construct of thriving, which can be measured using the total scale score; and (2) merges the Skills and Flow factors in just one factor, named “Skills for Flow”. At the same time, the different kinds of validity evidence collected both in previous validation studies and in the current replication study indicate high overlap among thriving sub-dimensions and poor validity evidence. We concluded that the CIT in its present form is not an adequate instrument to assess thriving, thus mono-dimensional scales (e.g. Brief Inventory of Thriving) should be currently preferred. Suggestions to develop a multi-dimensional scale measuring thriving (both using a theory-driven approach or a data-drive approach) are discussed.