Although enormous efforts have been devoted to the development of new types of fluorometric immunochromatographic test strip (ICTS) with improved sensitivity over the past years, it still remains a big challenge to design ICTS with colorimetric and fluorescent bimodal signal readout for rapid yet accurate detection of cancer markers in a clinic. Scientists have tried to prepare bimodal reporters by combining fluorescent dyes with metal nanomaterials, but their fluorescence was easily quenched by metal nanomaterials through surface energy transfer, making dual colorimetric and fluorometric ICTS very difficult to be achieved. As compared to conventional fluorescent probes, semiconducting polymer dots (Pdots) exhibit extraordinary fluorescence brightness and facile surface functionalization, which are very suitable to be engineered as bimodal signal reporting reagents. Here, we integrated highly fluorescent Pdots with strongly plasmonic Au nanorods to form Pdot–Au hybrid nanocomposites with dual colorimetric and fluorescent readout abilities. We further utilized these nanohybrids in ICTS for qualitatively fast screening (colorimetry) as well as quantitatively accurate determination (fluorometry) of prostate-specific antigen (PSA) within 10 min. By taking advantage of the plasmon-enhanced fluorescence of Pdots on Au nanorods, this immunoassay possesses much better detection sensitivity of 1.07 pg/mL for PSA, which is at least 2 orders of magnitude lower than that of conventional fluorometric ICTS. Moreover, the direct detection of PSA from human whole blood collected without sample pretreatment makes this Pdot-based ICTS platform promising for on-site point-of-care diagnostics.