We investigate the evolution of model populations of ultraluminous X-ray sources (ULXs), consisting of a black hole accretor in a binary with a donor star. Two of the models we consider invoke stellar-mass (up to $25 M ) black hole binaries (LMBHs), generated with a binary population synthesis code, while a third model uses intermediatemass ($1000 M ) black hole accretors ( IMBHs). For each model, we computed 30,000 binary evolution sequences. A scheme for calculating the optical flux from ULXs is discussed. We present ''probability images'' for the colormagnitude diagrams (CMDs) and for the orbital periodYX-ray luminosity (P orb -L x ) plane. We show how a population of luminous X-ray sources in a cluster of stars evolves with time. The most probable ULX system parameters correspond to high-mass donors (of initial mass k25 M ) with effective O through late B spectral types, and P orb between 1 and 10 days. Estimates of the numbers of ULXs in a typical galaxy as a function of L x are also presented. We find that if LMBHs are allowed to have super-Eddington L x , the binding energy parameter for the stellar envelope of the black hole progenitor must be k P 0:03 in order not to overproduce ULXs. Comparison of six known ULX counterparts with our model CMDs indicates that the IMBH model somewhat more closely matches the observations. We find that a significant contribution to the optical flux from the IMBH systems comes from intrinsic accretion disk radiation. In effect, IMBH systems, when operating at their maximum luminosities (10 41 Y10 42 ergs s À1 ), are milli-AGNs. While models of IMBH systems during the X-ray phase are attractive, their formation mechanism remains uncertain.