We investigate the optical response of a charge-transfer complex in a condensed phase driven by an external laser field. Our model includes an instantaneous short-range Coulomb interaction and a local optical vibrational mode described by the Holstein-Hubbard (HH) model. Although characterization of the HH model for a bulk system has typically been conducted using a complex phase diagram, this approach is not sufficient for investigations of dynamical behavior at finite temperature, in particular for studies of nonlinear optical properties, where the time irreversibility of the dynamics that arises from the environment becomes significant. We therefore include heat baths with infinite heat capacity in the model to introduce thermal effects characterized by fluctuation and dissipation to the system dynamics. By reducing the number of degrees of freedom of the heat baths, we derive numerically "exact" hierarchical equations of motion (HEOM) for the reduced density matrix of the HH system. As demonstrations, we calculate the optical response of the system in two-and four-site cases under external electric fields.The results indicate that the effective strength of the system-bath coupling becomes large as the number of sites increases. Excitation of electrons promotes the conductivity when the Coulomb repulsion is equivalent to or dominates the electron-phonon coupling, whereas excitation of optical vibrations always suppresses the conductivity.