We study a model for a pair of qubits which interact with a single off-resonant cavity mode and, in addition, exhibit a direct inter-qubit coupling. Possible realizations for such a system include coupled superconducting qubits in a line resonator as well as exciton states or electron spin states of quantum dots in a cavity. The emergent dynamical phenomena are strongly dependent on the relative energy scales of the inter-qubit coupling strength, the coupling strength between qubits and cavity mode, and the cavity mode detuning. We show that the cavity mode dispersion enables a measurement of the state of the coupled-qubit system in the perturbative regime. We discuss the effect of the direct inter-qubit interaction on a cavity-mediated two-qubit gate. Further, we show that for asymmetric coupling of the two qubits to the cavity, the direct inter-qubit coupling can be controlled optically via the ac Stark effect.