The nonlinear interaction of two temporally separated solitons inside optical fibers has been studied in the scalar case in which both solitons remain linearly polarized during their mutual interaction. Here we consider two arbitrarily polarized vector solitons inside birefringent optical fibers where they interact nonlinearly. We develop a general formalism based on two coupled nonlinear Schrödinger equations that include both the Raman and the Kerr nonlinearities. We use it to study how the two vector solitons evolve in the temporal, spectral, and polarization domains in optical fibers exhibiting no, medium, or large birefringence. The attractive force that leads to collisions of two in-phase solitons in isotropic fibers is affected considerably by fiber's birefringence. In particular, no collisions occur even in the case of medium birefringence. Rather, the two vector solitons exchange energy as they approach each other and then move away from each other. Moreover, considerably energy is shed in the form of dispersive waves. Polarization of the two solitons is found to be affected considerably by the fiber nonlinearity in the case of medium birefringence but linear effects dominate its evolution in fibers with large birefringence.