We formulate an optimal transport problem for matrix-valued density functions. This is pertinent in the spectral analysis of multi variable time-series. The "mass" represents energy at various frequencies whereas, in addition to a usual transportation cost across frequencies, a cost of rotation is also taken into account. We show that it is natural to seek the transportation plan in the tensor product of the spaces for the two matrix-valued marginals. In contrast to the classical Monge-Kantorovich setting, the transportation plan is no longer supported on a thin zero-measure set.