A Hamiltonian-based model of many harmonically interacting massive particles that are subject to linear friction and coupled to heat baths at different temperatures is used to study the dynamic approach to equilibrium and non-equilibrium stationary states. An equilibrium system is here defined as a system whose stationary distribution equals the Boltzmann distribution, the relation of this definition to the conditions of detailed balance and vanishing probability current is discussed both for underdamped as well as for overdamped systems. Based on the exactly calculated dynamic approach to the stationary distribution, the functional that governs this approach, which is called the free entropy S free (t), is constructed. For the stationary distribution S free (t) becomes maximal and its time derivative, the free entropy productionṠ free (t), is minimal and vanishes. Thus, S free (t) characterizes equilibrium as well as non-equilibrium stationary distributions by their extremal and stability properties. For an equilibrium system, i.e. if all heat baths have the same temperature, the free entropy equals the negative free energy divided by temperature and thus corresponds to the Massieu function which was previously introduced in an alternative formulation of statistical mechanics. Using a systematic perturbative scheme for calculating velocity and position correlations in the overdamped massless limit, explicit results for few particles are presented: For two particles localization in position and momentum space is demonstrated in the non-equilibrium stationary state, indicative of a tendency to phase separate. For three elastically interacting particles heat flows from a particle coupled to a cold reservoir to a particle coupled to a warm reservoir if the third reservoir is sufficiently hot. This does not constitute a violation of the second law of thermodynamics, but rather demonstrates that a particle in such a non-equilibrium system is not characterized by an effective temperature which equals the temperature of the heat bath it is coupled to. Active particle models can be described in the same general framework, which thereby allows to characterize their entropy production not only in the stationary state but also in the approach to the stationary nonequilibrium state. Finally, the connection to non-equilibrium thermodynamics formulations that include the reservoir entropy production is discussed.