Active matter, as other types of self-organizing systems, relies on the take-up of energy that can be used for different activities, such as active motion or structure formation. Here we provide an agent-based framework to model these processes at different levels of organization, physical, biological and social, using the same dynamic approach. Driving variables describe the take-up, storage and conversion of energy, whereas driven variables describe the energy consuming activities. The stochastic dynamics of both types of variables follow a modified Langevin equation. Additional non-linear functions allow to encode system-specific hypotheses about the relation between driving and driven variables. To demonstrate the applicability of this framework, we recast a number of existing models of Brownian agents and Active Brownian Particles. Specifically, active motion, clustering and self-wiring of networks based on chemotactic interactions, online communication and polarization of opinions based on emotional influence are discussed. The framework allows to obtain critical parameters for active motion and the emergence of collective phenomena. This highlights the role of energy take-up and dissipation in obtaining different dynamic regimes. An agent-based framework of active matter with applications in biological and social systems European Journal of Physics (Revised submission) Some historical remarks. To put the recent research about active matter into perspective, it is useful to remind on some relations to existing scientific paradigms. The systemic capabilities of active matter to develop and to maintain coherent structures, or collective states, based on the input and the conversion of energy were previously described using the term self-organization. It was heuristically defined as the "spontaneous formation, evolution and differentiation of complex order structures forming in non-linear dynamic systems by way of feedback mechanisms involving the elements of the systems, when these systems have passed a critical distance from the statical equilibrium as a result of the influx of unspecific energy, matter or information." [40]. The physics of self-organization and evolution [13] has made fundamental contributions to the scientific understanding of self-organizing systems -the thermodynamics of irreversible processes, deterministic and stochastic models of nonlinear dynamics, the theory of reaction-diffusion processes, information-theoretical approaches to sequences, to name just a few.While self-organization could be seen as the leading paradigm of the 1970th and 1980th, it was gradually replaced by complex systems as the leading paradigm after the year 1990. This development came along with a shifting focus. Self-organization and pattern formation could be well described on the macroscopic or systemic level, for example by coupled partial differential equations. With complex systems, i.e. systems composed of a large number of (strongly) interacting elements, the focus was more on the emergence of systemic pro...