Ptolemy is an open-source and extensible modeling and simulation framework. It offers heterogeneous modeling capabilities by allowing different models of computation, both untimed and timed, to be composed hierarchically in an arbitrary fashion. This paper proposes a formal semantics for Ptolemy which is modular, in the sense that atomic actors and their compositions are treated in a unified way. In particular, all actors conform to an executable interface that contains four functions: fire (produce outputs given current state and inputs), postfire (update state instantaneously), deadline (how much time the actor is willing to let elapse) and time-update (update state with passage of time). Composite actors are obtained from composition operators that in Ptolemy are called directors. Different directors realize different models of computation. This paper defines formally the directors for the following models of computation: Synchronous-Reactive, Discrete Event, Continuous Time, Process Networks, and Modal Models.