Recently, over-the-air computation (AirComp) has emerged as an efficient solution for access points (APs) to aggregate distributed data from many edge devices (e.g., sensors) by exploiting the waveform superposition property of multiple access (uplink) channels. While prior work focuses on the singlecell setting where inter-cell interference is absent, this paper considers a multi-cell AirComp network limited by such interference and investigates the optimal policies for controlling devices' transmit power to minimize the mean squared errors (MSEs) in aggregated signals received at different APs. First, we consider the scenario of centralized multi-cell power control. To quantify the fundamental AirComp performance tradeoff among different cells, we characterize the Pareto boundary of the multi-cell MSE region by minimizing the sum MSE subject to a set of constraints on individual MSEs. Though the sum-MSE minimization problem is non-convex and its direct solution intractable, we show that this problem can be optimally solved via equivalently solving a sequence of convex second-order cone program (SOCP) feasibility problems together with a bisection search. This results in an efficient algorithm for computing the optimal centralized multi-cell power control, which optimally balances the interferenceand-noise-induced errors and the signal misalignment errors unique for AirComp. Next, we consider the other scenario of distributed power control, e.g., when there lacks a centralized controller. In this scenario, we introduce a set of interference temperature (IT) constraints, each of which constrains the maximum total inter-cell interference power between a specific pair of cells. Accordingly, each AP only needs to individually control the power of its associated devices for single-cell MSE minimization, but subject to a set of IT constraints on their interference to neighboring cells. By optimizing the IT levels, the distributed power control is shown to provide an alternative method for characterizing the same multicell MSE Pareto boundary as the centralized counterpart. Building on this result, we further propose