Within a genuinely gauge invariant approach recently developed for the computation of the cosmological backreaction, we study, in a cosmological inflationary context and with respect to various observers, the impact of scalar fluctuations on the space-time dynamics in the long wavelength limit. We stress that such a quantum backreaction effect is evaluated in a truly gauge independent way using a set of effective equations which describe the dynamics of the averaged geometry. In particular we show under what conditions the free falling (geodetic) observers do not experience any scalar-induced backreaction in the effective Hubble rate and fluid equation of state.PACS numbers: 98.80. Cq, 04.62.+v Introduction. The computation of backreaction effects induced by cosmological fluctuations in an inflationary era [1,2], has been the subject of controversial analysis [3][4][5][6][7]. Such a task has been plagued by fundamental ambiguities in constructing perturbatively gauge invariant (GI) observables [3] and average quantities. The basic fact that the averaging procedure does not commute with the non linear evolution of Einstein equations [8] was first exploited to study the effective dynamics of the averaged geometry for a dust universe [9] (see [10] for a recent application in the context of inhomogeneity driven inflation). Gauge invariance of averaged quantities has recently been addressed in a novel context which introduces a GI but observer dependent averaging prescription [11] (see [12] for a recent application of such a prescription to the analysis of the present Hubble rate) whereas the effective equations for the averaged geometry [9] have been generalized in a covariant and GI form in [13]. Taking advantage of these recent results and having in mind the backreaction of quantum fluctuations during inflation, we devote this Letter to describing, for the first time in this context, an analysis of the GI effective equations which nevertheless depend on the different observers intrinsically used in the GI construction.Gauge Invariant Backreaction. We start by illustrating how, following a recent proposal [11,13], one may define observables, of a non local nature and constructed with quantum averages, which obey GI dynamical equations. Specifically what has been investigated is how to give a classical or quantum GI average of a scalar S(x), for a classical field or a composite quantum operator, assumed to be renormalized, respectively. In such an approach the fundamental point is the choice of a hypersurface, which defines a class of observers, with respect to which the average is done. In particular a hypersurface Σ A0 is defined, using another scalar field A(x) with a timelike gradient, through the constraint A(x) = A 0 , where A 0 is a con-