We present a higher-order finite volume method for solving elliptic PDEs with jump conditions on interfaces embedded in a 2D Cartesian grid. Second, fourth, and sixth order accuracy is demonstrated on a variety of tests including problems with high-contrast and spatially varying coefficients, large discontinuities in the source term, and complex interface geometries. We include a generalized truncation error analysis based on cell-centered Taylor series expansions, which then define stencils in terms of local discrete solution data and geometric information. In the process, we develop a simple method based on Green's theorem for computing exact geometric moments directly from an implicit function definition of the embedded interface. This approach produces stencils with a simple bilinear representation, where spatially-varying coefficients and jump conditions can be easily included and finite volume conservation can be enforced.