We introduce a simple determinant diagrammatic Monte Carlo algorithm to compute the groundstate properties of a particle interacting with a Fermi sea through a zero-range interaction. The fermionic sign does not cause any fundamental problem when going to high diagram orders, and we reach order N = 30. The data reveal that the diagrammatic series diverges exponentially as (−1/R) N with a radius of convergence R < 1. Furthermore, on the polaron side of the polarondimeron transition, the value of R is determined by a special class of three-body diagrams, corresponding to repeated scattering of the impurity between two particles of the Fermi sea. A powercounting argument explains why finite R is possible for zero-range interactions in three dimensions. Resumming the divergent series through a conformal mapping yields the polaron energy with record accuracy.