With the proposal of dual-wavelength pumping (DWP) scheme, DWP Er:ZBLAN fiber lasers at 3.5 μm have become a fascinating area of research. However, limited by the absence of suitable saturable absorber, passively Q-switched and mode-locked fiber lasers have not been realized in this spectral region. Based on the layer-dependent bandgap and excellent photoelectric characteristics of black phosphorus (BP), BP is a promising candidate for saturable absorber near 3.5 μm. Here, we fabricated a 3.5-μm saturable absorber mirror (SAM) by transferring BP flakes onto a Au-coated mirror. With the as-prepared BP SAM, we realized Q-switching and mode-locking operations in the DWP Er:ZBLAN fiber lasers at 3.5 μm. To the best of our knowledge, it is the first time to achieve passively Q-switched and mode-locked pulses in 3.5 μm spectral region. The research results will not only promote the development of 3.5-μm pulsed fiber lasers but also open the photonics application of two-dimensional materials in this spectral region.