The cytoplasmic male sterility/restorer-of-fertility (CMS/Rf) system plays a vital role in high-efficiency hybrid seed production in crops, including soybean (Glycine max (L.) Merr.). The markers linked to fertility restoration and the restorer-of-fertility (Rf) genes are essential because they can facilitate the breeding of new CMS lines and production of commercial hybrid soybean seeds. To date, several soybean Rf genes have been mapped to various genetic loci in diverse genetic populations. However, the mapping range of restorer genes remains narrow, with relatively limited practical applicability. Therefore, in the present study, F2 and F3 segregating populations derived from the CMS line JLCMS5A crossed with the restorer line JLR2 were developed and used for Rf3 gene fine mapping. Genetic investigation indicated that the restorer line JLR2 was controlled by a single dominant gene, Rf3. By integrating bulk-segregant analysis and next-generation sequencing, a 4 Mb region on chromosome 9 was identified, which was most likely the target region harboring the candidate gene responsible for fertility restoration. This region was further narrowed down to 86.44 Kb via fine mapping in F2 and F3 populations using SSR, InDel, and dCAPS markers. This region contained 10 putative genes (Glyma.09G171100–Glyma.09G172000). Finally, Glyma.09G171200, which encodes a mitochondria-targeted pentatricopeptide repeat protein, was proposed as the potential candidate for Rf3 using sequence alignment and expression analysis in restorer and CMS lines. Based on single-nucleotide polymorphisms in Glyma.09G172000, a CAPS marker co-segregated with Rf3 named CAPS1712 was developed. Our results will be fundamental in the assisted selection and creation of potent lines for the production and rapid selection of novel restorer lines.