DNA double strand breaks (DSBs) can be rejoined directly by the nonhomologous end-joining (NHEJ) pathway of repair. Nucleases and polymerases are required to promote accurate NHEJ when the terminal bases of the DSB are damaged. The same enzymes also participate in imprecise rejoining and joining of incompatible ends, important mutagenic events. Previous work has shown that the Pol X family polymerase Pol4 is required for some but not all NHEJ events that require gap filling in Saccharomyces cerevisiae. Here, we systematically analyzed DSB end configurations and found that gaps on both strands and overhang polarity are the principal factors that determine whether a joint requires Pol4. DSBs with 3-overhangs and a gap on each strand strongly depended on Pol4 for repair, DSBs with 5-overhangs of the same sequence did not. Pol4 was not required when 3-overhangs contained a gap on only one strand, however. Pol4 was equally required at 3-overhangs of all lengths within the NHEJ-dependent range but was dispensable outside of this range, indicating that Pol4 is specific to NHEJ. Loss of Pol4 did not affect the rejoining of DSBs that utilized a recessed microhomology or DSBs bearing 5-hydroxyls but no gap. Finally, mammalian Pol X polymerases were able to differentially complement a pol4 mutation depending on the joint structure, demonstrating that these polymerases can participate in yeast NHEJ but with distinct properties.