We have recently 1 argued that classical electrodynamics can predict nonlocal effects by showing an example of a topological and nonlocal electromagnetic angular momentum. In this paper we discuss the dual of this angular momentum which is also topological and nonlocal. We then unify both angular momenta by means of the electromagnetic angular momentum arising in the configuration formed by a dyon encircling an infinitely-long dual solenoid enclosing uniform electric and magnetic fluxes and show that this electromagnetic angular momentum is topological because it depends on a winding number, is nonlocal because the electric and magnetic fields of this dual solenoid act on the dyon in regions for which these fields are excluded and is invariant under electromagnetic duality transformations. We explicitly verify that this duality-invariant electromagnetic angular momentum is insensitive to the radiative effects of the Liénard-Wiechert fields of the encircling dyon. We also show how duality symmetry of this angular momentum suggests different physical interpretations for the corresponding angular momenta that it unifies.