We present BVRI colors of 13 Jovian and 8 Saturnian irregular satellites obtained with the 2.56m Nordic Optical Telescope on La Palma, the 6.5m Magellan Baade Telescope on La Campanas, and the 6m MMT on Mt. Hopkins. The observations were performed between December 2001 to March 2002. Nearly all of the known irregular satellites can be divided into two distinct classes based on their colors. One, the grey color class, has the similar colors to the C-type asteroid, and the other, the light red color class, has colors similar to P/D-type asteroids. We also find at least one object, the Jovian irregular J XXIII Kalyke, that has colors similar to the red colored Centaurs/TNOs, although its classification is unsecure.We find that there is a correlation between the physical properties and dynamical To which of these two clusters the remaining Saturnian irregulars with inclinations close to 174• belong is not clear from our analysis, but determination of their colors should help constrain this.We also show through analysis of possible fragmentation and dispersion of the six known Uranian irregulars that they most likely make up two clusters, one centered on U XVI Caliban and another centered on U XVII Sycorax. We further show that, although -4 -the two objects have similar colors, a catastrophic fragmentation event creating one cluster containing both U XVI Caliban and U XVII Sycorax would have involved a progenitor with a diameter of ∼ 395km. While such an event is not impossible it seems rather improbable, and further we show that such an event would leave 5-6 fragments with sizes comparable or lager than U XVI Caliban. Given that the stable region around Uranus has been extensively searched to limiting magnitudes far beyond that of U XVI Caliban. The fact that only U XVI Caliban and the larger U XVII Sycorax have been found leaves us with a distribution not compatible with a catastrophic event with such a large progenitor. The most likely solution is therefore two separate events creating two Uranian dynamical clusters.