We show that a functor category whose domain is a colored category is a topos. The topos structure enables us to introduce cohomology of colored categories including quasi-schemoids. If the given colored category arises from an association scheme, then the cohomology coincides with the group cohomology of the factor scheme by the thin residue. Moreover, it is shown that the cohomology of a colored category relates to the standard representation of an association scheme via the Leray spectral sequence.