In this paper we introduce a notion of duality for matrix valued orthogonal polynomials with respect to a measure supported on the nonnegative integers. We show that the dual families are closely related to certain difference operators acting on the matrix orthogonal polynomials. These operators belong to the so called Fourier algebras, which play a key role in the construction of the families. In order to illustrate duality, we describe a family of Charlier type matrix orthogonal polynomials with explicit shift operators which allow us to find explicit formulas for three term recurrences, difference operators and squared norms. These are the essential ingredients for the construction of different dual families.