In rural areas where most of the food production takes place, nutritional deprivations are also high. In 2015–16, only 38% (45%) of 6–23-month-old children in rural (urban) India reported milk consumption at least once in the previous day with an average of 2.5 times among those who consumed. Though dairy milk plays a very important role in the growth of young children, very few studies have analysed the pattern of consumption as in this study, which also provides an understanding of its associated factors and relationship with a child’s height for age Z-score. The zero-inflated Poisson model estimates show that girls are less likely to be given milk but irrespective of gender, milk consumption increases with age. Breastfeeding affects both milk consumption and its frequency while other food items except eggs and other dairy reduce the chance of milk consumption but not its frequency. Mother’s education, economic status of the household, ownership of bovine, and higher supply of milk at the district level have differing effects on milk consumed compared to the frequency of milk consumed. Milk consumption and its frequency is positively associated with a child’s height provided it is an endogenous variable. Policies that strengthen affordability, availability, and awareness towards milk consumption would reduce large disparities in child stunting in India and improve its abysmally low global ranking.