Mammalian dentition consists of teeth that develop as discrete organs. From anterior to posterior, the dentition is divided into regions of incisor, canine, premolar and molar tooth types. Particularly teeth in the molar region are very diverse in shape. The development of individual teeth involves epithelial-mesenchymal interactions that are mediated by signals shared with other organs. Parts of the molecular details of signaling networks have been established, particularly in the signal families BMP, FGF, Hh and Wnt, mostly by the analysis of gene expression and signaling responses in knockout mice with arrested tooth development. Recent evidence suggests that largely the same signaling cascade is used reiteratively throughout tooth development. The successional determination of tooth region, tooth type, tooth crown base and individual cusps involves signals that regulate tissue growth and differentiation. Tooth type appears to be determined by epithelial signals and to involve differential activation of homeobox genes in the mesenchyme. This differential signaling could have allowed the evolutionary divergence of tooth shapes among the four tooth types. The advancing tooth morphogenesis is punctuated by transient signaling centers in the epithelium corresponding to the initiation of tooth buds, tooth crowns and individual cusps. The latter two signaling centers, the primary enamel knot and the secondary enamel knot, have been well characterized and are thought to direct the differential growth and subsequent folding of the dental epithelium. Several members of the FGF signal family have been implicated in the control of cell proliferation around the non-dividing enamel knots. Spatiotemporal induction of the secondary enamel knots determines the cusp patterns of individual teeth and is likely to involve repeated activation and inhibition of signaling as suggested for patterning of other epithelial organs.

One motivation in the study of development is the discovery of mechanisms that may guide evolutionary change. Here we report how development governs relative size and number of cheek teeth, or molars, in the mouse. We constructed an inhibitory cascade model by experimentally uncovering the activator-inhibitor logic of sequential tooth development. The inhibitory cascade acts as a ratchet that determines molar size differences along the jaw, one effect being that the second molar always makes up one-third of total molar area. By using a macroevolutionary test, we demonstrate the success of the model in predicting dentition patterns found among murine rodent species with various diets, thereby providing an example of ecologically driven evolution along a developmentally favoured trajectory. In general, our work demonstrates how to construct and test developmental rules with evolutionary predictability in natural systems.

The study of mammalian evolution depends greatly on understanding the evolution of teeth and the relationship of tooth shape to diet. Links between gross tooth shape, function and diet have been proposed since antiquity, stretching from Aristotle to Cuvier, Owen and Osborn. So far, however, the possibilities for exhaustive, quantitative comparisons between greatly different tooth shapes have been limited. Cat teeth and mouse teeth, for example, are fundamentally distinct in shape and structure as a result of independent evolutionary change over tens of millions of years. There is difficulty in establishing homology between their tooth components or in summarizing their tooth shapes, yet both carnivorans and rodents possess a comparable spectrum of dietary specializations from animals to plants. Here we introduce homology-free techniques to measure the phenotypic complexity of the three-dimensional shape of tooth crowns. In our geographic information systems (GIS) analysis of 441 teeth from 81 species of carnivorans and rodents, we show that the surface complexity of tooth crowns directly reflects the foods they consume. Moreover, the absolute values of dental complexity for individual dietary classes correspond between carnivorans and rodents, illustrating a high-level similarity between overall tooth shapes despite a lack of low-level similarity of specific tooth components. These results suggest that scale-independent forces have determined the high-level dental shape in lineages that are widely divergent in size, ecology and life history. This link between diet and phenotype will be useful for inferring the ecology of extinct species and illustrates the potential of fast-throughput, high-level analysis of the phenotype.