Known as breadfruits for their diverse nutritious fruits, the Artocarpus (Moraceae) genus is prized for its high-quality timber, medicinal value, and economic importance. Breadfruits are native to Southeast Asia but have been introduced to other continents. The most commonly cultivated species are Artocarpus heterophyllus (Jackfruit) and Artocarpus altilis (Breadfruit). With numerous smaller but nutritionally comparable fruits on a large tree, Artocarpus hirsutus, also called "Wild jack" or "Ayani" is an elusive forest species endemic to Indian Western Ghats. In this study, we sequenced and assembled the whole genome of Artocarpus hirsutus sampled from sacred groves of Coorg, India. We compared our Wild jack genome with previously published Jackfruit and Breadfruit genomes to decipher demography and evolution. Demographic history reconstruction indicates a stronger effect of habitat rather than phylogeny on the population histories of these plants. Repetitive genomic regions, especially LTR Copia, strongly affected the demographic trajectory of A. heterophyllus. Upon further investigation, we found a recent lineage-specific accumulation of LTR Copia in A. heterophyllus, which results in its larger genome size. Gene family evolution and signatures of selection identified multiple genes from starch, sucrose metabolism, plant hormone signal transduction, etc., in Artocarpus species. Our comparative genomic framework provides important insights by incorporating endemic species such as the Wild jack.