The AgAID Institute is a National AI Research Institute focused on developing AI solutions for specialty crop agriculture. Specialty crops include a variety of fruits and vegetables, nut trees, grapes, berries, and different types of horticultural crops. In the United States, the specialty crop industry accounts for a multibillion dollar industry with over 300 crops grown just along the U.S. west coast. Specialty crop agriculture presents several unique challenges: they are labor‐intensive, are easily impacted by weather extremities, and are grown mostly on irrigated lands and hence are dependent on water. The AgAID Institute aims to develop AI solutions to address these challenges, particularly in the face of workforce shortages, water scarcity, and extreme weather events. Addressing this host of challenges requires advancing foundational AI research, including spatio‐temporal system modeling, robot sensing and control, multiscale site‐specific decision support, and designing effective human–AI workflows. This article provides examples of current AgAID efforts and points to open directions to be explored.