Advances in modern manufacturing have enabled the multiscalar patterning of dielectric media with nearly arbitrary layouts, presenting unique opportunities to revolutionize the design and fabrication pipeline for photonic technologies. In this Perspective, we discuss how algorithms based on classical optimization and deep learning are establishing a new conceptual framework for freeform optical engineering. These tools can specify suitable design parameters for a desired objective, automate the high-speed optimization of freeform devices, and augment manufacturing processes to mitigate challenges set by freeform fabrication. A central feature of many of these algorithms is their utilization of data and physics to model and exploit high-dimensional relationships between geometric structure and electromagnetic response within the constraints of Maxwell's equations. We anticipate that these algorithm-driven methods will streamline optical systems design at the physical limits of structured media and become standard academic and industry tools for scientists and engineers.