Chemical toxicity testing is moving steadily toward a human cell and organoid-based in vitro approach for reasons including scientific relevancy, efficiency, cost, and ethical rightfulness. Inferring human health risk from chemical exposure based on in vitro testing data is a challenging task, facing various data gaps along the way. This review identifies these gaps and makes a case for the in silico approach of computational dose-response and extrapolation modeling to address many of the challenges. Mathematical models that can mechanistically describe chemical toxicokinetics (TK) and toxicodynamics (TD), for both in vitro and in vivo conditions, are the founding pieces in this regard. Identifying toxicity pathways and in vitro point of departure (PoD) associated with adverse health outcomes requires an understanding of the molecular key events in the interacting transcriptome, proteome, and metabolome. Such an understanding will in turn help determine the sets of sensitive biomarkers to be measured in vitro and the scope of toxicity pathways to be modeled in silico. In vitro data reporting both pathway perturbation and chemical biokinetics in the culture medium serve to calibrate the toxicity pathway and virtual tissue models, which can then help predict PoDs in response to chemical dosimetry experienced by cells in vivo. Two types of in vitro to in vivo extrapolation (IVIVE) are needed. (1) For toxic effects involving systemic regulations, such as endocrine disruption, organism-level adverse outcome pathway (AOP) models are needed to extrapolate in vitro toxicity pathway perturbation to in vivo PoD. (2) Physiologically-based toxicokinetic (PBTK) modeling is needed to extrapolate in vitro PoD dose metrics into external doses for expected exposure scenarios. Linked PBTK and TD models can explore the parameter space to recapitulate human population variability in response to chemical insults. While challenges remain for applying these modeling tools to support in vitro toxicity testing, they open the door toward population-stratified and personalized risk assessment.