Blockade of immune checkpoints transformed the paradigm of systemic cancer therapy, enabling substitution of a cytotoxic chemotherapy backbone to one of immunostimulation in many settings. Invigorating host immune cells against tumor neo-antigens, however, can induce severe autoimmune toxicity which in many cases requires ongoing management. Many immune-related adverse events (irAEs) are clinically and pathologically indistinguishable from inborn errors of immunity arising from genetic polymorphisms of immune checkpoint genes, suggesting a possible shared driver for both conditions. Many endocrine irAEs, for example, have analogous primary genetic conditions with varied penetrance and severity despite consistent genetic change. This is akin to onset of irAEs in response to immune checkpoint inhibitors (ICIs), which vary in timing, severity and nature despite a consistent drug target. Host contribution to ICI response and irAEs, particularly those of endocrine origin, such as thyroiditis, hypophysitis, adrenalitis and diabetes mellitus, remains poorly defined. Improved understanding of host factors contributing to ICI outcomes is essential for tailoring care to an individual’s unique genetic predisposition to response and toxicity, and are discussed in detail in this review.