Climate change-induced (CCI) calamities have immense negative impacts on coastal fishers’ livelihoods by damaging their household and fishery-related assets, which triggers cyclic poverty among them. This study aimed to identify household-level economic penalties due to CCI calamities among the coastal fishers of Bangladesh. A cross-sectional study was conducted among 338 coastal fishers living in 11 coastal districts using a random sampling technique. Males outnumbered females [84.6% vs. 15.4%], with nearly all (99.1%) living below the poverty line. On average, coastal fishers had to struggle against at least three (3.12 ± 1.28) CCI catastrophes with considerable losses per incidence [domestic asset loss (DAL): 38318 ± 33071; loss in fisheries (LIF): 19236 ± 20486] within the past five years. CCI incidences such as severity of disaster (SoD), longevity of disaster (LoD), winter-cold shock (WCS), and water flow were significantly associated with DAL [95% C.I., p < 0.05]. Similarly, SoD, WCS, water flow, and sea waves during storms (SWDS) were also significantly associated with the impoverished status of coastal fishers in Bangladesh. Climate-resilient coastal management and livelihood improvement policies implemented by the relevant authorities could reduce impending economic penalties for coastal fishing communities.