Open multi-agent systems (OMASs) allow autonomous agents (AAs) to collaborate in coalitions to accomplish complex tasks (e.g., swarm robots exploring new terrain). In OMASs, AAs can arbitrarily join and leave the network. Thus, AAs must often collaborate with unknown AAs that may corrupt coalitions, leading to less robust systems. However, measures to improve robustness of OMASs are subject to challenges, decreasing their effectiveness. To understand how to improve coalition robustness in OMASs and address challenges of existing robustness measures, we carried out a literature review and revealed three types of robustness measures (i.e., collaboration coordination, normative control, and reliability prediction). Moreover, we found 21 challenges for the identified robustness measures and 24 corresponding solutions. By carrying out this literature review, we forge new connections between existing measures and identify challenges and measures that apply to multiple existing measures. Hereby, our work supports more robust collaborations between AAs in open systems.