Research shows that men diagnosed with male factor infertility experience more suffering than men with infertility due to other causes, and that it is socially unfavourable to be diagnosed with male factor infertility resulting in secrecy surrounding diagnosis, sometimes to the point that women take the blame for the couples' infertility. We investigated mental and physical health, support, and psychological and social stress in men (N = 256) prior to and after 12 months of unsuccessful treatment according to their diagnosis: unexplained, female factor, male factor, or mixed. Results suggest that men do not differ by diagnosis on any of the variables tested. When treatment was not successful, all men showed increased suffering in the form of decreased mental health, increased physical stress reactions, decreased social support, and increased negative social stress over time. These findings indicate that involuntary childlessness is difficult for all men, and is not dependent on with whom the cause lies. There was also a high level of agreement between couples and medical records on the cause of the couples' fertility arguing against the idea that women take the blame for male factor infertility.