The North American Multi-Model Ensemble (NMME) has become integral scientific infrastructure in subseasonal and seasonal climate prediction, advancing applied prediction and scientific understanding of the mechanisms and predictability of short-term climate. The NMME was designed to evolve as old models were retired and new models joined the ensemble. This study examines the assumption that prediction skill will increase as the system evolves, focusing on 2 m temperature, precipitation rate, and sea surface temperature prediction. The common period of 1982-2010 is studied for four configurations of the NMME, approximately representing the operational model suites of 2011, 2012, 2014-2018, and 2019-present. Substantial improvement in temperature prediction over both land and ocean is observed, with little change in global precipitation prediction. Sea surface temperature prediction at longer leads has improved over much of the globe, with the notable exception of the central-eastern tropical Pacific, where prediction skill has declined.Plain Language Summary Predicting the climate more than a few weeks in advance requires forecasters to bet on more than one computer model, similar to the method of diversifying an investment portfolio. Many earlier studies have shown that the combination of predictions generated by different computer models, known as a multi-model ensemble, almost always achieves better forecast skill than using a single model alone. The North American Multi-Model Ensemble (NMME), a subseasonal and seasonal prediction system combining individual North American state-of-the-art climate prediction models, has become an integral part of subseasonal and seasonal research and applications. The NMME has continually evolved, as newer models replace older ones; it is assumed that this evolution will produce more skillful predictions over time. But, until now, this assumption has not been tested. We examine the skill of NMME predictions in four different model combinations, including the oldest configuration, two transitional suites, and the operational configuration as of early 2020. Temperature prediction over both land and ocean has improved noticeably through the exchange of older models for newer ones, but precipitation prediction has not substantially improved. Overcoming the difficulty in precipitation prediction may require higher-resolution climate models in the NMME.