Scaffold-Lab: Critical Evaluation and Ranking of Protein Backbone Generation Methods in A Unified Framework

Abstract

De novo protein design has undergone a rapid development in recent years, particularly in fix-backbone sequence design and backbone generation. The latter stands out as more challenging yet valuable, offering the ability to design novel protein folds with fewer constraints. However, there is an absence of a comprehensive delineation of its potential for integration into practical applications in the field of protein engineering, as well as a standardized evaluation framework to accurately assess the diverse methodologies within this field. Here, we proposed Scaffold-Lab benchmark focusing on evaluating unconditional generation across metrics such as designability, novelty, diversity, efficiency and structural properties. We also extend our benchmark to include the motif-scaffolding problem, demonstrating the applicability of these conditional generation models. Our findings reveal that FrameFlow and RFdiffusion in unconditional generation and GPDL in conditional generation showcased the most outstanding performances. All data and script will be available at https://github.com/Immortals-33/Scaffold-Lab.