In recognizing the host cellular receptor and mediating fusion of virus and cell membranes, the spike (S) glycoprotein of coronaviruses is the most critical viral protein for cross-species transmission and infection. Here we determined the cryo-EM structures of the spikes from bat (RaTG13) and pangolin (PCoV_GX) coronaviruses, which are closely related to SARS-CoV-2. All three receptor-binding domains (RBDs) of these two spike trimers are in the “down” conformation, indicating they are more prone to adopt this receptor-binding inactive state. However, we found that the PCoV_GX, but not the RaTG13, spike is comparable to the SARS-CoV-2 spike in binding the human ACE2 receptor and supporting pseudovirus cell entry. Through structure and sequence comparisons, we identified critical residues in the RBD that underlie the different activities of the RaTG13 and PCoV_GX/SARS-CoV-2 spikes and propose that N-linked glycans serve as conformational control elements of the RBD. These results collectively indicate that strong RBD-ACE2 binding and efficient RBD conformational sampling are required for the evolution of SARS-CoV-2 to gain highly efficient infection.