The instability of layer-crack plate structure in coal wall is one of the causes of rock burst. In the present paper, we investigate the formation and instability processes of layer-crack plate structure in coal wall by experiments and theoretical analysis. The results reveal that layer-crack plate structure formed near the free surface of the coal wall during the loading. During the formation of the layer-crack plate structure, the lateral displacement curve of the coal wall experiences a jagged variation, which suggests the nonlinear instability failure of the coal wall with a sudden release of the elastic energy. Then, a dynamic model for the stability analysis of the layer-crack plate structure was proposed, which takes consideration of the dynamic disturbance factor. Based on the dynamic model, the criterion for dynamic instability of the layer-crack plate structure was determined and demonstrated by an example. According to the analytical results, some control methods of dynamic stability of the layer-crack plate structure was put forward.