The prediction of material structure from chemical composition has been a long-standing challenge in natural science. Although there have been various methodological developments and successes with computer simulations [1][2][3][4][5][6][7][8], the prediction of crystal structures comprising more than several tens of atoms in the unit cell still remains difficult due to the many degrees of freedom, which increase exponentially with the number of atoms. Here we show that when some experimental data is available, even if it is totally insufficient for conventional structure analysis, it can be utilized to support and substantially accelerate structure simulation. In particular, we formulate a cost function based on a weighted sum of interatomic potential energies and a penalty function referred to as "crystallinity", which is defined using limited X-ray diffraction data. This method is applied to well-known polymorphs of SiO 2 with up to 96 atoms in the simulation cell to find that it reproduces the correct structures efficiently with a very limited number of diffraction peaks. The penalty function is confirmed to destabilize the local minima of the potential energy surface, which facilitates finding the correct structure. This method opens a new avenue for determining and predicting structures that are difficult to determine by conventional methods, such as surface, interface, glass, and amorphous structures.