Using molecular dynamics simulation, we examine the dynamics of crystal, polycrystal, and glass in a Lennard-Jones binary mixture composed of small and large particles in two dimensions. The crossovers occur among these states as the composition c is varied at fixed size ratio. Shear is applied to a system of 9000 particles in contact with moving boundary layers composed of 1800 particles. The particle configurations are visualized with a sixfold orientation angle αj(t) and a disorder variable Dj(t) defined for particle j, where the latter represents the deviation from hexagonal order. Fundamental plastic elements are classified into dislocation gliding and grain boundary sliding. At any c, large-scale yielding events occur on the acoustic time scale. Moreover, they multiply occur in narrow fragile areas, forming shear bands. The dynamics of plastic flow is highly hierarchical with a wide range of time scales for slow shearing. We also clarify the relationship between the shear stress averaged in the bulk region and the wall stress applied at the boundaries.