Visual short-term and working memory can be disrupted by irrelevant, distracting input occurring after encoding. Distractors similar to the original memory are known to be interfering, but it is unclear whether dissimilar distractors have the same disruptive effect.The presence of dissimilar distraction would be problematic for views of similarity-based interference, hence the present study investigated modality-specific distraction using a procedure that required participants to compare single target and probe objects over a delay.An irrelevant distractor could be presented during the delay separating the target and probe, but it varied in its similarity to the target. In four experiments, recognition was disrupted by the presence of a distractor, even when the distractors were highly dissimilar to the target. Furthermore, the interference effect was not reduced when the same distractors were repeatedly used throughout the experiment, and interference from dissimilar distractors was only lessened when it was extremely predictable. These findings indicate that susceptibility to dissimilar distraction is a persistent limitation in visual short-term memory.