The ability to identify an unfamiliar target face from an identity lineup declines when the target is accompanied by a second face during visual encoding. This two-face disadvantage is still little studied and its basis remains poorly understood. We investigated several possible explanations for this phenomenon. Experiments 1 and 2 varied the number of potential targets (1 or 2) and the number of faces in a lineup (5 or 10) to explore if this effect arises from the number of identity comparisons that need to be made to detect a target in a lineup. We also explored if this effect arises from an uncertainty concerning which is the to-be-identified target in two-face displays, by cueing the relevant face during encoding. In experiment 3 we then examined whether the two-face disadvantage reflects the depth of face encoding or a memory effect. The results show that this effect arises from the additional comparisons that are necessary to compare two potential targets to an identity lineup when memory demands are minimised (experiment 1), but it reflects a difficulty in remembering several faces when targets and lineups cannot be viewed simultaneously (experiments 2 and 3). However, in both cases the two-face disadvantage could not be eliminated fully by cueing the target. This hints at a further possible locus for this effect, which might reflect perceptual interference during the initial encoding of the target. The implications of these findings are discussed.