Recent evidence suggests that observers can rapidly form an average representation based on a set of simultaneously presented faces. Here, we replicate this finding and show that the tendency to process sets of faces in terms of an average representation is greater for own-gender faces. Male and female participants viewed sets of four male or female faces before deciding whether or not a subsequently presented single test face had been present in the set. Incorrect endorsement that it was one of the set members was greater when the test face was a morphed average of the four faces than when it was an actual set member, and this effect was strongest when the gender of the faces was the same as the observer's. The finding that observers were more likely to incorrectly endorse own-gender (vs. othergender) faces forms an exception to the often reported owngender advantage in face recognition.