PremiseWhat maintains mixed mating is an evolutionary enigma. Cleistogamy, the production of both potentially outcrossing chasmogamous, and obligately selfing cleistogamous flowers on the same individual plant, is an excellent system to study the costs of selfing. Inbreeding depression can prevent the evolution of greater selfing within populations, and heterosis in crosses between populations may further tip the balance in favor of outcrossing. Few empirical estimates of inbreeding depression and heterosis in the same system exist for cleistogamous species.MethodsWe investigate the potential costs of selfing by quantifying inbreeding depression and heterosis in three populations of the cleistogamous perennial Ruellia humilis Nutt (Acanthaceae). We performed hand‐pollinations to self, and outcross within and between populations, and measured seed number, germination, total flower production, and estimated cumulative fitness for the resulting progeny in a greenhouse experiment.Key resultsWe found moderate inbreeding depression for cumulative fitness (<30%) in two populations, but outbreeding depression for crosses within a third population (‐26%). For between population crosses, there was weak to modest heterosis (11‐47%) in two of the population combinations, but modest to strong outbreeding depression (‐21 to ‐71%) in the other four combinations.ConclusionsNeither inbreeding depression nor heterosis was of sufficient magnitude to explain the continued production of CH flowers given the relative energetic advantage of CL flowers previously estimated for these populations. Outbreeding depression either within or between populations makes the maintenance of chasmogamous flowers even harder to explain. More information is needed on the genetic basis of cleistogamy in order to resolve this conundrum.This article is protected by copyright. All rights reserved.