The fish reproductive system is a complex biological system. Nonetheless, reproductive organ development is conserved, which starts with sex determination and then sex differentiation. The sex of a teleost is determined and differentiated from bipotential primordium by genetics, environmental factors, or both. These two processes are species-specific. There are several prominent genes and environmental factors involved during sex determination and differentiation. At the cellular level, most of the sex-determining genes suppress the female pathway. For environmental factors, there are temperature, density, hypoxia, pH, and social interaction. Once the sexual fate is determined, sex differentiation takes over the gonadal developmental process. Environmental factors involve activation and suppression of various male and female pathways depending on the sexual fate. Alongside these factors, the role of the brain during sex determination and differentiation remains elusive. Nonetheless, GnRH III knockout has promoted a male sex-biased population, which shows brain involvement during sex determination. During sex differentiation, LH and FSH might not affect the gonadal differentiation, but are required for regulating sex differentiation. This review discusses the role of prominent genes, environmental factors, and the brain in sex determination and differentiation across a few teleost species.