In the 1950s, Dutch ethologist Nikolaas Tinbergen defined the four essential questions for understanding the mechanisms of behavior. Two concerned ultimate causation: the evolution of a trait in a phylogenetic context, and the adaptive value of a trait. Two concerned proximate mechanisms: the physiological cause of the trait, and the developmental ontogeny of the trait. This framework formed the cornerstone of the integrative study of behavior. Here, I use this approach to investigate aspects of social behavior in mammals, and in particular house mice (Mus musculus). The mouse is an ideal model organism for the study of mammalian behavior because it has a well characterized natural history and abundant molecular and genetic resources. Social communication (e.g., mate choice, intrasexual competition and kin recognition) in the mouse is primarily mediated by olfaction, and two genetically encoded pheromones critical to this process are the major histocompatibility complex (MHC) and the major urinary proteins (MUPs).

In this dissertation, I first review literature on the role of MHC-mediated social behaviors in vertebrates, with special attention to the mouse. I show that MHC is widely used for the identification of kin, but mechanistic details differ across ecological and phylogenetic space. Second, I use a selection experiment in house mice to experimentally manipulate social competition and sexual selection to determine the ultimate fitness consequences of these behavioral processes. I show that when these processes are operating a male phenotype characterized by increased sexual attractiveness is quickly favored. Third, I show that this sexually attractive phenotype is driven by upregulation of MUP expression in a transgenerational fashion: parents who experience social competition have sons that have upregulated MUPs. An assessment of proximate mechanisms driving this phenotype indicates that it is not due to changes in hormone expression, but may be due to a loss of DNA methylation in the Mup promoter. I also provide evidence that upregulation of MUPs per se is a female attractant. Together, these studies show that MUPs and MHC have substantial impact on social interactions in mammals, and provide a rich system in which to understand the molecular and genetic basis of behavior.