Humans, perhaps more than any other animal, rely on visible facial features for non-verbal social communication. People scan the faces of others for clues to their internal affective state and monitor their gaze-direction in order to know their current focus of attention. Moreover, these things are done automatically and efficiently. Surprisingly, people also infer enduring traits like trustworthiness from facial appearance. In this dissertation I present a series of investigations of the neural substrate for non-verbal social communication during face perception. Specifically, we examined how neural systems for the perception of emotional expression and gaze direction are organized and how they interact with spatial attention and implicit trait inferences. We first present evidence of a functional dissociation of expression and gaze perception within the posterior superior temporal sulcus. Next, we describe the results of an fMRI experiment demonstrating that gaze and arrow cues evoke a qualitatively different response in the brain's attention control systems, lending support to the notion that gaze direction represents a special class of orienting stimuli. We then turned our attention to the curious phenomenon of rapid impression formation from faces. We began our series of experiments investigating trait impressions by using a behavioral adaptation paradigm to test the hypothesis that implicit trustworthiness judgments are due to an overgeneralization of expression perception. The results support this hypothesis by demonstrating that prolonged exposure to either angry or happy expressions modulates the trustworthiness evaluations of emotionally neutral faces, suggesting a common neural substrate for these behaviors. In the subsequent experiment, we found that amygdala activity during face perception is better predicted by group consensus evaluations of facial trustworthiness than an individual's own evaluations. This suggests that the amygdala is responding to universal traits for trustworthiness rather than idiosyncratic criteria. Finally, we report differential response profiles of particular nuclei in the amygdala during gaze and implicit trustworthiness perception. Together, these results highlight the specialized nature of face processing in the brain and how that specialization can sometimes come at the cost of erroneous estimations of the enduring traits of another person.