Situational judgment tests (SJTs) present examinees with a series of domain-specific situations that require a response. In the years following a seminal article by Motowidlo, Dunnette, & Carter (1990), researchers have articulated a number of SJT features: they are economical to develop and administer, offer a valid means of predicting outcomes of interest, are well-received by applicants, result in reduced adverse impact, and appear to be less fakeable than other non-cognitive measures such as personality. A growing literature suggests that the choices made when developing, administering, or researching an SJT can moderate its features. Two areas for which very little empirical guidance exists are the keying and scoring of SJTs. Although many authors have underscored the need to generate more insights in these areas for conceptual, legal, and empirical reasons, research on these topics is seldom published. The purpose of this study was to explore the effects of different keying and scoring approaches on an SJT's reliability, validity, and subgroup mean differences. The following exploratory research questions were proposed: (1) How do different answer keys compare in terms of agreement? (2) Do different keying and scoring approaches moderate the reliability and validity of SJTs? (3) How are Male-Female subgroup differences affected by keying and scoring approaches ? The data used for this study came from a nationwide sample of ROTC cadets and midshipmen (N = 153). Individual difference variables were measured for this sample (cognitive ability, Big Five personality), and respondents completed a rigorously-developed SJT assessing five dimensions of performance, which were also reflected in a criterion performance measure completed by multiple peer raters. The results of this research contribute to the science and practice of situational judgment testing by demonstrating the ways in which agreement, reliability, and validity vary as a function of keying and scoring decisions.