The purpose of this study was to analyze existing data collected using direct observation in a high school setting in order to understand more about the quality and contribution of physical education to public health goals. The System for Observing Fitness Instruction Time (SOFIT) was utilized to collect data related to student activity levels, lesson contexts, and teacher promotion of physical activity behavior. Two cross-sectional samples were observed in the spring 2005 and spring 2007 from seven high schools in a large urban school district in the eastern United States. In total, 164 lessons were observed yielding over 75 hours of observation. Descriptive statistics were calculated and logistic regression was utilized to determine the association between lesson contexts and student activity levels. The results showed the mean length of lessons was 29.1 minutes which translated to 32% shorter than scheduled. Students engaged in MVPA during 53% of the total intervals, however, only 13% were vigorous. Physical activity was not promoted 73% of the time, and, coincidently, a majority of the lessons did not meet public health guidelines (n = 93, 57%) and only engaged students in MVPA for 35% of the lesson length. “Skill practice” was the best predictor of MVPA (Odds Ratio = 1.7) and best source of vigorous physical activity, however was only observed in 4% of the total intervals. The dominant lesson contexts were “game play” (49%) followed by “fitness activity” (21%). Little time was spent in “knowledge” (4%). In this study environmental factors related to instructional goals (i.e., lack of knowledge, skill practice, and promotion of physical activity) and decreased lesson length diminished the quality and contribution of physical education to public health goals. The quality and contribution of high school physical education can be improved by increasing student participation in vigorous physical activity, modifying instructional goals to include more knowledge and skill related content, and increasing the promotion of physical activity. More studies should be conducted to examine the relationship between key environmental and policy influences (e.g., lesson length, time spent in contexts, professional development) on the quality and contribution of high school physical education to public health goals.