The purpose of this study was to determine the usefulness of using annotated concept maps as predictors of a teacher’s performance on complex and ill-structured problems. Specifically it looked at whether annotated concept map assessments correlate with teacher performance and understanding of complex and ill-structured problems, and can metrics that do not examine the content of nodes and links in a concept map be used to diagnose a relationship to expertise. In order to capture and evaluate the annotated concept maps created by the teachers a Web-based tool suite called HIMATT was used.

Thirty-two K-8 teachers were asked to create annotated concept maps responding to how they think about two complex problem scenarios they were given using the HIMATT tool suite. These teacher maps were then analyzed with six different measures included in the tool suite to determine the similarity between the maps and an expert reference map. The expert reference map was created by five recognized experts who contributed individual maps responding to the scenarios and then worked together to make one expert map from each of their contributions. The similarity measures were then compared with two independent variables used in this study: pedagogical experience (Novice or Expert) and technology integration experience, as well as looking at the correlations with student standardized test scores.

The results of this study indicated that there are distinct and significant differences on several of the similarity measures when compared to the expert reference map on each of the independent variables. One measure in particular, Gamma, was significantly different both independent variables. When looking at pedagogical expertise there was a significant difference on the Graphical Matching, Gamma, and Structural Matching dependent variables. When looking at the correlations between student standardized test scores and the dependent variables, Concept Matching and Gamma were statistically significant. Finally when looking at the level of technology integration variable, the Graphical Matching and Gamma measures reached statistical significance. These findings indicate the annotated concept maps may be useful in assessing a teacher’s level of understanding and performance on complex problem scenarios.