Abstract:

Three traditional methods for detecting DIF were investigated to determine their effectiveness with testlet data. In addition, an extended logistic regression and a newly proposed, modified Mantel-Haenszel procedure that account for item dependence were also investigated. Results showed that Type I error rates of these methods were unaffected by the presence of testlet data. In terms of power, DIF detection rates were the highest for the easy test item, as compared to the medium and high difficulty items. As expected, in general, the inclusion of a testlet effect resulted in a loss of power. The exception to this was the extended logistic regression procedure therefore, this method is preferred for use when investigating DIF in locally dependent data. The new Mantel-Haenszel procedure did not result in better power rates than the traditional Mantel-Haenszel procedure. Results from an empirical data study supported the findings of the simulation study in that high levels of agreement were observed in flagging DIF items across all five methods. Ignoring item dependence and using traditional methods to assess for DIF with testlets is the norm. This study set forth to determine if DIF methods are robust to violations of item independence. It was found that traditional methods are robust when sample sizes are large and DIF is moderate to large. When sample sizes and DIF are small it is better to directly model both a testlet and DIF effect in order to increase the power to detect DIF as traditional methods have less power than methods that model the testlet and DIF effect separately.