Abstract:

As evaluation continues to develop as a transdiscipline focused on applied social science methodology, new developments are rapidly occurring in evaluation theory, validity theory and the use of program theory in evaluation practice. Despite this intellectual growth, no one has attempted to provide a unified vision for the role of each of these domains of theory in both improving the practice of effectiveness evaluation and addressing unresolved epistemological and methodical issues. This study integrates evaluation theory, validity theory and program theory to develop an expanded definition of validity and propose a new model of conducting effectiveness evaluations that promises to increase the validity of the evaluation. Key characteristics of this model are: the substantive participation of a diverse group of stakeholders, the use of a program or intervention theory to guide the investigation for effectiveness, and the application of growth mixture modeling for data analyses. After briefly tracing the historical development of the modern effectiveness evaluation, paying close attention to the key issues for identifying causal relationships between interventions and outcomes, I propose an expanded definition of validity that incorporates both the traditional validity typologies associated with experimental design and newer conceptions of validity that consider the social consequences and cultural context of evaluation. Based upon this expanded definition of validity, I detail a model of effectiveness evaluation that I am tentatively calling participatory, theory-based effectiveness evaluation (PTBEE). Then, I report the empirical findings from a PTBEE of a child and adolescent managed mental health care intervention. Next, I share practice-based reflections on how to improve the implementation of a PTBEE. Lastly, I conclude the text with a discussion of two key issues that emerged from the study--defining validity and the role of theory building in applied research.