Morning Zen Guest Blog Posty ~Mansoor A. F. Kazi, PhD

To this day, most Systems of Care have focused on at-risk groups rather than the total school populations. These demonstrations have utilized big data continuously on entire school populations from the SAMHSA-funded SOC expansion in Chautauqua and Rockland counties (NY) and Manchester City Council (UK). Methods included a nonequivalent comparison group as well as matched quasi-experimental designs, combined with logistic regression to investigate what interventions worked and for whom, in real time. The primary purpose of the last 11 discussion demonstrations at the Research and Policy Conference on Child, Adolescent and Young Adult Behavior, held in Tampa, Florida each year, has been to show how we can continuously evaluate System of Care Expansion by repeatedly combining big data from all SOC human service agencies and school districts. A presentation on this research is being offered at the upcoming University of Maryland, Baltimore Training Institutes.

Research methods drawn from both epidemiology and effectiveness research traditions are demonstrated in a realist evaluation in partnership with human service agencies and the schools to investigate what programs of intervention work and for whom. Real live data from management information systems (schools, social services, mental health, youth justice) is used to investigate the effectiveness of the human service interventions. As the emphasis is on data naturally drawn from practice, quasi-experimental designs are demonstrated using demographic variables to match intervention and non-intervention groups. Binary logistic and linear regression are demonstrated as part of epidemiologic evidence based on association, environmental equivalence, and population equivalence.

Evaluators and agencies can make the best use of the available data to inform practice. Realist evaluation essentially involves the systematic analysis of data on 1) the service users’ circumstances; 2) the dosage, duration and frequency of each intervention in relation to each user; and 3) the repeated use of reliable outcome measures with each service user. The workshop being held at theUniversity of Maryland, Baltimore Training Institutes will show how evaluators work in partnership with these agencies, to clean the data, undertake data analysis with them at regular intervals and not just at the end of the year. In this way, the evaluators and the human service agencies can work together to evaluate the impact of interventions on the desired outcomes utilizing innovative methods and addressing issues relevant for practice including diversity and investigating where and with whom the interventions are more or less effective in real time. Establishing cause and effect is a particular theme of this demonstration. As the data mining includes all service users (e.g., all school children in all participating school districts), it is possible to investigate the differences in outcomes between intervention and nonintervention groups, and these groups can be matched using the demographic and contextual data. The innovative methods demonstrated using the same data would include those that are part of the family of methods used to determine epidemiologic evidence based on association, environmental equivalence, and population equivalence. For example, the presenters use datasets from their completed evaluations from Manchester and New York State and discuss real-world applications of the analyses. The didactic approach has been interactive, guiding the workshop participants through the requirements and limitations of each method. Binary logistic regression is used to investigate what interventions work and in what circumstances. In each example, the variables that may be influencing the outcome are identified through bivariate analysis and then entered into a forward-conditional model. The variables that are influencing the outcome are retained in the equation, and those that are significant provide an exponential beta indicating the odds of the intervention achieving the outcome where the significant factor(s) may be present. The interactive live demonstrations investigate where an intervention is more or less likely to be effective, and how to utilize findings and inform practice on demand.

Chautauqua Tapestry (NY) received SAMHSA’s Gold Award for Outstanding Local Evaluation, in 2010 having used realist evaluation strategies in schools, mental health, social services, youth justice and other human service agencies. Tapestry partners utilize these findings and inform practice in real time to better target and to develop all human services to meet the needs of the local communities. Realist Evaluation Inc. also provides similar services to Rockland County (NY) and Manchester City (England).

* * * * * * *

Mansoor A. F. Kazi, PhD is Director Program Evaluation Center, Fredonia State University of New York, Fredonia, NY. and President of Realistic Evaluation Inc. For the past ten years, Dr. Kazi has presented research findings at the University of South Florida’s Annual Research & Policy Conference on Child, Adolescent, and Young Adult Behavioral Health. Co-presenters have included Patricia Brinkman and Rachel Ludwig from Chautauqua County NY; Janet Sliva and Susan Hoerter from Rockland County  NY; Marie McLaughlin, Director of Manchester Youth Justice Service (England): and research assistant Yeongbin Kim.