Funding Agency: Center for Substance Abuse Prevention
Staff: Norm Constantine (principal evaluator), Kathleen Curry, Marycruz Diaz, Jeannie Huh-Kim
(This project was based at WestEd.) This project involved evaluation of the Mendocino County Department of Public Health's Safe Haven Violence Prevention Program, a five-year national demonstration program site funded by the federal Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Prevention (CSAP). We were contracted at the beginning of year 4 to design and implement a comprehensive evaluation of all program components.
The Safe Haven program integrates a classroom violence prevention curriculum (Project Yes), a peer conflict manager training program, group and individual anger management counseling for highest risk students, and community mobilization. We conducted a participatory utilization-focused evaluation, meeting monthly with an evaluation work group consisting of County and community-based organization staff, participating teachers, and paraprofessional counselors. The work group designed and implemented a developmental and outcome evaluation based on a theoretical framework developed by the work group under WestEd's facilitation. The evaluation consisted of the following five components:
1. Project Yes classroom curriculum: Randomized wait-list control design.
We developed and validated a theory-based violence prevention assessment instrument, the Safe Haven Student Survey, with elementary and middle school forms (reference #4 below). A fully randomized wait list control design was employed to detect and understand potential effects of the Project Yes component of program. This design involved pre, post, and multiple follow-up assessments and linked longitudinal individual student data. No effects of Project Yes were found on any of the attitude or behavior outcomes tested. These results were fully reported in the County program report to the federal funding agency (CSAP).
2. Peer conflict managers: Implementation evaluation.
We conducted a comprehensive implementation evaluation involving interviews and focus groups with 91 students, teachers, counselors, and principals. This involved systematic cross-site qualitative analyses of degree of integration, success factors, challenges, and participant recommendations (reference #3 below).
3. Anger management groups: Quasi-experimental pre/post design with multiple raters and cohort comparisons.
This involved a pre-post, multiple cohort, multiple rater design. Counselors administered the Social Skills Rating System (SSRS) assessment to participating students, their teachers, and their parents within one week of counseling group entry and within two weeks after the completing the group sessions. Group entry and completion were staggered, providing multiple cohorts of baseline comparisons. Assessment included several subscales each within the domains of social skills and problem behaviors. Significant positive effects were found across a variety of SSRS scales and subscales, within the teacher ratings for elementary students, and both the parent and teacher ratings for middle school students (reference #1 below). No effects were found based on student self-ratings at either grade level.
4. Community mobilization: Documentation logs and interviews.
Documentation logs were kept and analyzed for all community mobilization activities, such as radio interviews and other media activities, community meeting, etc. Interviews were conducted with key community informants to monitor the reach and potential effects of these activities. This component of the evaluation was reported in the County program report to the federal funding agency (CSAP).
5. Risk and protective factors for aggressive behavior.
In addition to the above evaluation components, a side study was conducted to examine the network of relationships between verbal and physical aggressive behavior and theoretically-derived measures of beliefs and attitudes, perceived norms, perceived efficacy, behavioral intentions, and concurrent behaviors. Consistent with and extending prior research, four factors were significant predictors of verbal aggression (beliefs about aggression, past month drug and alcohol use, and protectively, self-efficacy for nonviolent strategies and past month trusting communication); and five factors predicted levels of physical aggression (male gender, perceived norms about drugs and alcohol, past month verbal aggression, beliefs about aggression, and past month drug and alcohol use). For more details see reference #2 below.
References and Instruments