Program Implementation
Implementation Implementation
An issue that is pervasive to all universal programs, regardless of theoretical orientation is the issue of program implementation fidelity, or implementation quality. Implementation quality is the degree to which a program was implemented as designed (O’Donnell, 2008). One of the key recommendations of the Kids Matter project described above was that attention be given to the quality of program implementation (Slee et al., 2011). Implementation quality results are important to demonstrate that the program was delivered as designed, supporting expectations that similar results will be obtained if the intervention is implemented with fidelity in other schools.
Berman and McLaughlin (1976) offered one of the first examinations of school- based program evaluations, with key recommendations supporting the need for implementation quality that are still relevant today (Durlak, 2016; Sanetti & DiGennaro Reed, 2012; Zvoch, 2012). Berman and McLaughlin (1976) argued that exact implementation is generally unattainable in a school setting and suggested that a more realistic expectation is that a school and a program both adapt. They noted that stringent structures within a school can lead to program changes in the duration of delivery, methods used or program format. They recommended that such adjustments be recorded and reported to help readers assess the outcomes of an evaluation. Hence, implementation quality measures record the degree to which the program was delivered as designed, and help to establish whether evaluation results can be attributed to the intervention itself, or other unintentional influences (Moncher & Prinz, 1991).
Realistically, schools operate as macrosystems with competing demands that can influence decisions about the way an intervention is implemented within a school. For example, the recommended time and duration of delivery may be altered due to competing curriculum demands. The frequency of delivery, access to required equipment, ideal location or class size might be compromised. In addition, those who implement the intervention might create variation because of their prior teaching experiences, levels of training and individual engagement practices (Durlak, 2015). Variations can also occur due to differences in student participation, that is, a cohort might participate with varying degrees of attendance, capabilities and engagement. Furthermore, identification of learning needs or engagement issues can lead a highly skilled facilitator to modify a program to support the individual needs of their students. This type of modification can aid participant outcomes, however if unreported cannot be replicated. Hence, deviation from an intervention program design is highly likely within a school context, can impact participant outcomes negatively or positively, and therefore needs consideration when evaluating a program within a school (Durlak, 2015).
Measuring Implementation Quality
There are several strategies that can be incorporated into a school-based intervention evaluation to enhance implementation quality and to record implementation quality data (Bellg et al., 2004; Domitrovich & Greenberg, 2000). Key implementation promotion elements include: a training manual that articulates the fundamental aspects of the intervention, training and consultation, explanation of program features which could be adapted without compromising the integrity of the intervention, and provisions for internally created material that will be a best-fit with a specific population rather than fixed resources (Bellg et al., 2004). To evaluate a school-based intervention, it is also recommended that multiple schools and multiple facilitators are recruited (Durlak, 2016).
Implementation quality has been discussed as a research agenda since 1976 (Berman & McLaughlin, 1976), however uptake of implementation quality measures in school-based program evaluations has been gradual (Low et al., 2016). Gresham and Gansle (1993) reported that of the 181 published studies of school-based evaluations between 1980 and 1990, fewer than 14% mentioned implementation quality data. Dane and Schnider (1998) examined 162 studies between 1980 to 1994 and reported that only 24% included implementation quality data. They examined the extent to which evaluations reported on five core measures of implementation quality; Exposure (number, length and frequency of sessions), Adherence (extent to which program components were delivered as described), Quality of Delivery (capturing qualitative measures such as implementer enthusiasm and preparedness), Participant Responsiveness (levels of participation or enthusiasm), and a measure to ensure participants receive only the planned interventions. Dane and Schnider (1998) noted that of these measures, Exposure and Adherence were most commonly reported in the studies they had examined. They also reported that low adherence to the intervention was often associated with poorer outcomes. They suggested that limited reporting of implementation quality may stem from a lack of theory to guide researchers. This suggestion was echoed by O’Donnell (2008) following a similar examination of kindergarten to twelfth grade interventions.
To identify barriers to the inclusion and reporting of implementation quality data, Hagermoser et al. (2012) surveyed authors of 210 school-based evaluation papers published between 1995 and 2008. The school psychologists surveyed reported that a lack of consistent theory, general knowledge and explicit guidelines about quality of implementation research posed significant barriers to inclusion of implementation data in evaluative studies. Durlak noted that school-based interventions are currently lacking clear direction for accurately assessing implementation quality (2016). The school psychologists surveyed also indicated that time limitations, costs, labour demands and lack of editorial requirements from major journals detracted from the pursuit of quality implementation. To date, journals, self-ratings and observation are the most common procedures reported.
In spite of the difficulties described, there are examples of effective implementation quality measures in school-based evaluations. The Multisite Violence Prevention Project (2008) is an example of a school-based evaluation where implementation quality promotion procedures subsequently contributed to a more comprehensive understanding of the outcomes which varied between schools. The study design required numerous facilitators each to complete 36 hours of training. Thirty-seven schools were included in the study, with training workshops and consultation offered to class teachers throughout the year. Implementation was delivered in accordance with a 20-session curriculum that focussed on social information processing skills taught through role play, group activities and experiential learning techniques. Self-report checklists from facilitators indicated that most (95%) of the curriculum was delivered as designed, indicating high implementation quality. However, not all of the schools involved demonstrated change for their students, and this appeared to occur particularly in the schools that did not take up the offer for on- going consultation, training or supervision throughout the year. In this instance the recording of these implementation measures helped identify why the same results were not achieved by all participating schools (The Multisite Violence Prevention Project, 2008).
Durlak (2016) also stressed the importance of identifying a school’s readiness for a program. He suggested first assessing the specific need for the intervention and identifying program fit with the school’s values and resources to gain “genuine buy-in” from a school. Durlak also suggested that decisions regarding adaptation should be guided by the researcher, with a dedicated internal team to help oversee the implementation of the program. According to Durlak only after these factors are assured are staff ready for program training.
Reporting implementation data heightens capacity to replicate an intervention, and helps the reader to interpret the results of a school-based evaluation. This latter point can be illustrated with reference to an evaluation of the Zippy’s Friends program (Clarke et al., 2014). This is an SEL framework universal program aimed to improve children’s coping skills, problem solving skills, and promote mental health and emotional wellbeing. The program is delivered by the class teacher for 1 hour a week over 24 weeks.
The evaluation examined the impact of the Zippy’s Friends program across 52 classrooms. General psycho-social behaviour was measured pre- and post- intervention using the SDQ (Goodman, 1997), and participants’ emotional literacy was measured using the Emotional Literacy: Assessment and Intervention – Ages 7 to 11 (Faupel, 2003). This is a 20-item checklist completed by teachers regarding general behaviour. Implementation was measured using a implementation quality checklist completed by the class teacher after each lesson, to identify the degree of adherence to the session elements. The researcher also conducted class observations and completed the checklist. Agreement between the two raters was high (Kappa Measure 0.74, p<0.005).
The SDQ results from 52 classrooms indicated that general social-emotional capabilities did not improve, however, on measures of emotional literacy, participants skills did improve following program participation. Furthermore, program implementation quality was high across most classrooms, and higher levels of implementation quality equated to higher post-intervention emotional literacy scores. These findings highlight that adherence to a program, directly impacts student outcomes.
The quality of implementation of the Second Step Violence Prevention (SSVP) program (Committee for Children, 2012) was formally examined by Low et al. (2016). This is a widely used and evaluated universal program that aligns with the Social Competence and SEL frameworks and is designed to reduce overt aggression. The program teaches skills in social problem solving, empathy and managing emotions. The SSVP program includes 24 scripted lessons that are designed to be delivered for 30 minutes once a week.
A longitudinal evaluation was conducted, including 61 schools and 7300 student participants. Schools were either assigned to the early start or delayed start condition. Teachers completed several measures including the SDQ teacher-form (Goodman, 1997) as measures of participants’ psychosocial behaviour. The SDQ results allowed the participants to be identified as socially competent or at-risk. Analysis of the SDQ results post-intervention indicated differential program impact. The benefits of the program were most evident for children who were at or lower than the 50th percentile for social competency on the pre-intervention measure.
In total, 160 teachers were included in the evaluation and each of the teachers was asked to provide information on the implementation quality. Teachers completed a weekly self-report rating on adherence, engagement, generalisation to the broader curriculum and exposure (duration). Adherence items included questions such as “to what extent did you leave out parts of the lesson?” (Low et al., 2016, p. 984). Engagement items rated the degree to which students engaged. For example, “to what extent were students following along with the lesson?” (Low et al., 2016, p. 984). Generalisation measured reinforcement of themes and integration into daily activities for example “in the last week, to what extend did you do things to integrate Second Step skills into your academic lessons, as highlighted in the Academic Integration section?” (Low et al., 2016, p. 984). Exposure was measured by asking teachers to record how many lessons they completed by the end of the year. Overall, the implementation quality results suggested that indicators of adequate exposure and high levels of engagement were most closely associated with positive student outcomes (Low et al., 2016, p. 989).
Generally, we know that schools are busy hives of competing critical demands. The use of implementation questionnaires weekly is not typically sustainable for the average facilitator. However, implementation fidelity is an integral component of any formal evaluation research, and serves to reinforce the importance of delivering an intervention program as designed. In order to achieve program outcomes comparable to those that caught your attention in the first place, program fidelity during program delivering is essential.
