New Research Investigates How & Why Summer Learning Impacts Middle School Student Achievement
March 12, 2015 - Dorchester, MA - A new report was published today about BELL's summer learning model for middle school students. The report, An Analysis of the Effects of an Academic Summer Program for Middle School Students, details preliminary effect and implementation findings of a randomized controlled trial conducted by MDRC in 2012.
The study aimed to achieve the following goals:
- Build on the knowledge base about how time for learning can impact middle school student achievement.
- Determine if and how BELL and its school partners replicated the summer learning model with consistency and quality.
- Better understand the context in which these summer programs are implemented.
Math: The pattern of math impacts suggests that upon returning to school in the fall of 2012, BELL students had stronger math skills than the control group - equivalent to one month's academic learning compared to the control group. This impact is equivalent to what one would expect from a five-week program. While the study was not sufficiently powered to determine statistical significance, the math findings are promising given patterns of student underperformance in middle school.
Reading: In the average study district in fall of 2012, BELL students performed the same as controls on reading achievement tests.
The BELL Summer program was well implemented relative to the nonprofit's middle school model and national program quality standards, including those recommended by RAND Corporation and the National Summer Learning Association. The teaching staff was highly qualified: 70% of teachers had a master's or doctorate degree; 89% have five or more years of teaching experience.
Students attended the full-day, five-week program at a high rate even though the program was voluntary. In the average study district in Summer 2012, the attendance rate among program participants was 82 percent, above the program's target of 80 percent. Positive math impacts correlated with increased time engaging in math activities. Surprisingly, the control group reported engaging in reading and writing as frequently as the treatment group.
Despite the study's rigorous design, several limitations mean that the findings should be considered preliminary and suggestive, and are not generalizable to BELL's model as it exists today. For example, the study's sample size was smaller than expected, which diminishes the power of the study to detect realistic impacts of a five-week program. As such, the findings can and should be used to generate hypotheses that could be more definitively tested in a future evaluation, and applied to BELL's continuous improvement cycle.
"To date, there has been very little evidence on the effectiveness of summer academic programs for middle school students, especially programs in which participation is voluntary," said Gordon L. Berlin, President of MDRC. "Thus, the present study is important not only to BELL but also to leaders of other middle school summer programs."
The findings from MDRC's study are being used to bolster BELL's ongoing efforts to continuously improve program quality and scholar outcomes. In addition to the RCT conducted in 2012, BELL completed an earlier RCT and regularly collects data from pre- and post-program formative assessments, scholar, teacher, and parent surveys, engagement indicators such as scholar attendance and retention, and quality metrics such as classroom size and teacher engagement. Since the study was implemented in 2012, BELL and its school partners have enhanced training, curriculum, and assessment in an effort to further improve program quality and scholar outcomes. Assessment data from the summers of 2013 and 2014 suggest that these enhancements are working.
BELL continues to work with MDRC, its Evaluation Advisory Board, its new Middle School Advisory Board, and experts in the field to advance an ongoing learning agenda over the course of the next three years. Emerging questions include:
- What is the counterfactual experience of disadvantaged middle school youth that do not participate in BELL Summer?
- Which outcomes and what measures best capture the benefits of BELL's intervention during the summer months?
- How can promising contextual adaptations to our model be replicated?
- How can a five or six-week program be optimized for the greatest level of engagement among this group and the greatest impact?
The MDRC evaluation was funded by a Social Innovation Fund grant from the Corporation for National & Community Service and the Edna McConnell Clark Foundation, as well as several co-investors including The Wallace Foundation and others.