The Experiment Societyexperimentsociety.org⌕Get involved
LibrariesHuman capitalMixed

New York Public Library — Teen Learning Labs Impact Study

New York Public Library / Brooklyn Public Library / Queens Public Library / Mathematica evaluation · New York City, USA · 2018

Summary

The NYPL Teen Learning Labs evaluation is one of the few rigorous studies of library-based after-school programming, and its mixed results illustrate a recurring problem in the evaluation of cultural and civic-institution programming: the outcomes participants and program staff value most (engagement, agency, peer relationships, identity development) are not the outcomes most easily measured. The study found meaningful effects on self-reported engagement, digital skills, and college aspirations, but no measurable effect on standardized test scores or attendance. Skeptics read this as evidence the program isn't 'working.' Defenders read it as evidence the program is succeeding at the things it was designed to do, while school-instruction-focused metrics miss the actual mechanism. Both readings are defensible. The study's methodological contribution was demonstrating that propensity-score-matched designs can recover credible effect estimates for library programs that cannot ethically randomize access, and the model has been adopted by subsequent evaluations of cultural-institution youth programming.

Research question

"Does providing structured, mentored, project-based after-school programming in public-library teen learning labs improve adolescent academic engagement, digital literacy, and post-secondary aspirations relative to ad-hoc library teen programming?"

Methodology

Intervention

Six branches across the three NYC library systems established dedicated 'teen learning lab' spaces with paid teen mentors, project-based curricula (digital media, coding, civic journalism, design), and credentialed staff supervision. Teens who attended at least 12 sessions over a school year were considered active participants. Comparison teens used libraries without learning labs.

Assignment

Quasi-experimental — propensity score matching of participating teens to similar non-participating teens drawn from library card-holder records and school district data; supplementary in-depth observation and survey work

Sample size

Approximately 1,800 active teen participants across the three library systems; matched comparison sample of 2,200 non-participating teens

Primary outcome

Self-reported academic engagement, digital skills assessment, school attendance, college aspirations, and program persistence

Effect estimate

Active participants showed +0.18 SD improvement in self-reported academic engagement, +0.25 SD on digital skills assessment, and a 9 percentage point increase in expressing college aspirations vs. matched comparison. School attendance effects were small (~1 day/year) and non-significant. No effect on standardized test scores.

Decision

Findings supported continued IMLS and city-level investment in teen learning lab infrastructure. The model has been adopted by other large urban library systems (Chicago, Los Angeles, Philadelphia, Boston). The lack of standardized test-score effects has been interpreted variously as evidence against expecting library programming to substitute for school instruction, or as evidence that the program's value lies in non-academic outcomes that traditional evaluation undercounts.

Result

Mixed

Active participants showed +0.18 SD improvement in self-reported academic engagement, +0.25 SD on digital skills assessment, and a 9 percentage point increase in expressing college aspirations vs. matched comparison. School attendance effects were small (~1 day/year) and non-significant. No effect on standardized test scores.

Evidence strength

Moderate

Quasi-experimental design; causal interpretation requires care.

Replication status

Open for replication

Institution

New York Public Library / Brooklyn Public Library / Queens Public Library / Mathematica evaluation

Location

New York City, USA

Year

2018

Policy area

Libraries

Mechanism

Human capital

More from these institutions

Other trials of this mechanism

Cite this entry

New York Public Library / Brooklyn Public Library / Queens Public Library / Mathematica evaluation. (2018). New York Public Library — Teen Learning Labs Impact Study. The Experiment Society Registry. Retrieved from https://www.experimentsociety.org/registry/nypl-teen-learning-labs (primary report: https://www.mathematica.org/our-publications-and-findings/projects/teen-services-in-public-libraries)

Citation network

Cited by 1 other entry in the registry

Cross-references

Adjacent experiments — same domain, parallel pilots, or alternative mechanisms.

Related experiments

Volusia County, FL; Charlottesville, VA; Rochester, NY (USA) · 2007

Self-Selected Summer Books and Summer Reading Loss

Positive

United States (multiple cities) · 2018

Year Up Job Training Evaluation

Positive

United States · 2018

Year Up Young Adult Job Training Program

Positive

Open for replication

Could your jurisdiction test this?

This finding has not yet been independently replicated. A replication in a new context — a different city, country, or population — is one of the highest-value contributions to the evidence base. If your team could host a replication, we can help with study design, sample-size estimation, and analysis.

Propose a replication →What does "open for replication" mean?
← Back to registryWhat the evidence shows →