You’ve carefully planned your evaluation, collected data and compiled results. Now what? Brock University’s Corliss Bean and Caroline Hummell spoke with Chris Penrose of Lay-Up Youth Basketball for SIRC’s Experts in the House mini-series on Mastering the Art of Evaluation, presented in partnership with Brock University. As Director of Programs and Operations of Lay-Up Youth Basketball, Penrose shared insights and learnings from Lay-Up’s evaluation findings. Chris also shared how his team has made meaningful program improvements and communicated the findings to stakeholders.
Q: What’s the mentality around evaluation in your organization?
We’re always working to improve programming by taking an innovative approach, and evaluation helps us innovate. We want our approaches to say: Basketball is one of the most accessible and inclusive sports. But it could be more inclusive, more accessible, and help build life skills.
So, how can we do this more effectively? How do we develop the model in a way that taps into how inclusive, how accessible and how effective basketball can be at building these skills?
For us, it’s not just about learning from the evidence, but also learning about and reflecting on our evaluation practices. For example, through registration we were collecting broad demographic information by asking many questions such as about our participants’ immigration and employment status. Through evaluation we discovered that the grant that required us to report on recently immigrated youth is no longer in place. Now that we know there’s no need for status information, we’ve stopped asking unnecessary questions and collecting extra information.
Reflection is important. We’re making sure we work with key partners to develop an evaluation plan, so we know why we’re collecting information and we’re questioning whether we should collect it.
Q: Describe the role that evaluation has played in Lay-Up throughout the pandemic.
It hasn’t taken a backseat for us, although it easily could have. You can rationalize that it should have, but it hasn’t for us. We didn’t want to stop or slow down that process. This year, in the middle of our 12-week program, we had an evaluation period. The purpose of that was, we were adapting so many programs, piloting so many things and we’ve all these variations. We needed that time to gather ourselves, reflect on what we’re learning, what’s working and what’s not working. That’s never happened before under our regular programming.
It became more important because of how much uncertainty, change and adaptation has happened. When you’re just trying to survive, evaluation is often the last thing on your mind. But when you’re navigating so much change, evaluation becomes more important.
What’s the best practice at the end of the day, right? If you’re piloting something new and running it for a while, you need to have some process for reflection to decide if you continue to do this. If so, what’s working, what’s not, and how do you improve? So, we’re piloting a bunch of new programs and need to have some process for evaluation and reflection.
Q; What’s your process around learning from and communicating about your evaluation findings?
Communicating evaluation findings is like giving a TED Talk. The story you should tell is inside of you. You’ve got to find it, write it down, and get it to a point where it’s clear and someone else can engage with it, and internalize it. When you talk about evaluation and the different effects of organizations, we often look at evaluation as always being on paper, as something external that must be done a certain way.
Is there a process to move beyond the paper and internalize evaluation? Because I think that’s what’s going to bring value. That is, if we have a deeper understanding of evaluation that’s internalized, we can adapt our situation and our organization using best practices. We need to tell the stories about our work and how that influences participants and communities. Talk to anyone in an organization doing something that affects people in the world, so they know it, they care and see it. But how do we help people find what that is and articulate it in a way that’s clear and tells the story? Then internalize their story so that you really understand the impact and can share it. So, for us, how can we share our stories of what organizations are going through and how they approach evaluation to help other people beyond just our organization?
Q: What are some storytelling strategies that worked well for Lay-Up?
What worked really well was we did program documentation during the pandemic, including capturing videos and photography of our in-person program. We turned this into a virtual photography exhibition called At Home in the Game. A photographer visited 12 participants in their individual neighbourhoods and took outdoor portraits of each participant. Then, coaches interviewed each of them on Zoom, and we recorded their interviews. Next, we pulled quotes from those conversations and paired them with the images. That did a lot in terms of evaluation. For example, it documented the time we’re living through. It gave an interesting snapshot about our reach as a place-based organization in various Toronto neighbourhoods across the city. We also gained insight into the programming’s impact on the value of relationships between coaches and participants. Then, we used creative design, storytelling and social media to mobilize that knowledge to a range of audiences from participants and families, partners and stakeholders, and folks in the basketball and creative community who might not go read an evaluation report in summary. This was a successful example about approaching evaluation differently.
Q: What’s an example of how you use your evaluation findings? Who do you communicate them to?
One thing we’re interested in and have started developing is an entry assessment of the participants’ fundamental basketball skills. For example, we actually measure sport skills at the outset, like each participants’ dribbling, “What’s your dribbling with your left hand at your waist, and…dribbling with your right hand at your knee?,” “doing a lay-up?” and so on. And then we’re able to show the kids “Look at the beginning of the summer, this is what you could do. Check out your growth [by summer’s end].” Using this approach feeds into some important life skills.
I think that sport has unique evaluation opportunities in that you can measure things like confidence. You might not spot that change in confidence unless you measure these skills at baseline and share that information with the kids. Then you can use this information to ask how they feel about their improvement
About Lay-Up
Lay-Up is a cost-free, community-based basketball program designed for children and youth in Toronto’s Neighbourhood Improvement Areas. Programs are delivered year-round to children and youth, ages 6 to 14, of all skill levels. Lay-Up’s mission is “to harness the power of the sport and culture of basketball to develop the skills and confidence children and youth need to navigate their future.” The Lay-Up team is committed to providing experiences, activities and learning opportunities that help kids develop the skills they need on and off the court.
This blog and associated research were possible because of support from the Social Sciences and Humanities Research Council and the Government of Canada.