News – Blog
Assessing our impact: How we measure our progress
9 March 2022
In this post, we summarise the overarching process for how we challenge ourselves to understand the impact we have through our work, and how we seek the perspectives of others to improve our understanding of how challenge prizes can make a difference.
The core of Challenge Works work is to support innovators to create and implement breakthrough solutions to pressing social problems. Given this purpose, it’s vital that we assess our contribution to achieving this goal.
Recently, we have refreshed our theory of change. This guides what we do as an organisation and sets out where we think we can have the biggest impact in supporting socially valuable innovation – based on three core areas of: creating breakthrough innovations, helping innovators thrive and unlocking systemic change.
Refreshing our theory of change
Whilst theory on the role of challenge prizes in spurring innovation is relatively well-developed, empirical evidence from actual prizes about how challenges work is less so.
At Challenge Works, we are addressing this gap by working to gather robust insights from the prizes we develop, deliver and support. Central to this process is our theory of change, which sets out the rationale and intentions that drive our delivery and provision of challenge prizes.
We use the theory of change to provide a shared understanding of what we do, and to make clear the different hypotheses, assumptions and activities that underpin our work and how they are linked to high-quality delivery of challenge prizes and positive outcomes.
To support the assessment of our challenge prize methodology, we also draw on a measurement framework that incorporates the three key impact areas. Our measurement framework is intended to be sufficiently broad to work across all our challenge prizes, (whilst recognising that prizes have unique goals and work to specific contexts). These tools act as the foundation to our data collection and analysis / synthesis.
Our approach to data collection is open and broad, including:
- Independent evaluation of our challenge prizes;
- Programme monitoring and evaluation reports;
- Information from technical experts, assessors and judges;
- Case studies that capture stories of change for prize participants; and
- Knowledge and experiences from the Challenge Works prize teams.
The challenges in assessing challenge prizes
Building an evidence-base for an intervention that tends to be complex and involves diverse participants, with diverse motivations, working on diverse solutions presents numerous challenges. Evaluating complex programmes and then attributing impact is not straightforward.
Challenge prizes are also foremost mechanisms for stimulating and supporting solutions to real-world social problems. As a result, the process of identifying suitable points for collecting data and evidence is itself a complex task that requires a balanced approach.
Some outcomes and impacts are more difficult to measure, and it can be problematic to disentangle the many social, economic, organisational and policy factors that could also be having an influence in an area from the direct effects of a challenge prize.
Despite these challenges, we are committed to collecting the most robust evidence that is possible and identifying how we can improve our proof points, where we recognise that the type and standard of evidence we have is variable.
Challenge Works are pursuing opportunities to run experiments that test some of our core thinking about prizes – and would welcome connecting with other organisations interested in trying out different approaches to evidencing the impact of challenge prizes.
