Why challenge prizes need pre-launch estimates of leverage, not blanket targets

The directive that agencies should aim for at least a 3:1 leverage of private to federal investment was likely glossed over by a majority of readers; even those that made it to the annexes. But for someone whose primary interest is understanding the impact of challenge prizes and thinking about how we can generate the evidence to support more effective use of pull mechanisms in funding innovation, it prompted lots of thoughts. 

Prize leverage – the multiplier effect that occurs when a challenge prize mobilises additional financial resources that exceed the monetary value of the prize purse itself – is arguably one of the mechanism’s key effects. Prize leverage ratios can be significant; the estimate for the Ansari X Prize was a ratio of 40:1 based on a competition with a $10 million prize purse, while the DARPA Grand Challenge reported a 50:1 ratio from a $2 million cash purse. On that basis, a 3:1 minimum-floor target for federal prizes could be seen as a low-bar and facilitative to the use of more prizes. But the issue isn’t really about the 3:1 ratio itself (though it would be helpful to understand more on the thinking behind it); the core issue is actually how we construct and report on prize leverage as a measure.

This opacity on how prize leverage is defined and reported exacerbates a fundamental problem in the field – despite the resurgence in the use of prizes by governments and philanthropic funders, there is still no standardised evaluation framework that can help to build and collate robust evidence for how prizes work and when they should be used.

What, then, would be a better approach for policymakers to take that goes beyond the setting of arbitrary benchmarks and moves towards more of a metascience-based approach to estimating, measuring and targeting prize leverage? I think we need to start with a three-fold approach that does the following: provide rigorous definitions in presenting data on prize leverage ratios, and disaggregate between different types of leverage where necessary; develop models that enable us to develop ex-ante (pre launch) estimates of prize leverage pre-competition launch; and systematically measure ex-post (post award) leverages, applying defendable attribution factors. This would leave us much better placed to understand which prizes have under or over-performed in terms of leverage and to explore the factors that might explain performance on this key measure. Over time, it would provide critical insights on the type of innovation problems, the different funding structures, the optimal solution stage at entry and the combination of non-financial support that is most effective for crowding-in private capital and resources – meaning we can get much better at matching funding mechanisms to problem domains.

Making sense of reported ex-post leverage ratios

The earliest prizes for which leverage ratios have been reported include the Royal Agricultural Society of England (RASE) contests, dating back to 1839, with a ratio of 3x and the Orteig Prize, which took place between 1919–1927, with a ratio of 16x. More recent landmark prizes with reported leverage ratios include the previously mentioned Ansari XPRIZE (with a ratio range of 10x to 40x) and the DARPA Grand Challenge (50x), as well as the Northrop Grumman Lunar Lander Challenge, with a reported ratio of 10x. Where prize leverage ratios have been published for specific prizes, though, reporting typically takes the form of a headline statistic celebrating a significant multiplier effect and then it’s often a struggle to find more detail. 

The general opaqueness around reporting on prize leverage is significant because it likely means caution should be exercised in how we understand and interpret this data. Currently, there generally isn’t the information available to help us understand how methodologically robust these leverage ratios are. There’s also a lack of clarity on how concepts are being defined and, arguably, definitions being conflated in unhelpful ways. Prize leverage can be used as a catch-all term that combines different and conceptually distinct types of resource flows. Within the prize literature, for example, different measures of prize leverage include in-competition team spending on R&D; in and post-competition private capital secured by competing teams; in-kind support from partners; and publicity or earned media ad-equivalency values. The range of components that can come under prize leverage also makes clear that leverage extends beyond cash. The leverage of non-financial resources is an equally important consideration, not least because innovators are highly driven by non-financial  motivators such as publicity, brand recognition, and reputation, and learning, and they frequently consider these as far more valuable than the cash purse itself.

Beyond the blurring of definitions and concepts, it appears that the majority of reporting on ex-post prize leverage relies on the use of raw leverage ratios – in other words, ratios are simply calculated as either total participant follow-on funding or estimated contestant R&D spend divided by the cash value of the prize purse. Presenting these raw ratio figures might be a worthwhile thing to do, but doing so also risks crediting a prize competition for 100 per cent of the private capital that participating teams attract. Claiming full attribution of participant R&D spending or follow-on investment to the prize purse creates the risk of conflating general market-driven investment with true prize-induced effort. Hence, alongside any reporting of a raw leverage ratio, we need to also consider the reporting of discounted ratios that apply some form of counterfactual weighting to provide a more defensible ratio of what leverage the prize actually caused. 

Adjusting for the context of specific domains and problem-spaces

Any setting of targets for leverage has to be tailored to the problem-space and domain in which a prize is situated – rather than any mandating of a hurdle that all prizes have to achieve. Complex technology fields such as clean energy or space technology, where physical products and hardware need to be developed, can be expensive and risky. Problem-areas and domains with significant capital requirements often require public funding to absorb the early-stage and core risk that private investors are reluctant to take on. Similarly, challenges focused on public and social good are less likely to attract follow-on private investment. Certain problems that might be well-suited to prizes, but in high-risk or low-profit areas, will potentially drive the leverage ratio down. Leverage shouldn’t be a barrier that precludes the use of prizes, particularly those concentrated on social good or capital-intensive technology challenges.

Applying metascience lens to prize leverage

Understanding the returns to innovation investment is critically important. If we can be as precise as possible in estimating and calculating the value of research, development and innovation, then we can better assess whether our allocation of innovation funding is too low, too high, or about right. If we have robust evidence that a mechanism is particularly effective at mobilising capital and other resources to tackle a problem, then we can ensure the appropriate use of that funding tool. To get to that point, we need to take a more scientific approach to prize leverage, starting with three core pillars:

1. Disaggregate the components that make up prize leverage – we need to establish clear, distinct categories of leverage that allow us to better evaluate effects and performance. The prize literature points to various different components, including, most notably, the R&D effort of innovators, the external capital received by competing teams during and post-prize, and the signalling and prestige effect prizes confer through the halo effect and seal of quality on new technologies and firms. 
2. Develop ex-ante estimates of prize leverage – funders and prize delivery organisations should develop tools and models to provide estimates of leverage, tailored to the specific problem and domain. This could be based on key variables such as the expected stage-by-stage R&D costs; the technical uncertainty of the problem and related solutions; the potential commercial opportunities and market size; and the prestige utility of the prize. 
3. Rigorous ex-post tracking and measurement – transparent approaches to collecting and reporting data on prize leverage during and post-competition, which uses robust approaches, such as discount factors, to more accurately measure or attribute what leverage the prize actually caused.   

Being in a position to evaluate the ‘leverage gap’

Comparing ex ante (pre launch) predictions against ex post (post award) audited results is core to the scientific method; the ex-ante measures provide an explicit hypothesis (and supports the underpinning logic and pathway of an intervention), whilst the ex-post measures capture the observed empirical reality. Comparing ex-ante predictions to ex-post results allows you to work out the ‘leverage gap’, which means funders and policymakers can determine where prizes have succeeded on a key measure, where they have under or over-performed, and where pull funding should be usefully targeted. Instead of arbitrary targets, an approach based on ex-ante modelling and rigorous ex-post attribution supports an evidence-based approach for innovation funding and policy.