A deep dive into criteria, breakthrough statements and some ideas we have for a Longitude Prize on nature restoration
I spend a lot of time thinking about what teams should have to do in order to win a prize.
In innovation jargon, challenge prizes are an ‘outcomes-based’ funding mechanism. In more accessible language, that means teams are supposed to achieve something – have created a gizmo, solved a tricky problem, invented a new approach – and the grand prize is awarded only once they’ve done this.
In most of our challenge prizes, there are two short but very important pieces of text that sum this up.
One is the problem statement: a brief articulation of the issue we are hoping to make an impact on. In the Longitude Prize on Antimicrobial Resistance (AMR), for instance, our problem statement focused on the growing issue of antibiotic overuse, and in particular, the widespread prescription of antibiotics in cases where healthcare professionals don’t even know if they’re dealing with a bacterial infection or not.
The other is the challenge statement: this is the thing we’re asking the teams to create. This is where we set teams competing in the Longitude Prize on AMR the challenge of creating a point-of-care diagnostic device that would let healthcare professionals distinguish between bacterial infections (that need antibiotics) and viral infections (that don’t).
At Challenge Works, we spend a lot of time and thought crafting these sentences. They are the headline of the prize: what appears in press releases, on the home page of the website, in briefings to entrants and to the public. They shape what sorts of solutions we’ll reward in the prize – and the kind of impact they’ll have on the world.
Hiding behind the challenge statement, though, is something more in-the-weeds, which is: how would we know? This isn’t a secret, but it’s usually buried more deeply in the documentation we share with entrants and brief the judges with.
Often this is just a detailed set of qualitative criteria and a scoring matrix: who has done best, and hence who wins the grand prize, is left to the consensus of our expert judging panel, weighing up the quality of the entries.
For instance, in the Manchester Prize on AI (focused on AI solutions for energy, environment and infrastructure), teams’ entries were scored for innovation, expected impact, commercial viability, technical feasibility and safety.
We can do better than this.
I personally think that qualitative criteria like these are a bit of a cop-out. They are, at best, proxies for the outcomes that we’re seeking to create through the prize. Pragmatically it often makes sense to define the prize around these – but if you can, I think it’s far better for prize funders and organisers to cut out the proxies and award the prize directly for demonstrating the outcomes.
That requires us to ask ourselves a different question: not ‘what is the list of qualities we would like to see in a solution’, but rather ‘what is the achievement that would convince us that our challenge statement has been met’?
Our friends at XPRIZE Foundation call this a ‘breakthrough statement’. They ask the question: what’s the breakthrough we want teams to demonstrate.
Me? I studied philosophy of science – so in my more pretentious moments, I occasionally talk about an experimentum crucis. Newton and Boyle used this term to describe ‘crucial experiments’ in science – experiments which are, on their own, capable of decisively determining whether a theory is correct.
Whatever you call it, defining the victory condition of a prize in this way is hard, and forces you to grapple very seriously with exactly what it is you want teams to show they have achieved, without falling back on wooly platitudes about qualities you’d like a solution to have.
Sometimes we can’t: the part of government that commissioned us to run the Manchester Prize wanted it to support a broad portfolio of teams addressing a range of topics, so defining a breakthrough objective like that wasn’t possible. You can’t define a victory condition around achieving a goal, if there isn’t a single goal.
Sometimes we do manage: the Longitude Prize on AMR was judged, in part, on a demonstration of the winning solution in a healthcare setting: to win the prize it actually had to accurately and quickly tell apart viral and bacterial infections in real life. Had they failed, the prize would not have been won.
Sometimes we come tantalisingly close: in the Longitude Prize on Dementia we knew exactly what breakthrough we wanted. We wanted teams to demonstrate that users of their assistive tech continued living independently at home for measurably longer than a control group. The problem was that the clinical trial that we would have needed to prove this would have taken too long and cost too much. We were able to define a crucial experiment – but it just wasn’t practical to deliver, and so we fell back on a list of qualitative criteria.
Defining victory conditions for the next Longitude Prize
I mention all this because we have a few ideas for the next Longitude Prize which we are excited about but seem daunting to define the winning post for.
I think we can theoretically define breakthrough goals that separate winners from losers. But I’m not sure how practical they are. And rather than giving up, I’d like to see if anyone can help us define something more achievable.
So here are three ideas we have for things we’d like to reward with the next Longitude Prize – all ideas generated in our recent expert workshop on nature restoration.
(Thanks to everyone who contributed to this, and in particular to UCL Policy Impact Unit and GO Science for helping convene the participants.)
Prize idea: forecasting ecosystem tipping points
We’d like to get to a place where you can monitor an ecosystem – through sensors, satellite data and so on – and use that data to get much earlier warning that something is about to go badly wrong – like a crash in insect numbers, or a mass die-off of trees. (Early enough that you can do something about it.) The prize would therefore reward the creation of a forecasting system that gives the earliest possible, and most accurate possible, readout of what’s in store for an ecosystem.
How might we pick the winner?
This is fundamentally a prediction challenge – like our Metascience Novelty Indicators Challenge, or the Predicting Evolution programme currently being considered at @ARIA. To state the obvious, you judge the success of a prediction against what actually happens.
What we need, then, is to track one or more real ecosystems, and gather lots of sensor and satellite data. All teams have to do is to supply us with predictions of how this data will evolve in future – which we can then compare with the real thing. We’d have to decide exactly how to calculate scores and which measures to prioritise – not straightforward, but not overwhelmingly difficult either. The team with the most accurate prediction wins.
The problem
How much data would we actually need? How many forests, bogs, glaciers, grasslands would we need to be observing – and for how long? Unless this data already exists and we can access it, how practical is this prediction challenge?
If we do it with historical data – ‘hindcasting’ tipping points that have already occurred – can we ensure the integrity of the judging process and that no teams will have been able to access this?
Prize idea: smart ocean management
Protecting marine ecosystems sometimes relies on creating exclusion zones where fishing isn’t allowed. What if this could be much more dynamic – reflecting the movement of marine animals and the changing marine environment, as well as interventions that go beyond simple geofencing?
A prize could incentivise the creation of a dynamic ocean management system that balances ecological and economic needs – and lead to better tradeoffs between fishing and conservation.
How might we pick the winner?
For this, I think we really need teams to demonstrate the finalists’ interventions in the real world. We need to be able to compare the performance of teams against each other, as well as against business as usual. It’s conceptually quite straightforward – the team that wins is the one that does best at protecting marine life in their patch.
The problem
It’s the execution that would be daunting: for one, teams would need to be able to control access to an area of sea. That alone doesn’t sound straightforward, and would need a very cooperative government. And not just one area: we’d need to do it more than once per team – to filter out any flukes or bad luck. This would have to be something like a randomised control trial – like the experiments that pit new drugs against placebos in thousands of volunteer patients to see if they work. How many pilots would we need to do? Would judging this prize end up needing literally hundreds (or thousands) of trials in different parts of the sea?
Are there any other, easier, approaches to testing the real-world performance of such tools? Could we instead focus our testing on a technical milestone that falls short of large-scale demonstration? And if we did, how would we ensure that it was an appropriate proxy for the real thing?
Prize idea: predictive restoration
When you take action to restore a degraded ecosystem – plant a forest, re-establish a wetland – there’s a lot of guesswork involved and the intervention can fail. That’s particularly true now because our fast-changing climate means designing interventions around a moving target. A prize could incentivise innovators to create tools that can help design an intervention to be maximally effective, including future-proofing it against different climate change scenarios.
How might we pick a winner?
The principle is the same as for smart ocean management – a randomised control trial that compares the effectiveness of the different interventions.
The problem
Demonstrating this could be even tougher than for smart ocean management. First because of the long timescales concerned, and second because these would be far more complex interventions than simply managing ocean exclusion zones.
This isn’t just short-term management of access restrictions in an area of sea, which might be evaluated over a period of a few months or a few years. The nature restoration interventions here would be multi-faceted, play out, and interact with climate change, literally over decades.
Are we really going to track hundreds of experiments, each tracking numerous indicators, for decades before handing out a prize?
But if not – how else could we fairly assess the success of the teams? What’s the minimum viable experiment that would effectively identify the best teams – a simple indicator or proxy that would be good enough for us to award the prize against? And if we took that approach, could we avoid Goodhart’s Law – teams optimising their solutions for the proxy, rather than the real impact we are seeking?
What should we do?
The trouble with all three of these is that, at least in the way I’ve set out here, I think it would take a lot to conclusively demonstrate success.
We’re really excited about the potential impact of these ideas. On paper they are well suited to the prize method – addressable through a variety of approaches, yet with a clear articulation of the outcomes we want to be achieved.
The fact it would be hard is also good – we want to be bold, even audacious.
But a prize also has to be deliverable.
I would love to hear my network’s thoughts on how we should approach this. What shortcuts can we take that let us accurately judge outcomes without spending decades and millions of pounds doing so?
If we do have to fall back on proxies, what are some clever, specific and accurate ones we could use?
And if you’re a funder with this level of ambition, please reach out. Our Longitude Prizes change the world thanks to the ambition and the financial support of our sponsors.
Thanks to Vidal Kumar for constructive input into an earlier version of this article.
