We are thrilled to announce the winner of the Metascience Novelty Indicators Challenge, a competition designed to find innovative, data-driven ways to identify and celebrate truly novel scientific contributions.
Meet the winner Taking home the grand prize is an interdisciplinary research team from the Juelich Research Center in Germany. Team members Jan-Maris Göpfert, Samuel Kieling, Titan Hartono, Patrick Kuckertz, and Jann Weinand usually specialise in AI systems for energy research. However, they combined their expertise in natural language processing (NLP), large language models (LLMs), data science, bibliometrics, and research data management to build something revolutionary: LENS. What is LENS?"LENS works by quantifying the novelty of scientific publications using LLMs. The system analyses the full text of an article, examines key references to reconstruct what was known at the time of publication, and then conducts an evaluation, weighing arguments for novelty against sceptical counterarguments."
How the scoring works Based on this deep content analysis, LENS generates:
- A score from 0 to 100: Higher scores indicate major contributions, such as solving critical problems, introducing valuable new concepts, or offering surprising findings.
- A qualitative justification: The system provides written pros and cons regarding the paper's novelty.
By capturing nuance instead of just raw numbers, LENS can help researchers, institutions, and funding bodies spot game-changing research early on, ensuring valuable work doesn't slip through the cracks. What’s next for the novelty project? The conclusion of the challenge is just the beginning. The Science Policy Research Unit (SPRU) at the University of Sussex and RAND Europe are taking these insights and running with them. Here is what is on the horizon:
- Deeper empirical analysis: The SPRU team will expand their surveys and interviews to better understand how human assessors conceptualize "novelty" and how this impacts research evaluation.
- A new research toolkit: SPRU and RAND Europe are co-developing a toolkit packed with learnings from this massive human validation study, to be shared via peer-reviewed papers.
- Open-source data: To support the broader metascience community, the corpus of papers used during this project will be made publicly available as a scholarly resource.
- Stakeholder workshops: RAND Europe and SPRU will host engagement workshops with policymakers, funders, and metascientists to discuss how indicators like LENS can be integrated into real-world decision-making.
- Policy paper: A comprehensive policy paper will be published detailing the Challenge's outcomes and its broader implications for global innovation policy.
Get involved If you're interested in Challenge Works' future Frontier Technologies work, reach out to the team.
