In July 2023, I shared my thoughts on linkedIn regarding Trust in Science , a publication by Australia’s Chief Scientist, Cathy Foley. This insightful essay delves into the core elements of scientific quality – research integrity, quality, excellence, and impact – and offers a rare, well-informed perspective on the subject.
Foley’s essay provides clear definitions for these four elements, which differ from more traditional interpretations, particularly around quality and excellence. Typically, “excellence” is used as a synonym for the usefulness or influence of research and is often viewed as part of research quality. However, Foley emphasises the importance of distinguishing excellence as its own dimension, separate from quality, and on equal footing with impact and integrity. These distinctions are crucial, as these concepts are deeply interdependent and often overlap, making them challenging to define.
Building on Foley’s work, I approach the concept of research quality with a slightly different framework. I break it down into four key elements: rigour, novelty, influence, and impact. There’s no definitive way to split these elements since they are interconnected and overlap, but this categorisation has proven useful in my scientometric analyses.
Rigour evaluates the soundness, solidity, and plausibility of research findings and methodologies. This is a relative measure and is typically the focus of peer review when assessing research quality. I also group research integrity under rigour, as it refers to “adherence to ethical principles and professional standards essential for the responsible conduct of research.“
Novelty considers the originality of the research, ranging from incremental advancements in established knowledge to disruptive breakthroughs. While novelty can be indicated by quantitative metrics, it primarily depends on identifying emerging ideas in science.
Influence looks at the value and significance of research within the broader scientific community. Often referred to as “excellence,” influence is commonly measured by citation counts or esteem metrics such as awards. Although citation “impact” serves as a useful proxy for influence, it primarily reflects the research’s reach within the scientific institution.
Impact addresses the broader outcomes of research beyond the scientific sphere, including social, political, environmental, or economic effects. Referred to as “spillovers” from science, impact can be measured through case studies, patent citations, or altmetrics, which assess relevance and public engagement.
The relative importance of these elements varies across disciplines. For example, novelty may be valued differently in the natural sciences compared to the humanities. Additionally, these dimensions are often correlated—for instance, novel research is likely to attract more citations, and a robust methodology or clear presentation can enhance a study’s influence.
Understanding these nuances helps us better evaluate scientific work, allowing for more informed decisions in research and innovation.