Research Guides: Research Impact & Intelligence: Responsible Research Assessment

Responsible Research Assessment: Video Overviews

Responsible Use of Rankings: 10 Principles
Ten principles developed by CWTS that are intended to guide the responsible use of university rankings. These principles apply to university rankings in general.

Resources and Readings on Responsible Research Assessment

San Francisco Declaration on Research Assessment (DORA)
The Declaration on Research Assessment (DORA) recognizes the need to improve the ways in which the outputs of scholarly research are evaluated; focuses primarily on not relying on journal metrics, such as the Journal Impact Factor (JIF), for the assessment of quality of research, hiring decisions, grant dollars, etc. DORA focuses more on the STEM fields.
Bibliometrics: The Leiden Manifesto for research metrics
Can be more broadly applied to other fields outside STEM. Urges qualitative evaluation in conjunction or complementary to quantitative assessment in all cases but especially of individual researchers. Written by leading experts in research evaluation and bibliometrics.
The Metric Tide
"This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management."
Highly recommended that you read the Executive Summary and Recommendations on pages vii - xiv.
HuMetricsHSS
HuMetricsHSS is an initiative for rethinking humane indicators of excellence in academia, focused particularly on the humanities and social sciences (HSS). HuMetricsHSS endeavors to create and support a values-based framework for understanding and evaluating all aspects of the scholarly life well-lived and for promoting the nurturing of these values in scholarly practice.
The Hong Kong Principles for assessing researchers: Fostering research integrity
Authors present five principles: responsible research practices, transparent reporting, open science/research, valuing diverse types of research, and recognizing all contributions to research and scholarly activity.

DORA University Case Studies
Case studies of universities and national consortia highlight key elements of institutional change to improve academic career assessment.
Statements of Responsible Metrics
Research institutes across the world have developed statements regarding the use of metrics, some in response to the Leiden Manifesto and the San Francisco Declaration on Research Assessment, and some independently. Collected here are some examples of these statements.
SCOPE Framework for Research Evaluation
The SCOPE framework for research evaluation is a five-stage model for evaluating responsibly. It is a practical step-by-step process designed to help research managers, or anyone involved in conducting research evaluations, in planning new evaluations as well as check existing evaluations. SCOPE is an acronym, where S stands for START with what you value, C for CONTEXT considerations, O for OPTIONS for evaluating, P for PROBE deeply, and E for EVALUATE your evaluation.
The evaluative inquiry: a new approach to academic evaluation
This is another nice way of applying principles to process and practice. It's written by some of the leading experts on research evaluation and bibliometrics on their blog at the Center for Science and Technology (CWTS) at Leiden University (yes, some of those at CWTS are the authors of the Leiden Manifesto!)
Responsible metrics: One size doesn't fit all
How to evaluate research based on the type and size of your analysis.

Rethinking The Rankings
Initial findings of the INORMS Research Evaluation Working Group’s efforts to rate the World University Rankings.
The Order of Things What college rankings really tell us
This journalist breaks down some of the issues with college rankings and even compares them to car rankings.
The unsustainable goal of university ranking
Ranking organisations are seeking to diversify the measures use to evaluate universities. But without addressing the fundamental flaws in their methods, they will crush rather than embrace the rich complexity of our institutions of higher learning.

Against metrics: how measuring performance by numbers backfires
A summary and review of the book, The Tyranny of Metrics, by Jerry Z. Muller.
The Tyranny of Metrics - Colleges and Universities (Book Chapter)
Virginia Tech users can read this book chapter.
Excerpt: "Comprising a huge sector of the national economy and a central institution of all advanced societies, colleges and universities exemplify many of the characteristic flaws and unintended consequences of measured performance, as well as some of its advantages. Once we become fixated on measurement, we easily slip into believing that more is better."
Rethinking impact factors: better ways to judge a journal
"We need a broader, more-transparent suite of metrics to improve science publishing, say Paul Wouters, colleagues and co-signatories."
Bibliometrics and Research Evaluation: Uses and Abuses
Why bibliometrics is useful for understanding the global dynamics of science but generate perverse effects when applied inappropriately in research evaluation and university rankings.

The Bibliomagician Resource Hub
Includes lots of readings and responsible research assessment policies at various universities (mostly in the UK).
Over-optimization of academic publishing metrics: observing Goodhart’s Law in action
Peer-reviewed article from GigaScience. "The metrics used to measure academic success have not changed for decades, though recent studies indicate that these metrics have become targets and follow Goodhart’s Law, according to which, 'when a measure becomes a target, it ceases to be a good measure.'"
Goodhart’s Law: Are Academic Metrics Being Gamed?
A blog post and summary of the results of the article above ("Over-optimization..")
When Do Citations Reflect "Impact?"
Karin Wulf, editor of the William and Mary Quarterly, offers a humanities perspective on citations and the meaning behind them.

What is responsible research assessment?

Responsible research assessment is becoming increasingly popular among academics and particularly among the bibliometrics and librarian communities, who are the knowledge producers and keepers of the data and metrics used to evaluate research and researchers. With increasing reliance on quantitative measures, this page will provide valuable resources on how to more responsibly, equitably, and fairly approach research evaluation.

Research Impact Librarian

Rachel Miles

she/her/hers

Email Me

Contact:

540-231-7469