Academic journals are increasingly asking authors to use transparent reporting practices to “trust, but verify” that outcomes are not being reported in a biased way and to enable other researchers to reproduce the results. To implement these reporting practices, most journals rely on the process of peer review — in which other scholars review research findings before publication — but relatively few journals measure the quality and effectiveness of the process.
In a commentary published July 20 in the journal Science, lead author Carole Lee and co-author David Moher identify incentives that could encourage journals to “open the black box of peer review” for the sake of improving transparency, reproducibility, and trust in published research. Lee is an associate professor of philosophy at the University of Washington; Moher is a senior scientist at The Ottawa Hospital and associate professor of epidemiology at the University of Ottawa.
Lee and Moher see this as a collective action problem requiring leadership and investment by publishers.
“Science would be better off if journals allowed for and participated in the empirical study and quality assurance of their peer review processes,” they write. “However, doing so is resource-intensive and comes at considerable risk for individual journals in the form of unfavorable evidence and bad press.”
To help journals manage the reputational risk associated with auditing their own peer review processes, Lee and Moher suggest revising the Transparency and Openness (TOP) Guidelines, a set of voluntary reporting standards to which 2,900 journals and organizations are now signatories. These guidelines were published in Science in 2015 by a committee of researchers and representatives from nonprofit scientific organizations, grant agencies, philanthropic organizations and elite journals.
Lee and Moher suggest adding a new category to the TOP guidelines “indicating a journal’s willingness to facilitate meta-research on the effectiveness of its own peer review practices.” With these, journals can choose which tier or level they take on. Higher levels of transparency would involve higher risk.
- For the lowest tier, journals would publicly disclose whether they are conducting internal evaluations of peer review, in which they are able to retain the study results for internal use.
- At the middle tier, journals would disclose the results of their internal evaluations of peer review, but could maintain flexibility in how they report their results for external use. For example, results could be aggregated across several journals to reduce risk to any single journal.
- At the upper tier, journals could agree to relinquish data and analyses to researchers outside their institution for third-party verification. This is an option, Lee and Moher write, “that might appeal especially to publishers with fewer resources, as it places the financial burden on those conducting the meta-research.” Journals conducting their own analyses could preregister their study designs then deposit their data publicly online.
By agreeing to these more stringent guidelines, the authors write, publishers and journals would have the chance to legitimize and advertise the relative quality of their peer review process in an age when predatory journals, which falsely claim to use peer review, continue to proliferate.
“Illegitimate journals are becoming a big problem for science,” said Moher. “True scientific journals can distinguish themselves with transparence about their peer review processes.”
Investing in research on journal peer review will be costly, they agree. Lee and Moher suggest that large experimental studies are needed to judge the effectiveness of different web-based peer review templates to enforce reporting standards, and of ways one might train authors, reviewers and editors to use such tools and evaluate research.
Also needed, they say, are ways to detect shortcomings in statistical and methodological reporting on a research paper, and to understand how the number and relative expertise of peer reviewers can improve assessment.
The largest publishers, whose profit margins compete with those of pharmaceutical and tech giants, can afford to invest in the requisite technology and resources needed to carry out these audits, the researchers say.
“Publishers should invest in their own brands and reputations by investing in the quality of their peer review processes,” said Lee. “Ultimately, this would improve the quality of the published scientific literature.”
Learn more: Bringing a ‘trust but verify’ model to journal peer review
The Latest on: Journal peer review
[google_news title=”” keyword=”journal peer review” num_posts=”10″ blurb_length=”0″ show_thumb=”left”]- Du Bois Review: Social Science Research on Raceon May 10, 2024 at 2:26 pm
This journal utilises an Online Peer Review Service (OPRS) for submissions. By clicking "Continue" you will be taken to our partner site https://mc.manuscriptcentral ...
- Journal retracts 6 further articles and corrects 2 others authored by former editoron May 9, 2024 at 3:39 pm
following an extensive investigation of his sole authored content in the journal.* The retractions comprise four ‘warm up’ editorials and one book review due to plagiarism. A letter has also been ...
- Eleventh Nano Research Award goes to Louis E. Brus and Moungi Bawendion May 9, 2024 at 6:05 am
Nano Research is a peer-reviewed interdisciplinary research journal, publishes all aspects of nano science and technology, featured in rapid review and fast publishing, sponsored by Tsinghua ...
- Infinitopes' Article in Peer-Reviewed Journal Seeks to Unlock the Potential of Cancer Vaccineson May 9, 2024 at 12:29 am
Infinitopes Precision Immunomics, an integrated cancer biotech combining world leading platforms in precision antigen discovery with vaccine vectors capable of durably stimulating protective immune ...
- Military Coups in West Africa and Future of ECOWASon May 7, 2024 at 1:09 am
West Africa has experience of military coups, characterized by the overthrow of democratically elected governments by military forces. These coups have been driven by various factors, including ...
- Is the Science journals editor showing remorse?on April 23, 2024 at 10:30 am
Hauled in front of Congress recently, Science Editor-in-Chief Holden Thorp showed signs of being open to views other than his own, which may be a first for him. Thorp testified last Tuesday during a ...
- BioMed Central to publish new journal Research Integrity and Peer Reviewon April 1, 2024 at 5:00 pm
In response, BioMed Central is pleased to announce the launch of a new open access journal Research Integrity and Peer Review, which will act as an academic forum where these discussions can take ...
- AI scandal rocks academia as nearly 200 studies are found to have been partly generated by ChatGPTon March 19, 2024 at 11:07 am
These papers slipped through because of lax or nonexistent peer-review processes at for-profit journals, stoking wider fears that the body of human scientific knowledge is being rapidly ...
- Peer Reviewon March 5, 2024 at 10:05 pm
Suggest potential reviewers Make a decision on the paper without further review ... ESA journal. If the decision was “Revise,” upon receipt of the revision, the SME will evaluate the revised ...
- Instructions for peer reviewerson September 7, 2023 at 5:50 am
Peer review is the foundation of quality in research for both books and journals, ensuring that published research is rigorous and ethical. Peer reviewers can access a number of resources to assist ...
via Google News and Bing News