The future of peer review: How AI is changing the game

March 27, 2023

eLife, a prominent open-access scientific journal, has caused controversy by publishing all papers received for peer review, as well as reviewers’ reports (Nature, 17 March 2023). While some applaud it as a step towards authors’ empowerment and transparency, others are concerned it could damage the journal’s reputation and affect the traditional role of peer review. Journals have long been viewed as gatekeepers, deciding what research to distribute and influence the scientific community. However, the notion of gatekeeping has been challenged by open access publishing and preprint servers. Consequently, it is suggested that journals should focus on supporting scientific communication and making peer review more open and transparence. This controversy at eLife raises important questions concerning the future of scientific publishing: Should journals remain gatekeepers or take a more open and transparent approach? How can we improve peer review to guarantee quality and fairness while being more accessible and transparent? These questions will likely shape the scientific publishing landscape in the future.

What is peer reviewing?

Peer review is essential for scholarly publishing, with experts in a field assessing a manuscript’s quality and validity before publication. Unfortunately, the traditional process is slow and opaque, with manuscripts potentially taking years to be reviewed, revised and published. This is a major issue for authors who need to publish immediately for funding or job positions. Additionally, peer review is susceptible to bias, with reviewers sometimes rejecting manuscripts based on personal opinions or affiliations, rather than on research quality.

Traditional peer reviewing

In scholarly publishing, there are several traditional ways of peer review:

  1. Single-Blind Review: This is the most common form of peer review, where the reviewer’s identity is kept anonymous, but the author’s identity is known to the reviewer.

  2. Double-Blind Review: In this method, both the reviewer’s identity and the author’s identity are kept anonymous.

  3. Triple-Blind Review: Triple-blind peer reviews are a type of peer review where author, reviewer, and editor identities are kept anonymous.

  4. Cascading Peer Review: Journals may refer rejected manuscripts to another journal within the same publisher’s portfolio for peer review. This allows authors to ensure that their article is read and published.

  5. Open Review: This form of review involves disclosing the reviewer’s identity to the author, and sometimes publishing the reviewer’s name alongside the article.

Each method has its strengths and weaknesses, and there is ongoing debate about which approach is the most effective in ensuring quality and fairness in the peer-review process.

Emerging trends

While there are ongoing debates about the effectiveness of peer review in scholarly publishing, there is currently no widely accepted alternative that can replace the peer-review process entirely. However, there are several emerging trends that could complement or supplement traditional peer review:

  1. Open peer review: Open peer review is a transparent and accountable model gaining popularity. In this process, authors and readers can identify reviewers, collaborating throughout the review process to achieve constructive feedback and inclusive manuscript development. F1000Research is well-known for embracing Open Peer Review. Additionally, Open Access publishers like PeerJ, BMC, and Frontiers offer traditional and open peer review options. Challenges arise due to reviewer reluctance to reveal identity and time needed engaging in discussions and revisions.

  2. Preprint servers: These online repositories allow authors to share their research findings before peer review and publication, making research available to the public much more quickly.

  3. Collaborative and community-based peer review: This peer review method involves inviting a variety of experts and stakeholders, including patients and the public, to provide input. Journals facilitate the process by providing a platform where authors and reviewers can work together to review the manuscript. This collaborative approach ensures that the manuscript is accurate and valid, though it can be a lengthy process.

  4. Third party peer review: In third-party peer reviews, authors engage with independent reviewers and make necessary manuscript edits based on their feedback before submitting to journals. While this type of peer review may help decrease the chances of desk rejection, it could at times prove ineffective – especially if the peer reviewers are not aware of the target journal’s scope. 

  5. Artificial intelligence and machine learning: AI and machine learning algorithms are being developed to help analyze and evaluate research, including assessing the quality of the research, detecting errors or fraud, and identifying potential areas of bias.

  6. Open data and open access: Making research data freely available to the public can improve transparency, accountability, and reproducibility, leading to a more rigorous and transparent scientific process.

  7. Post-publication review: This approach involves publishing articles first and then inviting feedback and comments from the scientific community, which can be used to further refine the research.

It is important to note that these emerging trends are not intended to replace peer review entirely, but rather to complement and improve the traditional peer-review process. Peer review remains an essential component of the scientific publishing process, ensuring that research is rigorously evaluated and vetted by experts in the field.

How AI can help?

AI tools can help streamline the peer review process, reducing the time and labor required for authors and reviewers. They can match manuscripts with appropriate reviewers, check for ethical concerns, and help identify weaknesses in manuscripts by analysing data and suggesting improvements. Specifically, AI can:

  • Help editors match manuscripts to appropriate reviewers – These tools use machine learning algorithms to analyse the text of manuscripts and identify the most appropriate reviewers based on their areas of expertise, publication records, and previous reviewing history. Some examples of reviewer recommender systems include Publons, ReviewerCredits, and Peerage of Science. 

  • Check for plagiarism, fraud, or other ethical concerns – These tools use AI algorithms to compare the text of manuscripts to a large database of published work to identify any instances of plagiarism or duplication. Examples of such tools include iThenticate, Turnitin, and CrossCheck. 

  • Assist reviewers in identifying potential weaknesses in manuscripts, by analysing data and methods – Tools like Tableau, Plotly, and DataWrapper use AI algorithms to analyse and visualise the data presented in manuscripts, helping reviewers identify trends, patterns, and potential issues. 

  • Assist in editing and proof reading – These tools use AI algorithms to identify and correct grammatical errors, spelling mistakes, and other language-related issues in manuscripts. Examples of such tools include Grammarly,LanguageTool, and ProWritingAid.

  • Assess quality of review – These tools use AI algorithms to analyse the quality of reviews submitted by reviewers, helping editors to identify potential issues and provide feedback to improve the quality of reviews. A lot of discussion is around and many of the publishing firms are experimenting with several in-house products.

Innovations by publishers

Many of the academic publishers use AI tools to improve the speed, quality, and fairness of their peer review process. AI tools can automate manual tasks like identifying reviewers and detecting plagiarism, streamlining and accelerating the publishing process. They can also identify weaknesses or issues in manuscripts, as well as potential bias from reviewers, helping to ensure quality and integrity of research. Here are some examples of emerging AI tools that can be used in peer reviewing:

  1. SciScore: This AI tool developed by the Allen Institute for AI evaluates research articles based on several factors, including the quality and relevance of the research, the clarity of the writing, and the overall impact of the research.

  2. ScholarOne: This peer-review management system used by several publishers, including Elsevier and Wiley, includes several AI-powered features, such as automated formatting and checking for potential conflicts of interest. 

  3. Scholastica: A peer review software designed to help journals work smarter, not harder — with all the features editors, authors, and reviewers need for smooth submissions and manuscript management and none of the complexities they don’t.

  4. Editorial Manager: This peer review management system is used by many publishers, including Springer, Taylor & Francis, and Wiley. It includes AI-powered features like automated plagiarism detection, reviewer matching, and manuscript tracking.

  5. BenchSci: This AI-powered search engine helps researchers find relevant antibodies for their experiments. It uses machine learning algorithms to analyse millions of scientific papers and extract relevant information about antibodies, including their specificity and performance.

  6. Typeset: Hindawi uses an AI tool called “Typeset” to help authors format their manuscripts according to the journal’s guidelines and standards.

Conclusion

In conclusion, peer review is an essential aspect of scholarly publishing, but the traditional process has limitations. However, emerging models of open peer review and the use of AI can provide potential solutions to improve the process. AI technology continues to evolve, we can expect to see more tools and platforms being developed to support peer review and enhance its effectiveness and efficiency. By embracing these reforms, we can create a more efficient and effective peer review process that benefits authors, reviewers, and readers alike.

Additional Reading

  1. Abbot A. 2023. Strife at eLife: inside a journal’s quest to upend science publishing. Nature https://doi.org/10.1038/d41586-023-00831-6

  2. Abdul Razack et al. 2021. Artificial intelligence-assisted tools for redefining the communication landscape of the scholarly world. Science Editing 8(2):134-144. https://doi.org/10.6087/kcse.244

  3. Woods, H.B. et al. 2023. An overview of innovations in the external peer review of journal manuscripts. Wellcome Open Research 7: 82