AI and the review process

Will artificial intelligence (AI) replace peer reviewing?

A note of caution - the field of AI is moving so quickly, that anything written here is likely to be out of date when you read this. So please search for information yourself about this!

What is clear, is that AI is already entrenched by many journals in their review systems. To help cope with large numbers of submissions, journals use AI systems to link to plagiarism software, to suggest reviewers based on content, to check on various characteristics of the reviewers such as previous publications and conflicts of interest, and to send reminders and thank you letters.

This paper AI-assisted peer review lists a number of areas where AI is used. The figure allows us to consider at which part of the review process AI might be able to make a contribution:

From: AI-assisted peer review published under a CC 4.0 licence.

A blog summarising the paper states: 'Manuscript submissions to peer-review journals are growing roughly 6% annually. Every year, over 15 million hours are spent on reviewing manuscripts previously rejected and then resubmitted to other journals....even using only rather superficial metrics to perform the training, the machine learning system was often able to successfully predict the peer review outcome reached as a result of human reviewers’ recommendations. In other words, there was a strong correlation between word distribution, readability and formatting scores, and the outcome of the review process as a whole. Thus, if a manuscript was well written, used appropriate terminology and was well presented, it was more likely to be accepted'.

While the Frontiers' Artificial Intelligence Review Assistant has been shown to highlight 'where critical decisions need to be made, this technology does not replace researchers but empowers them to make editorial decisions more effectively. It will support editors and reviewers by directing their attention to potential issues in manuscripts. These issues can then be addressed or clarified as part of the manuscript review'. Authors can also access and be guided by this system.

The World Association of Medical Editors (WAME) have a set of Recommendations on Chatbots and Generative Artificial Intelligence in Relation to Scholarly Publications. The five recommendations stipulate that  1) Only humans can be authors; 2) Authors should acknowledge the sources of their materials; 3) Authors must take public responsibility for their work; 4) Editors and reviewers should specify, to authors and each other, any use of chatbots in evaluation of the manuscript and generation of reviews and correspondence; and 5) Editors need appropriate digital tools to deal with the effects of chatbots on publishing. Unfortunately, any content fed to a chatbot is retained and not kept confidential, so using it for review is potentially problematic.

You may want to explore this set of slides which cover Artificial Intelligence (AI) and associated tools and policies in editorial decision making, from the Academy of Science in South Africa.

The future

At the time of writing, the use of AI can be summarised as in this blog: Will ChatGPT Disrupt Peer Review? 'The integration of AI tools into the peer review process can be beneficial in assisting with certain tasks such as language editing and conflict of interest detection. However, the use of AI tools must be continually evaluated and responsibly implemented to ensure that they are not perpetuating biases or impacting the quality and reliability of scholarly literature. The expertise and judgment of human reviewers will always be essential in ensuring the rigor and dependability of the peer review process, and the continued integration of AI tools should be viewed as a complementary tool rather than a replacement'.

Ethical considerations: Here is an interesting article published on an open platform: A Critical Examination of the Ethics of AI-Mediated Peer Review. It '...emphasizes the need to critically assess the legitimacy of AI-driven peer review, addressing the benefits and downsides...' It also makes the point that 'human peer review systems are also fraught with related problems, such as biases, abuses, and a lack of transparency, which already diminish credibility.'

However in view of the speedy development in AI - watch this space!!!!