Topic outline

  • About this course: Peer reviewing theory and practice

    Collapse all Expand all

    This course was designed by Peoples-Praxis on behalf of the Coalition for Open Publishing of Public Health in Africa (COPPHA)as part of building capacity for innovative publishing options.

    Note: if you want to gain a certificate for completing this course, you will have to create an account and log in as a student.

    The quality of scientific publications has been maintained over the years by a process of peer review. Peers, usually other researchers with expertise in the area of research, examine the research report that has been submitted for publication, and make recommendations to the journal editor. It is important for researchers to take part in the peer review process and agree to requests to act as peer reviewers - it is a vital way in which research maintains high standards and scientific integrity.

    This depends on the reviewers doing a good job, and part of this course offers access to various guides about how to review journal articles.

    There have been many criticisms of the peer review process, and the course discusses some of these criticisms.

    There are a number of new approaches to the review process as well as to the way in which research may be published. There are now ways for researchers to put their research reports online for all to access before they are reviewed, these are called preprints. Peer review is sought for these preprints in a process of open review. The author(s) can then review the paper according to the review. The paper may now be submitted for publication to a more traditional journal, and after submission will remain available for open access. This is an evolving field, and we discuss this in a later course.

    Course learning outcomes. At the end of this course you should be able to:

    • Understand the ways in which peer review underpins the academic publication process
    • Reflect on the challenges of the peer review process
    • Appreciate the characteristics of a good peer review and be able to access guidelines
    • Post a good peer review using a standard format
    • Reflect on the issues of AI in the review process

    This course should be explored in combination with the Critical Appraisal and the Open Science courses which you will find elsewhere in this series of courses.

    How to navigate the course

    Each section comprises a set of resources that we think you will find interesting - click on the collections of resources in each section. There is a forum in each topic for reflection.

    We encourage you to reflect on what you have learned or comment on the course. When you click on the hyperlink in each topic labelled reflection, you will be able to add a new topic or respond to a previous one. You may want to share your learning from this and other readings, comment on the topics from your own experience, comment on others' posts, or provide feedback on how we can improve the content and presentation.

    In the final section you will see that you can gain a Certificate of Completion - the requirements for this are to access the resources, post a reflection in each section and pass the assignment.

    We encourage you to reflect on the issues, maybe make notes as you go along as this is a good way to be sure that you can internalise the information presented and the lessons to learn from it.

    Creative Commons License

    This work is licensed under a Creative Commons Attribution 4.0 International License.

  • Fundamentals and challenges of the peer review process


    You might want to access slides and recordings of a session on the fundamentals and challenges of the peer review process - from when the course was previously offered: Access the slides here and the recording here. These are optional activities

    The fundamentals of the process

    "Scholarly peer review or academic peer review (also known as refereeing) is the process of having a draft version of a researcher's methods and findings reviewed (usually anonymously) by experts (or "peers") in the same field. Peer review is widely used for helping the academic publisher (that is, the editor-in-chief, the editorial board or the program committee) decide whether the work should be accepted, considered acceptable with revisions, or rejected for official publication in an academic journal, a monograph or in the proceedings of an academic conference. If the identities of authors are not revealed to each other, the procedure is called dual-anonymous peer review (from Scholarly peer review).

    Here is a diagram which describes a typical peer review process:


    From Mohini & Ahmed, published under CC BY 4.0

    Each journal takes a slightly different approach, but this is the way the process flows. Peer review is not limited to publishing in journals. The same procedures are applied to ensure content quality in books and there is also a rising practice of reviewing preprints.

    We will be discussing the benefits and limitations of various parts of this process later in the course, in particular the alternative types of review - single of double blind, transparent or fully open. We will also have the opportunity to practice reviewing and posting open reviews.

    Challenges of the peer review process

    As Richard Smith, Editor of the British Medical Journal for many years has said, 'Peer review is at the heart of the processes of not just medical journals but of all of science. It is the method by which grants are allocated, papers published, academics promoted, and Nobel prizes won. Yet it is hard to define'...'But it is something to do with a grant application or a paper being scrutinized by a third party—who is neither the author nor the person making a judgement on whether a grant should be given or a paper published.'

    Here are some of the problems as identified by Richard Smith and others:

    Exploitative. Publishing companies can make large profits from academic journals. In 2022, the publisher Elsevier is reported to have made a profit of 1.2 Billion Euros with a profit margin of 37% (higher than that of Google at 21% or Apple at 25%). For some context, the whole of the European Research Council's annual budget is 2.4 Billion Euros. Meanwhile most of the journal editors and reviewers are academics who take on these tasks as volunteers.

    Slow. It can take a year or more from submitting a paper until it is published. Much of this time is taken by the review process.

    Inconsistent. A number of studies have shown differences between reviewers in their assessments. Of course this may reflect good choice of the reviewers to represent different skill sets, but is more likely to represent observer variation. This inconsistency may represent bias - towards papers that come from recognised authors or institutions, or according to the country or gender of the authors. There is a difference of opinion and practice in blinding the reviewers to the names and institutions of the authors. There is quite a large literature on the topic of bias.

    (There are a number of other possible biases in the publication of research - such as the difficulty in getting negative results published or improved chance of acceptance if the same journal is quoted among the references. These are editorial decisions - editors are the first arbiters of whether to send a submitted paper to referees and whether to publish based on reviewer comments.)

    Lack of transparency. Most of the processes involved in the review and acceptance of submitted papers are closed to scrutiny. Many of the issues discussed above can be resolved by increasing transparency - one of the options shown above in the diagram from BMC. This opens the door to Open Publishing and Open Science, where open reviewing has become a key part of the open access publishing movement.

    Incentives and responsibilities.
    As mentioned in the Introduction, it is important that researchers act as peer reviewers to maintain the scientific integrity of the system. However, even though the publisher may make a profit the reviewers and editors are usually volunteers. Editors often find it frustrating when potential reviewers refuse to help. There are ways of offering incentives and rewards to reviewers to encourage their participation in the system. We will discuss open publishing of reviews in the next course, in which case credit can be given to published reviews.

    • forum icon
      Reflection on the peer review process and its challenges Forum
      View Make forum posts: 1

      Please reflect on and share any experiences you may have of peer review. Are there any parts of the peer review process that surprise you? Does this process accord with your own experience as an author or reviewer, or editor? What do you think are the benefits and problems of the process as described? Can you add to the challenges mentioned above, and do you have any suggestions for change to the system.

      Posting your reflection is a requirement for gaining a certificate.

  • AI and the review process


    You might want to access slides and recordings of a Zoom session on AI and peer review peer review process - from when the course was previously offered: Access the slides here and the recording here.



    Will artificial intelligence (AI) replace peer reviewing?

    A note of caution - the field of AI is moving so quickly, that anything written here is likely to be out of date when you read this. So please search for information yourself about this!

    What is clear, is that AI is already entrenched by many journals in their review systems. To help cope with large numbers of submissions, journals use AI systems to link to plagiarism software, to suggest reviewers based on content, to check on various characteristics of the reviewers such as previous publications and conflicts of interest, and to send reminders and thank you letters.

    This paper AI-assisted peer review lists a number of areas where AI is used. The figure allows us to consider at which part of the review process AI might be able to make a contribution:

    From: AI-assisted peer review published under a CC 4.0 licence.

    A blog summarising the paper states: 'Manuscript submissions to peer-review journals are growing roughly 6% annually. Every year, over 15 million hours are spent on reviewing manuscripts previously rejected and then resubmitted to other journals....even using only rather superficial metrics to perform the training, the machine learning system was often able to successfully predict the peer review outcome reached as a result of human reviewers’ recommendations. In other words, there was a strong correlation between word distribution, readability and formatting scores, and the outcome of the review process as a whole. Thus, if a manuscript was well written, used appropriate terminology and was well presented, it was more likely to be accepted'.

    While the Frontiers' Artificial Intelligence Review Assistant has been shown to highlight 'where critical decisions need to be made, this technology does not replace researchers but empowers them to make editorial decisions more effectively. It will support editors and reviewers by directing their attention to potential issues in manuscripts. These issues can then be addressed or clarified as part of the manuscript review'. Authors can also access and be guided by this system.

    The World Association of Medical Editors (WAME) have a set of Recommendations on Chatbots and Generative Artificial Intelligence in Relation to Scholarly Publications. The five recommendations stipulate that  1) Only humans can be authors; 2) Authors should acknowledge the sources of their materials; 3) Authors must take public responsibility for their work; 4) Editors and reviewers should specify, to authors and each other, any use of chatbots in evaluation of the manuscript and generation of reviews and correspondence; and 5) Editors need appropriate digital tools to deal with the effects of chatbots on publishing. Unfortunately, any content fed to a chatbot is retained and not kept confidential, so using it for review is potentially problematic.

    You may want to explore this set of slides which cover Artificial Intelligence (AI) and associated tools and policies in editorial decision making, from the Academy of Science in South Africa.

    The future

    At the time of writing, the use of AI can be summarised as in this blog: Will ChatGPT Disrupt Peer Review? 'The integration of AI tools into the peer review process can be beneficial in assisting with certain tasks such as language editing and conflict of interest detection. However, the use of AI tools must be continually evaluated and responsibly implemented to ensure that they are not perpetuating biases or impacting the quality and reliability of scholarly literature. The expertise and judgment of human reviewers will always be essential in ensuring the rigor and dependability of the peer review process, and the continued integration of AI tools should be viewed as a complementary tool rather than a replacement'.

    Ethical considerations: Here is an interesting article published on an open platform: A Critical Examination of the Ethics of AI-Mediated Peer Review. It '...emphasizes the need to critically assess the legitimacy of AI-driven peer review, addressing the benefits and downsides...' It also makes the point that 'human peer review systems are also fraught with related problems, such as biases, abuses, and a lack of transparency, which already diminish credibility.'

    However in view of the speedy development in AI - watch this space!!!!


  • How to review a journal article


    You might want to access slides and recordings of a Zoom session on how to review a journal article - from when the course was previously offered: Access the slides here and recording here.


    Once you have been asked to be a reviewer, there are a number of important issues to consider. Here is a list  from PREreview in What makes a good reviewer:

    'The number one attribute editors look for in a reviewer is an appropriate level of expertise within the fields of study referenced by the manuscript. Other traits that we believe are equally important to qualify you as a “good” reviewer are the following:

    • Respectful. A good reviewer values respect above all and knows not to make their peers feel diminished or personally attacked by disrespectful comments.
    • Constructive. A good reviewer ensures that their feedback is constructive and actionable so that authors can easily respond to the feedback and possibly integrate the suggestions into the final publication.
    • Honest. A good reviewer knows that constructive does not mean they need to lie or only bring up positive comments. It means they need to write their suggestions in a way that is not insulting to the authors and that can lead to their easy integration in the manuscript. Constructive negative comments followed by examples and suggestions on how to improve the issue are welcome.
    • Clear. A good reviewer strives to present suggestions in a clear language, avoiding jargon and, when possible, providing examples and links to additional information that can help the authors make an informed decision on whether or not to integrate such suggestions.
    • Humble. A good reviewer is willing to be wrong and corrects themselves along the way.
    • Aware. A good reviewer is self-reflective and takes time to assess their biases and examines how they think and operate in the world.'

    We strongly recommend that you explore the Open Reviewers Toolkit. The guides are openly available for download on Zenodo under CC-BY 4.0 license. Each of the PREreview guides in the toolkit are also available for download below - Note: you will have to download each of these guides from the course site to gain a course completion certificate.

    Other review guides.

    How to Write a Peer Review from PLoS. A very helpful and focused guide that 'provides quick tips for writing and organizing your reviewer report'.

    Reviewing journal articles - by David Pannell. 'This post addresses various issues related to the journal reviewing/refereeing process, including tips on how to go about conducting a high-quality review efficiently and fairly.'

    Examples of journal policies and practices

    The South African Journal of Science has a number of policies and practices: Peer review process; Editing peer reviews; Peer review mentoring; Publishing peer review reports; and Guidelines for Reviewers.

    We also recommend this resource: 'What every new reviewer should know about peer review: A workshop hosted by the South African Journal of Science aimed at early career researchers with little or no experience in peer reviewing journal articles.' You can explore the slides or access the recording.

  • Performing a peer review: assignment


    You might want to view the slides and recording of a Zoom meeting to discuss performing a peer review assignment, which was held during a previous offering of this course. See the presentation slides here, and the session recording here.

    This section of the course builds on the previous one. Below you will see a journal article and a proforma with you  can use to complete a review of this paper. The assignment is to submit your review. Download the proforma, fill in the spaces, save the file and upload it to the assignment site. Once you have submitted your assignment, you will be able to access our model answer. If you have any questions, there is a forum for you to ask questions.

  • Gain a certificate