Section outline


  •
    You might want to access slides and recordings of a session on the fundamentals and challenges of the peer review process - from when the course was previously offered: Access the slides here and the recording here. These are optional activities

    The fundamentals of the process

    "Scholarly peer review or academic peer review (also known as refereeing) is the process of having a draft version of a researcher's methods and findings reviewed (usually anonymously) by experts (or "peers") in the same field. Peer review is widely used for helping the academic publisher (that is, the editor-in-chief, the editorial board or the program committee) decide whether the work should be accepted, considered acceptable with revisions, or rejected for official publication in an academic journal, a monograph or in the proceedings of an academic conference. If the identities of authors are not revealed to each other, the procedure is called dual-anonymous peer review (from Scholarly peer review).

    Here is a diagram which describes a typical peer review process:


    From Mohini & Ahmed, published under CC BY 4.0

    Each journal takes a slightly different approach, but this is the way the process flows. Peer review is not limited to publishing in journals. The same procedures are applied to ensure content quality in books and there is also a rising practice of reviewing preprints.

    We will be discussing the benefits and limitations of various parts of this process later in the course, in particular the alternative types of review - single of double blind, transparent or fully open. We will also have the opportunity to practice reviewing and posting open reviews.

    Challenges of the peer review process

    As Richard Smith, Editor of the British Medical Journal for many years has said, 'Peer review is at the heart of the processes of not just medical journals but of all of science. It is the method by which grants are allocated, papers published, academics promoted, and Nobel prizes won. Yet it is hard to define'...'But it is something to do with a grant application or a paper being scrutinized by a third party—who is neither the author nor the person making a judgement on whether a grant should be given or a paper published.'

    Here are some of the problems as identified by Richard Smith and others:

    Exploitative. Publishing companies can make large profits from academic journals. In 2022, the publisher Elsevier is reported to have made a profit of 1.2 Billion Euros with a profit margin of 37% (higher than that of Google at 21% or Apple at 25%). For some context, the whole of the European Research Council's annual budget is 2.4 Billion Euros. Meanwhile most of the journal editors and reviewers are academics who take on these tasks as volunteers.

    Slow. It can take a year or more from submitting a paper until it is published. Much of this time is taken by the review process.

    Inconsistent. A number of studies have shown differences between reviewers in their assessments. Of course this may reflect good choice of the reviewers to represent different skill sets, but is more likely to represent observer variation. This inconsistency may represent bias - towards papers that come from recognised authors or institutions, or according to the country or gender of the authors. There is a difference of opinion and practice in blinding the reviewers to the names and institutions of the authors. There is quite a large literature on the topic of bias.

    (There are a number of other possible biases in the publication of research - such as the difficulty in getting negative results published or improved chance of acceptance if the same journal is quoted among the references. These are editorial decisions - editors are the first arbiters of whether to send a submitted paper to referees and whether to publish based on reviewer comments.)

    Lack of transparency. Most of the processes involved in the review and acceptance of submitted papers are closed to scrutiny. Many of the issues discussed above can be resolved by increasing transparency - one of the options shown above in the diagram from BMC. This opens the door to Open Publishing and Open Science, where open reviewing has become a key part of the open access publishing movement.

    Incentives and responsibilities.
    As mentioned in the Introduction, it is important that researchers act as peer reviewers to maintain the scientific integrity of the system. However, even though the publisher may make a profit the reviewers and editors are usually volunteers. Editors often find it frustrating when potential reviewers refuse to help. There are ways of offering incentives and rewards to reviewers to encourage their participation in the system. We will discuss open publishing of reviews in the next course, in which case credit can be given to published reviews.

    • Reflection on the peer review process and its challenges Forum
      Students must
      View
      Make forum posts: 1

      Please reflect on and share any experiences you may have of peer review. Are there any parts of the peer review process that surprise you? Does this process accord with your own experience as an author or reviewer, or editor? What do you think are the benefits and problems of the process as described? Can you add to the challenges mentioned above, and do you have any suggestions for change to the system.

      Posting your reflection is a requirement for gaining a certificate.