Dedicated to the production of quality scholarship in collegiate- or campus-based emergency medical services (CBEMS), JCEMS strives to uphold the most rigorous standards for peer review. “Peer review” is a commonly-used term in academic medicine, but there is often confusion regarding what peer review is and why or how it is conducted. In this editorial, we will (1) introduce peer review and articulate its importance, (2) describe the unique JCEMS peer review process, and (3) discuss how JCEMS seeks to confront the challenges and limitations associated with peer review.
Introduction to Peer Review
In the context of scholarly literature, peer review refers to the pre-publication review of submitted manuscripts by independent subject-matter experts (ie, the “peers”).1,2 When peer review is conducted properly, reviewers are well-versed in the research methods described and have a thorough knowledge of the existing body of literature relevant to the topic. Reviewers rigorously examine a manuscript for its quality, accuracy, scope, methodological rigor, depth of research, originality/novelty, and style/organization. Reviewers provide journal editors with their opinions regarding whether or not a manuscript should be published. Reviewers may also provide recommendations for authors to improve a manuscript prior to publication.
Peer review therefore serves two broad functions.2 First, peer review aids journal editors in determining which manuscripts are acceptable to publish.1,2 Manuscript topics and research methods can be wide ranging – even for a journal focused on a narrow field – and it would be unreasonable to expect editors to have sufficiently broad expertise to critically evaluate all aspects of every submitted manuscript. The input of carefully selected subject-matter experts is therefore necessary for editors to make informed decisions. In effect, the peer review process serves as a filter to select manuscripts that are high quality, interesting, and relevant ie, peer review serves as a quality control mechanism.1 Second, peer review serves to strengthen the quality of submitted manuscripts.1,2 Authors are often given the opportunity to revise their manuscript – based on feedback from reviewers – resulting in a published product that is superior to the initial submission. For these reasons, peer review remains the standard for journals that publish scientific research and other scholarly articles, and readers often consider the peer-reviewed status of a journal to be a marker of credibility.1
Peer review is particularly important for research and scholarship in the CBEMS community. As we discussed in the inaugural issue of JCEMS, there is a scarcity of scholarly literature that is focused on CBEMS.3 As a result, the few published articles on CBEMS are frequently cited in the literature and discussed in practice. Each submitted manuscript must therefore be critically evaluated to ensure its accuracy and reliability. Moreover, CBEMS research may be conducted by early-stage scholars without substantial experience in EMS research and scholarly writing. Coupled with the JCEMS Research Mentorship Program,4 peer review serves as a tool to provide developing EMS scholars with guidance in producing quality scholarship.
JCEMS Peer Review Process
There is a wide diversity of peer review processes amongst scholarly journals,1 and JCEMS is proud to offer a transparent description of its own process. For all manuscripts classified as Original Research, Case Reports, and Reviews, JCEMS employs what is known as double-blind peer review.† In brief, each submitted manuscript is reviewed by at least two independent reviewers. To promote unbiased and appropriately critical reviews, neither author(s) nor reviewers are made aware of the identify of each other – hence the name “double-blind.” Additional details on the process follow and are described in the schematic (Figure 1).
Initial Review
Manuscripts are initially reviewed by editorial staff for relevance to the collegiate EMS community and suitability for JCEMS. Manuscripts may be declined at this stage without further review.
Identification of Reviewers
Each manuscript that fits the scope of JCEMS is assigned to at least two reviewers – subject-matter experts who do not serve on the JCEMS Editorial Board or staff. Reviewers are identified based on established records of expertise in the given manuscript topic, as often – but not exclusively – evidenced by relevant publications, presentations, advanced academic degrees, and prior experience as an editor or reviewer. All manuscripts with a clinical focus are reviewed by physicians and/or advanced practice providers. In addition, all original research manuscripts are reviewed by established investigators well-versed in the research methodology employed. For all manuscripts that present quantitative findings, a focused statistical review is performed by at least one reviewer with expertise in data analysis. JCEMS will occasionally invite undergraduate students or recent graduates to review manuscripts in select cases where the individual has relevant expertise (eg, a CBEMS leader who spearheaded a CPR training program might be invited to review a manuscript on campus CPR outreach). In practice, it is not uncommon for a JCEMS manuscript to be reviewed by three or four reviewers, particularly if a manuscript spans diverse subfields.
De-identification of Manuscripts
Before a manuscript is sent to reviewers, the editorial staff ensures that the author(s) have removed all identifying details from the manuscript; any remaining details that might enable one to reasonably infer the identity of the author(s) or their institution(s)/organization(s) are removed. The anonymity of author(s) is crucial to maintaining objectivity during peer review – reviewers are expected to base their evaluations solely on the quality of the manuscript and research, not on author(s)’ reputation, academic status, gender, race, country of origin, etc.
Manuscript Review
Reviewers evaluate manuscripts for their quality, accuracy, scope, methodological rigor, depth of research, originality/novelty, style/organization, and practical implications for the collegiate EMS community. Reviewers recommend to the JCEMS Editors that manuscripts be rejected, accepted pending major revisions, accepted pending minor revisions, or accepted without revision. If a reviewer believes that revisions are necessary, the reviewer is expected to provide recommendations for how the author(s) can improve their manuscript. The JCEMS Editors (ie, Editor-in-Chief and Executive Editor) and/or Editorial Board members will also conduct their own review with particular consideration for A) whether the research and writing were conducted in accordance with ethical guidelines (eg, was Institutional Review Board approval obtained if necessary) and B) whether any disclosed conflicts of interest may pose a threat to the validity of the manuscript.
Initial Decision
The Editors carefully consider the comments and recommendations of the reviewers. In cases where reviewers’ recommendations conflict with each other – or in cases where aspects of the manuscript are not sufficiently evaluated – the Editors may request additional reviews. Once sufficient reviews are obtained, the Editors make an initial decision regarding publication, and authors are promptly notified. It is exceedingly rare for manuscripts to be initially accepted without requests for revisions.
Revision, Resubmission, and Continued Review
Authors typically have the opportunity to revise and resubmit their manuscripts. Minor revisions might include the incorporation of additional details, corrections of misstatements, or minor alterations in the writing, organization, and style. Major revisions might include the need for additional data, the reinterpretation of findings, significant additions from the literature, or major restructuring of the organization. After revising their manuscript, author(s) may re-submit their work with accompanying replies to reviewers’ comments. The Editors evaluate the revisions and may send the manuscript back to the original reviewers, or additional reviewers, particularly if major revisions were incorporated. It is not uncommon for multiple rounds of revisions to be requested. The revision, resubmission, and continued review process will continue until a final decision is made.
Final Decision
The final decision to accept or reject a given manuscript is at the discretion of the Editor-in-Chief. Manuscripts that are accepted undergo copy-editing and the final edited manuscript is approved by the author(s) prior to publication.
§
JCEMS acknowledges that many authors may lack experience submitting to a peer-reviewed journal. New authors should be aware that their manuscript may not be accepted and that multiple sets of revisions may be requested. While the process is challenging, JCEMS is proud to offer authors a high degree of individualized attention – a rarity amongst scholarly journals. JCEMS provides authors with extensive guidance throughout the process, including opportunities to discuss new ideas for manuscripts, informal reviews of rough drafts, detailed recommendations to improve content, and advice on replying to reviewers’ comments. The level of attention that we offer authors reflects our commitment to the education of new researchers and the development of a scholarly culture within the CBEMS community.
Challenges and Limitations
Peer review is rightly regarded as a critical component of academic publishing, but its challenges and limitations must be acknowledged. First, peer review relies on the contributions of experts, who are almost always uncompensated.1 Identifying suitable experts can be challenging, particularly if a manuscript discusses a novel concept or method, or focuses on a niche topic. Suitable reviewers may not be readily available or have sufficient time to complete an in-depth review.1 In addition, without financial compensation or public recognition, there are concerns that reviewers may not be motivated to conduct a thorough review.5
Additional challenges exist surrounding the possibility of bias in the review process.1,6 Even in double-blind peer review, reviewers may be able to infer the identity of the author(s) or their institution(s) based on the subject matter, writing style, citations, or other details – for small fields, the concern is especially relevant.6 If author(s) or their institution(s) are identified, reviewers may bias their evaluation – positively or negatively – based on characteristics unrelated to the quality of the manuscript (eg, prestige of the author(s)’ academic institution or the author(s)’ gender).7 Bias may also enter the process without identification of the author(s) or their institution(s). For example, a reviewer’s evaluation may be consciously or unconsciously biased by a financial conflict of interest, a personal belief that conflicts with statements expressed by the author(s), or a desire to advance their own research careers at the expense of the author(s). Beyond bias, cases of outright peer review fraud have also been uncovered in which authors have created fake email accounts to review their own manuscripts.8
Novel forms of peer review have been developed in an effort to overcome the challenges discussed. For example, in “open review,” reviewers and authors are informed of each other’s identities, and reviewers’ names are typically noted in published articles.6,9 It is thought that reviewers will be motivated to conduct higher quality reviews if they will receive recognition and their names will be publicly associated with published work. The natural concern with open review, however, is that the risk of bias increases with the disclosure of authors’ identities. Publicly-identified reviewers may also be hesitant to provide negative reviews, even if warranted, out of fear of retribution or ill-treatment from displeased authors.9
As the Editors of JCEMS, we firmly stand by the JCEMS peer review process while acknowledging that all forms of peer review have advantages and drawbacks. We believe that double-blind peer review minimizes bias, promotes fairness, and encourages appropriate criticism. Despite concerns surrounding the motivation of reviewers in many fields, JCEMS is fortunate to be able to draw from an expanding community of scholars, many of whom have served in CBEMS organizations. Our reviewers are motivated to contribute to both the advancement of a growing field of research as well as the development of early-stage scholars.
Unfortunately, even motivated, experienced, and unbiased reviewers with relevant subject-matter expertise may fail to identify inaccuracies or areas for improvement in a manuscript; conversely, excellent reviewers may fail to identify the importance or innovation of a manuscript.1 The peer-reviewed status of a JCEMS manuscript should never be relied on as the sole indicator of its quality, accuracy, or reliability. We encourage readers to critically appraise the work published in JCEMS and, in effect, perform their own review – readers must determine if the research design, interpretation of findings, and conclusions are appropriate and actionable. Readers are encouraged to develop, critique, and discuss the implications of published work through formal Letters to the Editor, which may be published in JCEMS, or through informal “comments” on our website. The peer review process should never be considered complete, but rather a continuous process pre- and post-publication.
Conclusions
The study of peer review is itself an evolving field of scientific inquiry. Researchers are actively investigating methods to improve quality and equity throughout the process.6,10 The JCEMS editorial team actively follows developments in the science of peer review, and we are open to modifying our process as new findings reveal opportunities for improvement.
While the JCEMS peer review process may develop over time, our commitment to promoting a research culture in the CBEMS community will remain constant. Peer review serves as a filter for publication and strengthens the quality of submitted content. Peer review is one tool in our arsenal to ensure that published CBEMS scholarship meets the level of accuracy, reliability, and credibility that the CBEMS community deserves.
Acknowledgments
We thank the Editorial Board for their contributions to the development of the JCEMS peer review process. We are also deeply grateful for all anonymous, independent reviewers who dedicated their time and expertise to contribute to collegiate EMS scholarship.
Footnotes
† JCEMS readers should note that note all content published in JCEMS has undergone peer review. News pieces and Editorials, as well as articles classified as Perspectives and Opinions or Advice and Practice, may be reviewed solely by Editors and Editorial Board members. Articles in these categories are designed to present commentary or the personal opinions of author(s), rather than scientific research or clinical recommendations. In practice, a manuscript in one of these categories would undergo double-blind peer review in the event that it provides clinical information, a description of a novel program, or in-depth reference to the literature.
References
1. Hames I. Peer Review and Manuscript Management in Scientific Journals: Guidelines for Good Practice. Malden, MA: Blackwell Publishing; 2007.
2. International Committee of Medical Journal Editors. Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly Work in Medical Journals. https://www.icmje.org. Updated November 29, 2018. Accessed February 15, 2019.
3. Friedman NMG, Dingler BJ, Nable JV, Koenig GJ. Research and Scholarship in Collegiate Emergency Medical Services: Current State and Opportunities for Progress. J Coll Emerg Med Serv. 2018;1(1):5-12. https://doi.org/10.30542/JCEMS.2018.01.01.01
4. Friedman NMG, Dingler BJ. The JCEMS Research Mentorship Program: Objectives and Opportunities. J Coll Emerg Med Serv. 2018;1(2):4-7. https://doi.org/10.30542/JCEMS.2018.01.02.01
5. Nature Publishing Group (No author listed). Review Rewards. Nature. 2014;514(7522):274. https://doi.org/10.1038/514274a
6. Rennie D. Make peer review scientific. Nature. 2016;535(7610):31-33. https://doi.org/10.1038/535031a
7. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. PNAS. 2017;114(48):12708-12713. https://doi.org/10.1073/pnas.170732311
8. Haug CJ. Peer-Review Fraud – Hacking the Scientific Publication Process. N Engl J Med. 2015;373(25):2393-2395. https://doi.org/10.1056/NEJMp1512330
9. DeCoursey F. The pros and cons of open peer review [web-only perspective]. Nature. Published on-line 2006. https://doi.org/10.1038/nature04991
10. Rennie D, Flanagin A. Three Decades of Peer Review Congresses. JAMA. 2018;319(4):350-353. https://doi.org/10.1001/jama.2017.20606
Author & Article Information
Brittany J. Dingler, BA, PA-S is the Executive Editor of JCEMS and a student in the Physician Assistant program at Drexel University.
Nicholas M.G. Friedman, BA, EMT is the Editor-in-Chief of JCEMS and the former director of Skidmore College EMS.
Author Affiliations: From The Journal of Collegiate Emergency Medical Services, Philadelphia, PA, USA (B.J.D.,N.M.G.F); Physician Assistant Program, Drexel University College of Nursing and Health Professions, Philadelphia, PA, USA (B.J.D.).
Address for Correspondence: Brittany J, Dingler, BA, PA-S. The Journal of Collegiate Emergency Medical Services, 117 N 15th St., Philadelphia, PA 19102, USA. E-mail: JCEMS@CollegeEMS.com
Conflicts of Interest/Funding Sources: By the JCEMS Submission Declaration Form, all authors are required to disclose all potential conflicts of interest and funding sources. N.M.G.F and B.J.D. serve in uncompensated editorial roles for JCEMS. All authors declared that they have no others conflicts of interest. All authors declared that they did not receive funding to conduct the research and/or writing associated with this work.
Authorship Criteria: By the JCEMS Submission Declaration Form, all authors are required to attest to meeting the four ICMJE.org authorship criteria: (1) Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND (2) Drafting the work or revising it critically for important intellectual content; AND (3) Final approval of the version to be published; AND (4) Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Submission History: Received February 15, 2019; accepted for publication February 15, 2019.
Published Online: February 21, 2019
Published in Print: February 22, 2019 (Volume 2: Issue 1)
Reviewer Information: In accordance with JCEMS editorial policy, Editorial manuscripts are reviewed by the JCEMS Editorial Board and published by the Editor-in-Chief or designee. JCEMS thanks the Editorial Board members who contributed to the review of this work.
Copyright: © 2018 Dingler & Friedman. This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. The full license is available at: https://creativecommons.org/licenses/by/4.0/
Electronic Link: https://doi.org/10.30542/JCEMS.2019.02.01.01
