Reviewing for JOSS

Firstly, thank you so much for agreeing to review for the Journal of Open Source Software (JOSS), we’re delighted to have your help. This document is designed to outline our editorial guidelines and help you understand our requirements for accepting a submission into the JOSS. Our review process is based on a tried-and-tested approach of the rOpenSci collaboration.

Guiding principles

We like to think of JOSS as a ‘developer friendly’ journal. That is, if the submitting authors have followed best practices (have documentation, tests, continuous integration, and a license) then their review should be rapid.

For those submissions that don’t quite meet the bar, please try to give clear feedback on how authors could improve their submission. A key goal of JOSS is to raise the quality of research software generally and you (the experienced reviewer) are well placed to give this feedback.

We ask reviewers to complete their reviews in 4-6 weeks, and we generally ask the same commitment from authors: they should respond to your feedback within 2 weeks and complete requested changes within 4-6 weeks, unless otherwise negotiated with the editor (for example, if the changes requested are particularly substantial). If an author becomes unresponsive, the editor will follow up to keep the review moving forward.

A JOSS review involves checking submissions against a checklist covering software quality, development practices, and required paper content. This should be objective, not subjective; it should be based on the materials in the submission as perceived without distortion by personal feelings, prejudices, or interpretations.

We encourage reviewers to file issues against the submitted repository’s issue tracker. When you have completed your review, please leave a comment in the review issue saying so.

You can include in your review links to any new issues that you the reviewer believe to be impeding the acceptance of the repository. (Similarly, if the submitted repository is a GitHub repository, mentioning the review issue URL in the submitted repository’s issue tracker will create a mention in the review issue’s history.)

Time Expectations

We aim for the first pass of reviews to be completed within about 2-4 weeks. Please let the editor know if you require more time. It is best to be up front and clear about your availability. The more the editor knows, the more they can help work with you and the authors to get the papers through the process.

You can also use EditorialBot (our bot) to set automatic reminders if you know you’ll be away for a known period of time.

JOSS Conflict of Interest Policy

The definition of a conflict of Interest in peer review is a circumstance that makes you “unable to make an impartial scientific judgment or evaluation.” (PNAS Conflict of Interest Policy). JOSS is concerned with avoiding any actual conflicts of interest, and being sufficiently transparent that we avoid the appearance of conflicts of interest as well.

As a reviewer (or editor), COIs are your present or previous association with any authors of a submission: recent (past four years) collaborators in funded research or work that is published; and lifetime for the family members, business partners, and thesis student/advisor or mentor. In addition, your recent (past year) association with the same organization of a submitter is a COI, for example, being employed at the same institution.

If you have a conflict of interest with a submission, you should disclose the specific reason to the submissions’ editor (or to an EiC if you are an editor, or to the other EiCs if you are an EiC). This may lead to you not being able to review or edit the submission, but some conflicts may be recorded and then waived, and if you think you are able to make an impartial assessment of the work, you should request that the conflict be waived. For example, if you and a submitter were two of 2000 authors of a high energy physics paper but did not actually collaborate. Or if you and a submitter worked together 6 years ago, but due to delays in the publishing industry, a paper from that collaboration with both of you as authors was published 2 years ago. Or if you and a submitter are both employed by the same very large organization but in different units without any knowledge of each other.

Declaring actual, perceived, and potential conflicts of interest is required under professional ethics. If in doubt: ask the editors.