Rayyan for Systematic Reviews

Once inclusion and exclusion criteria have been agreed, it is time to start screening your articles.  

Rayyan offers simple keyboard shortcuts to speed up this process:

To make a decision on a particular article, select it and then use one of these keyboard shortcuts.  While it is possible to change your decision (from, say, include to exclude) it is not possible to reset the article and remove its screened status once a decision has been added.

Rayyan allows screeners to label articles: this can be used to

1. indicate a screener's reason for excluding an article, either by selecting a reason from a pre-defined list, or by creating a new reason;
2. specify a reason for inclusion;
3. highlight particularly pertinent points raised by the article which may be useful during data extraction.

The step-by-step guide below explains this process.

Adding a pre-defined or custom screening decision

STEP 1: Select the article being screened.  Once the article is displayed in the pane below the listing, select "Reason" in the box above the article abstract.  A pre-defined list of exclusion reasons will appear. Select one of the pre-defined exclusion reasons that applies to your article.

STEP 2: If you feel that none of these reasons is appropriate, you can add your own.  For example, you may want to only include systematic reviews, and to mark that the article does not have this study design.  To do this, type your reasons in the box.  Rayyan will then invite you to add this to the list of exclusion reasons, in this case the screener would do so by clicking on the "+ Not systematic review (new)" box that appears.

Your article will then be marked as excluded, and a label with your exclusion reason will then be added to the article in red.

Any custom exclusion reason added by a screener will NOT be available to all other screeners in the project - we therefore recommend that your team agree an appropriate wording for your exclusion criteria to avoid confusion.

Removing a custom screening decision

STEP 1:  It is also possible to remove a screening reason from the list.  To do this, click on the "Reason" box, just as if you were adding a reason for exclusion.  At this point, we see an option at the bottom of the drop down list to remove our custom exclusion reason.  We can remove the custom-added reason by simply clicking on it.

Inclusion labels

It is also possible to label articles, for example to indicate an inclusion reason, or to highlight a particularly pertinent point it raises which you may later use during your data extraction stage.  Labels work in exactly the same way as exclusion reasons, except that they are labelled in blue.

Rayyan and PRISMA

To facilitate management of numbers for your PRISMA diagram at manuscript time, Rayyan will keep a count of numbers of articles tagged with each exclusion reason. While screening is blinded, the numbers shown will be for your screening results only.  Upon lifting of blinding, the total numbers of articles for all screeners will be displayed.

Researchers can also screen articles with the Rayyan mobile app, which is available on the Google Play store.  At the time of writing, the Rayyan app appears to have been removed from the Apple App Store.  

The mobile app has limited functionality and only allows screening of articles, and addition of labels.  The review owner can also change the blinding mode for the review through the mobile app.

After logging into Rayyan through the app, select the review.  A listing of articles to be screened will then be displayed, in the format shown on the left.

Once screening has been completed, it is time for the reviewers to make final decisions for those articles where they currently disagree. This practice is known as resolving conflicts, and is usually performed either by discussion between reviewers, or by requesting a tie-breaking decision from a reviewer who has not previously been involved in screening articles.

Switching off blinding

During the screening process, blinding is on.  This means each reviewer can only view their own decisions, and cannot view the decisions made by other reviewers.  Before the process of resolving conflicts can be started, blinding needs to be switched off, which can only be done by the review owner (the person who originally created the review in Rayyan).  This process is shown at the start of the video below.

Resolution of conflicts

Once blinding has been switched off, reviewers are able to see those articles where there is currently disagreement by selecting Conflicts in the left hand bar.  As agreement is reached on each article, it is then a simple matter for the relevant reviewer to change their opinion.  In the example given in the video, reviewer Martin is changing his decision, and including several articles which he had previously elected to exclude, by using the keyboard shortcuts given above.

Rayyan does not offer PDF annotation capability. You must rely on the labeling system to either provide annotations for your decisions or use labels to categories the citations during screening for further analysis, export, etc.