Skip to Main Content

Systematic review guide

A step by step guide to doing a systematic review

Supplementary searching

Databases will be the main sources for relevant material for your review, but they should not necessarily be the only search method. Supplementary search methods aim to mitigate publication bias (the practice of publishing studies with positive effects and rejecting studies with zero or negative effects) and the inherent bias in individual databases caused inclusion decisions made by their publishers and incomplete or incorrect indexing of articles. They include "citation chaining" and "hand searching"
These methods are trying to identify published studies that have been missed by database and trial register searching, not grey literature such as policy documents, theses and other such "unpublished" material 

The topic, scope and resourcing of your review will determine if supplementary searching should be undertaken, and which methods will be most suitable. Not every Systematic Review will benefit from supplementary searches. A new or niche research area may benefit from more rigorous supplementary searching. A well trodden topic may benefit less. 

Supplementary searches can be seen as a searching "safety net". They are useful in producing unique results in the case of hard to identify documents, particularly in topics and fields with non-standardised language( or abstract conceptulisations of phenomena such as psychology, management and social science) and will likely improve comprehensiveness over database searching alone. The TARCis Statement's Recommendations 2 & 3 will help you to judge if you should undertake them or not.

Conducting supplementary searches

Citation chaining is a method that allows you to search forward (snowballing) or backward (pearling) in time. Any new (previously unidentified) studies that meet your eligibility criteria found by citation chaining should be included in your data analysis.

  • Forward citation searching involves identifying articles that have cited your Included studies (those that will be included in the data analysis). Scopus is the first line option for forward citation searching. Find the study in Scopus and open the full article record, then look to the top right hand corner, the number of citing articles will be available. Select "View all citing documents" to see the full list.
  • Backward citation searching looks for the citations in the references lists in your Included studies. This can also be done in Scopus. Search for each article, opening the full record of each one, and scrolling down to the bottom of the page to see the reference list. If the article is not indexed in Scopus try the database in which the article you are searching from was originally found.

If the list of included articles is long, you may consider prioritising forward/backward citation search to only the most cited or most current studies.
CINAHL, PsycINFO and Google Scholar include cited references in their records so these are suitable for citation searching. The TARCiS Statement recommendations 4, 5 and 6 will help your decision making on citation searching methods, including which/how many citation indexes to search with. 

'Hand-searching' involves looking through the contents pages of journals and conference proceedings/abstracts which you know focus on your topic area. Whilst this used to be done by going page by page through a hard copy (hence the name) this can of course now be done online via journal and conference websites. This process can identify articles (and other items, e.g. letters) which have not been included in electronic databases, and those which are not indexed or have been indexed incorrectly. Deciding which journals/sites to search through in this way can be done by analysing the results of your database searches to see which journals contain the largest number of relevant studies. 2-3 Journals should be sufficient.

More information on Handsearching is available in the Cochrane Handbook Technical Supplement to Chapter 4 section 1.3.1.

Managing, documenting and reporting supplementary searches

Just as for database searching, methods used for supplementary searching in Systematic Reviews should be planned in advance and clearly reported. This means you will need to have a strategy (how and where) before searching and keep detailed records along the way, as well as ensuring you are properly managing search results.

You should manage supplementary search results in Reference Manager software, including deduplication. A spreadsheet will be a useful way to record where and when you searched and how many records were found. 

Numbers relating to indentification, retrieval and screening must all be documented (incuding reasons for exclusion) and should be reported in PRISMA 2020 flow diagram for new systematic reviews which included searches of databases, registers and other sources. Note that supplementary search reporting is recorded on the right hand side of this flowchart. Studies that are eligable for review inclusion are then added to the left hand side of the flowchart, joining the included studies from database searching.

As well reporting via the PRISMA diagram, you should describe your supplementary searches in your methods section.
Recommendation 10 of the TARCiS Statement details the information to report on.

Using Covidence with supplementary search results

You can use Covidence to aid screening but you must not do this in your main review. Instead, start a seperate review for records discovered via Grey literature and and other supplementary searching methods. This will allow the automatic PRISMA flowchart recording to stay separate from the PRISMA diagram of your main review containing database search results.

If you are also using Covidence for data extraction and Quality Assessment activites, you will want to add your your final group of included supplmentary reports to those found via database searches.

  • In your supplementary search Covidence review, export your  included studies as a RIS file.
  • Return to your main review, go to import, find the RIS file and choose the import into option of "Included", adding new source type as required.
  • Your included supplementary searches will now be added to your other included studies.