Skip to Main Content

Systematic review guide

A step by step guide to doing a systematic review

Test your search results

As you've probably discovered in creating your search strategy so far, searching is an iterative process. Database searching is as much an art as a science so a mandatory part of this process is testing and refining search strategies. You need to be confident your searches are working as you expect and giving you the "right" results before you start translating your master search into other databases and doing supplementary searching.

  • While a bit of "noise" (results that are correctly identified but not relevant) is expected in systematic search results, if your search is too sensitive you will find screening a long and tiresome task. Too much "noise" ends up as 1000s of irrelevant references you have to scan over and subsquently exclude after discovering they are totally unrelated to your topic.
  • A search that is too specific will yield a suspiciously smaller number of results all which seem very relevant, meaning you have likely missed unknown yet relevant results because your search has been too narrow or precise in its approach.
  • Long, complex search strings and strategies are also vulnerable to spelling mistakes, accidental omissions of search terms and incorrectly used boolean combinations, which again means you will have missed identifying relevant results.

The consequences of not properly testing searches can be severe!

If issues with searches are discovered during later review stages (often when an expected article has not appeared during the ti/ab screening) the resulting processes are very time consuming. To continue conduct your review correctly you will have to delete all previous results from reference management software and/or Covidence, rewrite, retest and rerun your searches (so as to properly identify any previously missed material) and begin screening from scratch using your new more comprehensive search results.

Testing and refining

The scoping searches done early in your review have likely helped you identify your “gold set", which is also sometimes called the “test set” or “gold standard papers”. This is a small selection of around 5-10 papers that are highly relevant to your topic and closely match your research question. They are studies that would be essential to be identified in your final searches and you would expect to be included in your analysis.

This gold set can be used when testing your final search strategies by checking your search strategy has actually found them. You must perform a gold set test on your master search in Ovid MEDLINE before moving on to other databases. To perform the test:

  1. Run your search stategy
  2. Search for a gold set article by it's title (if it is not indexed move on to the next article).
  3.  Check the article is included in your search strategy results by combining your final search strategy results line with the gold set article using AND.
  4.  Repeat with the rest of your gold set, the more articles you test, the more assurance you will have your saerch strategy is not missing relevant results.
  5. If articles are present in the database but not in your search results, you must play the detective to understand why. Check the articles' full records to see what keywords or subject headings your search is missing or if limits or filters have filtered out the results. You will need to change your search to include what has been missing.
  6.  Don't panic if MEDLINE doesn't contain one or more of your gold set articles, they may be indexed in the next database you test your strategy in. We suggest testing each strategy in each database to see you have translated your original strategy correctly or if it needs further adaptation.

Once you are confident your strategy is retrieving the known studies it was designed to find, a common question is "what is the right number of results?".  A suggested "good" number is to end up with around or under 1000 results in a database. This is to avoid a lengthy screening process. But even then it is a simple answer to a compex issue. EMBASE will generally retrieve more results than MEDLINE for the same strategy. A smaller, more niche database will likely bring back less results as it may index less journals. When you eventually combine your results across your searches you will find that many are duplicates but you won't know how many unique references you will need to screen until this point. Even a well focused question could reveal a lot of relevant results if it is being widely researched. In that case what might be considered too many results for one review are still the "right" number in this case.

In the end the "right" number of results is the results that contain all the studies out there that will help you answer your question and this figure is specific to every individual review.
Have you designed a strategy that is the right balance of sensitive and specific? Some questions are difficult to create a strategy for that doesn't exclude the studies you need because the information required is not explicity described in titles or abstracts. In this case you may need to have a more sensitive and less specific search than normal and put up with screening 1000s of references at Title/Abstract level so as to not miss anything. Conversely, a search for a questions where the concepts are likely to be well described that retrieves too many "noise" results is probably balanced wrongly and a lengthy screening process will be wasting time, rather than revealing hidden gems like in the example previously discussed. In this case you would want to adapt your search to make it more specific, to keep the relevant results and exclude the noise.

Section 6 of the Library's Medicine Literature Search Tutorial guides you through understanding if you have the right results for your question and provides techniques to continue to adapt and refine your search to solve common problems such as too few or many results.