A well-defined research question should be specific, clear, and answerable based on the studies included in your review.  Often, fixed frameworks can be used to structure the question effectively. Examples are:  PICO and SPIDER

PICO (often used for clinical reviews)

E.g. 'What is the effectiveness of cognitive-behavioral therapy (CBT) compared to medication in reducing symptoms of anxiety in adults?'

• P (Population): Adults;
• I (Intervention): CBT;
• C (Comparison): Medication;
• O (Outcome): Reducing symptoms of anxiety.

SPIDER (often used for qualitative reviews)

E.g. 'How do women who have experienced domestic violence perceive the effectiveness of support groups in improving their well-being?'

• S (Sample): Women who have experienced domestic violence;
• PI (Phenomenon of Interest): Perceptions of support group effectiveness;
• D (Design): Qualitative interviews or case studies;
• E (Evaluation): Improvement in well-being;
• R (Research Type): Qualitative.

Predefining the protocol ensures that the review is conducted systematically and consistently, minimizing subjective decisions during the process. It involves clearly outlining the objectives, methodology, and criteria for conducting the review before it begins.

This includes specifying:

• the research question;
• inclusion and exclusion criteria;
• search strategies;
• data extraction methods;
• the process for analyzing and synthesizing the evidence.

A checklist for systematic review protocols (PRISMA-P) is available at the PRISMA portal.

Registering the protocol in a publicly accessible database, like PROSPERO, allows others to assess the rigor of the review and helps avoid duplication of research efforts.

Inclusion | exclusion criteria help narrow the scope by ruling out studies that do not meet methodological standards or are published in non-academic sources, such as opinion pieces or grey literature with insufficient data. These parameters can relate to the subject matter of the studies or methodological and/or quality aspects.

Content criteria are directly related to the review's focus and include PICO or SPIDER elements such as

• population characteristics;
• intervention type;
• comparison;
• outcomes measured;
• study design;
• methodology.

Examples of non-content criteria:

• Language;
• Publication date range;
• Publication type (e.g. considering only peer-reviewed articles or excluding conference abstracts);
• Quality or risk of bias (e.g. assessing the methodological quality of studies using tools like the Cochrane Risk of Bias tool).

Clearly defining inclusion and exclusion criteria upfront helps to avoid bias, ensures transparency, and maintains the replicability of the review process.

Identify key databases relevant to your topic and construct a search string using relevant keywords, Boolean operators, and database-specific filters.

While the Discovery Service (the search bar on the University Library homepage) allows you to search many important databases simultaneously, it is not the best choice for subject-specific reviews that require a high level of detail and transparency in reporting.

It is advisable to search each database individually through its own interface. You can find an overview of the digital resources available through UHasselt in the e-sources list.

Document your searches

Make sure to document the search process carefully, including

• the search string used;
• any filters applied;
• the number of results.

• The PRISMA 2020 flow diagram is a useful tool to document results throughout the search and screening process.
• The 'save search' and 'save results' features in databases can help you keep track of this information accurately.
• 'Search alerts' can help you keep the results up-to-date throughout the search process.

Controlled vocabularies

Controlled vocabularies are standardized sets of terms, used to index and categorize articles consistently across databases, e.g. MeSH (Medical Subject Headings) in PubMed and Thesaurus Terms in databases like ProQuest or ERIC.

By utilizing these pre-defined terms, researchers can avoid the pitfalls of keyword variations or synonyms that may miss relevant studies. Controlled vocabularies ensure that related articles are grouped together, even if the authors use different terminology.

Snowballing

You can expand the scope of your literature review by identifying additional relevant studies from the references of the papers you have already found.

Two key approaches:

• backward snowballing: examine the reference lists of selected articles to discover earlier works that may be relevant.
• forward snowballing: use citation tracking tools (like Web of Science or the citation data in Discovery Service results) to identify later studies that have cited the paper in question.

This process helps uncover studies that might not have appeared in your initial database searches, especially in cases where terminology varies or relevant works are published in lesser-known journals.

Title | Abstract screening

At this stage, the elimination process focuses on core aspects that can be easily identified. The goal is quickly ruling out irrelevant studies while retaining those that might potentially meet the review's scope.

Titles and abstracts often provide only a limited amount of information, so decisions are preferably conservative, leaning towards retaining studies for further evaluation during the full-text screening phase.

Full text screening

This stage allows for a deeper assessment of whether a study fully aligns with the systematic review’s research question and the predefined inclusion and exclusion criteria.

Full-text screening also checks for methodological quality, like study design robustness, adequate sample sizes, and completeness of data reporting, thereby maintaining the integrity and validity of the review's findings.

Eliminating duplicates

When conducting a systematic review using multiple databases, it is common to encounter duplicate records, as the same studies may be indexed in more than one source.

To ensure efficiency and accuracy, it is crucial to identify and remove duplicates early in the process. Duplicate studies can skew the analysis and synthesis by inadvertently over-representing certain findings.

Many reference management tools, such as EndNote, Zotero, or specialized software like Covidence, have built-in features to detect and eliminate duplicates.

After the initial title and abstract screening, you will retrieve the full text of the remaining results.

Many bibliographic databases available at UHasselt have been equipped with link resolvers, allowing you to directly access the electronic version of a reference (if available through UHasselt) or providing alternative options for obtaining the publication, such as through interlibrary loan.

Additionally, the University Library has a subscription to LibKey Nomad, an extension that can be easily installed in your browser, providing quick and direct access to the scientific articles licensed by Hasselt University.

If you use bibliographic software for your systematic review, you can choose to store the PDFs within the tool.

After retrieval, the remaining articles are screened for close alignment with the research question and predetermined inclusion and exclusion criteria.

Evaluating the results in a systematic review is a critical step that involves assessing the quality, relevance, and impact of the included studies. This process typically begins with a thorough critical appraisal.

Systematically examine the quality of each included study for methodological rigor, potential bias, and reliability using standardized critical appraisal tools like:

• CASP checklists
• AMSTAR
• ROBIS checklist

Extract key data from the selected studies, such as author details, study design, sample size, and main findings. Use a pre-defined data extraction form to ensure consistency.

Organize and synthesize the data. Depending on the nature of your research, this could include qualitative synthesis, quantitative meta-analysis, or narrative synthesis.

In the reporting of a systematic review, several essential elements are required to ensure transparency, replicability, and comprehensiveness.

The PRISMA checklist is the most commonly used tool for ensuring that all critical elements are reported. PRISMA provides

• a 27-item checklist that covers each section of the systematic review;
• a flow diagram to guide authors through the process of transparent. and complete reporting.

By adhering to PRISMA guidelines, systematic reviews achieve a high standard of rigor and transparency.

Online

• Cochrane Handbook for Systematic Reviews of Interventions: This handbook is considered the gold standard for conducting systematic reviews, particularly in healthcare. It provides detailed guidance on each step of the systematic review process, from protocol development to data analysis.
• PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses): The PRISMA guidelines are widely adopted for reporting systematic reviews and meta-analyses. The website also offers resources for researchers to ensure transparency and reproducibility.
• JBI (Joanna Briggs Institute) Manual for Evidence Synthesis: The JBI manual offers guidance on conducting systematic reviews across various types of evidence.
• Centre for Reviews and Dissemination (CRD) Guidance: The CRD offers in-depth guides for undertaking systematic reviews in areas such as healthcare and public policy. The website also provides tools and checklists for researchers.

Digitally accessible via UHasselt

• Purssell (2020). How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners, and Students;
• Dekkers et al. (2022) Making Literature Reviews Work: A Multidisciplinary Guide to Systematic Approaches.
• Glasziou et al. (2001). Systematic Reviews in Health Care: A Practical Guide.

Available in our physical collection

• Boland et al. (2017). Doing a Systematic Review: A Student's Guide | Campus Diepenbeek 001 BOLA 2017;
• Booth et al. (2012). Systematic Approaches to a Successful Literature Review | Campus Diepenbeek 001 BOOT 2012;
• Booth et al. (2016). Systematic Approaches to a Successful Literature Review | Campus Diepenbeek 001 BOOT 2016;
• Hart (1998). Doing a literature review: releasing the social science research imagination | Campus Diepenbeek 001 HART 1998.
• Ridley (1998). The Literature Review: A Step-by-Step Guide for Students | Campus Diepenbeek 001 RIDL 2012.

Checklists and flowcharts are valuable tools for guiding the systematic review process, ensuring that each stage is followed rigorously. Here are some commonly used checklists and flowcharts for systematic reviews:

• PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses): PRISMA provides a 27-item checklist to ensure transparency in reporting systematic reviews and meta-analyses. The flowchart helps map out the number of records identified, screened, and included in the final analysis.
• Joanna Briggs Institute (JBI) Critical Appraisal Checklists:JBI provides specific checklists for appraising different types of studies (e.g., qualitative studies, randomized trials).
• AMSTAR 2 (A MeaSurement Tool to Assess Systematic Reviews): a tool to appraise the methodological quality of systematic reviews, particularly those involving randomized or non-randomized studies.
• CASP checklists: widely used in evidence-based practice to ensure the quality and reliability of studies included in reviews and to guide decision-making based on the strength of the evidence.
• The ROBIS checklist is a valuable tool for checking bias in systematic reviews (primarily tailored to medicine and nursing, but broadly applicable to other fields as well).
• Campbell EGM reporting standards: outlines the standards for reporting Evidence and Gap Maps, which are tools used to visually represent the evidence landscape on a specific topic.

The use of software and tools leveraging machine learning algorithms and text-mining techniques to help manage large volumes of data in order to automate aspects of systematic reviews, such as screening, data extraction, and synthesis, can significantly streamline the process and reduce time and effort.

However, automation is not without risks, such as the possibility of overlooking nuanced details that require (human) expert judgment. Moreover, it is essential to maintain transparency throughout the process, explicitly documenting in the protocol or report the use of tools, the criteria for their application, and any decisions influenced by automated methods.

Free

• Rayyan (with optional paid premium features): designed to help researchers collaborate on screening literature for systematic reviews. It enables rapid title and abstract screening, coding, and labeling of studies for inclusion or exclusion.
  Caveats: Limited to the screening process; it doesn't assist with later stages like data extraction or synthesis. The learning curve for customization may be steep for new users.
• Abstrackr (with registration): uses machine learning to assist in the abstract screening phase of systematic reviews. It learns from your inclusion and exclusion decisions to predict and suggest relevant abstracts for inclusion.
  Caveats: Limited to abstract screening; does not handle later stages of the review process such as full-text screening or data extraction. Requires manual verification of AI suggestions, as predictions may not be fully accurate.
• SRDR+ (with registration): free platform for extracting, archiving, and sharing data during systematic reviews and accessing shared data related to systematic reviews.
• RobotReviewer: uses machine learning to automatically assess the risk of bias in randomized controlled trial. Generates risk-of-bias tables and summaries.
  Caveats: Installation required. Focused only on risk-of-bias assessment; does not assist with other systematic review steps. Requires human oversight.
• SRA Helper (open-source): set of R packages designed to facilitate systematic reviews and meta-analyses in R. Automates various steps in the review process, such as importing references, conducting meta-analyses, and generating PRISMA diagrams.
  Caveats: requires knowledge of R programming.
• SR-Accelerator: set of free tools to speed up the systematic review process, including reference de-duplication, automated text extraction, and a keyword highlighting tool to aid in title and abstract screening.
• RevMan (with registration): developed by Cochrane, RevMan helps with the writing of systematic reviews and meta-analyses. It includes tools for data analysis, risk-of-bias tables, and generating forest plots. Caveats: Primarily suited for healthcare fields and may not be ideal for systematic reviews in other disciplines.
• Elicit (free basic tier): assists in literature reviews, provides relevant studies, and helps structure systematic reviews by automating part of the review process.
  Caveats: does not handle in-depth data extraction or meta-analyses. Limited database access compared to more specialized tools.
• Colandr (open-source): AI-assisted tool for conducting systematic reviews. It helps with study screening, full-text review, and data extraction. Caveats: Requires manual verification of AI-suggested inclusion/exclusion decisions. Limited to the study screening process.

Paid subscription

• Covidence: facilitates every step of the systematic review process, from importing references, screening, full-text review, data extraction, and PRISMA flow diagram generation.
• EPPI-Reviewer: supports all aspects of systematic reviews, including study management, coding, and advanced text mining using machine learning. It has features for meta-analysis and qualitative synthesis. 
  Caveats: Complex interface with a learning curve for users unfamiliar with text mining or machine learning techniques.
• DistillerSR: supports reference management, screening, data extraction, and reporting. Offers features for collaboration, PRISMA flowchart generation, and audit trails to track decision-making. Caveats: High cost. Advanced features may require training to utilize fully.
• JBI SUMARI: supports review protocols, study appraisal, data extraction, and meta-analyses. Designed for rigorous systematic reviews.

• Campbell Collaboration: a resource focused on social sciences, offering systematic reviews and protocols in education, crime and justice, and social welfare.
• PROSPERO (International Prospective Register of Systematic Reviews): register your systematic review protocol, which helps avoid duplication and increases transparency.
• Clarivate's Systematic Review Tips: practical advice for systematic reviews such as refining search strategies, managing references, using specialized databases, and ensuring comprehensive literature coverage.

Library guides

• AUB Library Guide: detailed instructions on conducting systematic reviews, with a focus on developing search strategies, managing references, and evaluating evidence. It also provides links to useful tools and databases for effective literature searches.
• Cambridge Systematic Reviews LibGuide: detailed guide to the systematic review process, from defining research questions to synthesizing and reporting results.
• Harvard Meta-Analysis Guide: in-depth overview of the meta-analysis process. It offers resources on selecting appropriate databases, developing search strategies, and using software tools for data synthesis.
• John Hopkins Systematic Reviews and Other Expert Reviews: guidance on conducting systematic reviews, including formulating research questions, designing search strategies, and managing the review process.
• UCL Library Guide for Systematic Reviews: comprehensive resources on the systematic review process, covering everything from formulating research questions to data synthesis and reporting. It includes guidance on selecting databases, search strategies, and tools to ensure rigorous and reproducible reviews.
• The University of Hawaii's Health Sciences Library guide on systematic reviews: provides comprehensive resources for conducting systematic reviews, including guidance on formulating research questions, database searching, and managing references.

Webinars

• Elsevier's Researcher Academy: comprehensive introduction to systematic reviews, covering the basics of planning, conducting, and writing systematic reviews.
• UCL Library Systematic Reviews Webinar Series: offers step-by-step guidance on key aspects of conducting systematic reviews, including developing research questions, designing search strategies, screening studies, and synthesizing evidence.