Should mindfulness-based cognitive therapy be used for psychosis? A systematic review of the literature and meta-analysis

Should mindfulness-based cognitive therapy be used for psychosis? A systematic review of the literature and meta-analysis

Carlo Lazzari1, Yasuhiro Kotera2, Marco Rabottini1

1International Centre for Healthcare and Medical Education, Bristol, United Kingdom; 2University of Nottingham, United Kingdom.

Summary. Background. In England, psychosis incidence is 31.7×100,000 persons per year. Mindfulness-based interventions for psychosis (MBIp) might reduce its symptoms; however, the research outcomes on its effect size (ES) vary considerably. This project aims to ascertain the existing evidence. Methods. Eight publications from a pool of over 260 studies were extracted and analysed at meta-analysis for ES as satisfying the inclusion criteria. Results. MBIp has a moderate ES (r=0.34; p<.001) on psychosis with a 95% confidence interval (CI) of 0.26-0.42 (small to high). Discussion. MBIp improves psychosis symptoms. However, the studies analysed show heterogeneity in ES. Hence only conditional recommendations can be made for MBIp.

Key words. Meta-analysis, mindfulness, psychosis, systematic review, therapy.

La terapia cognitiva basata sulla mindfulness dovrebbe essere usata per la psicosi? Una revisione sistematica della letteratura e meta-analisi.

Riassunto. Background. In Inghilterra, l’incidenza di psicosi è di 31,7×100.000 persone all’anno. Gli interventi basati sulla terapia cognitiva fondata sulla mindfulness per la psicosi (MBIp) potrebbero ridurre i suoi sintomi; tuttavia, i risultati della ricerca sulla dimensione del suo effetto (ES) variano considerevolmente. Questo progetto mira ad accertare le prove esistenti. Metodi. Da un pool di oltre 260 studi solo otto pubblicazioni soddisfacenti i requisiti di inclusione sono state estratte e analizzate in meta-analisi per ES. Risultati. MBIp ha un ES moderato (r=0,34; p<0,001) sulla psicosi con un intervallo di confidenza del 95% (CI) di 0,26-0,42 (da piccolo ad alto). Discussione. MBIp migliora i sintomi della psicosi. Tuttavia, gli studi analizzati mostrano eterogeneità in ES. Quindi solo raccomandazioni condizionali possono essere fatte per la MBIp.

Parole chiave. Meta-analisi, mindfulness, psicosi, revisione sistematica, terapia.

Introduction

Context

In England, psychosis incidence is 31.7×100,000 persons per year and 54% of them receive combined therapy consisting of psychotropic medication and psychological interventions, including cognitive behavioural therapy (CBT) and mindfulness-based interventions for psychosis (MBIp), which appear to be effective in reducing severe and lasting psychotic symptoms and are endorsed by the National Institute for Health and Care Excellence (NICE)1-5. A previous meta-analysis, a multicentric study with 101 patients with psychosis (PWP) and a randomised controlled trial (RCT) with 138 PWP, found a moderate effect size (ES) of MBIp in pre-post studies and a small-to-moderate effect when compared with treatment as usual (TAU)5-7. The authors work for the mental health services in England, where it is felt that MBIp could help PWP deal with distressing delusions and hallucinations.

Definition of terms

Mindfulness, originating from Buddhist meditation, is a practice that leads to awareness of the essence of the mind in an insightful and non-judgemental experience while dealing with reality as it is, empowering self-awareness, reducing anxiety and depression, decreasing rumination and obsessive worrying about things and lowering the evasion, repression and denial of ideas, perceptions and memories2,8-13.

MBIp is psychotherapy usually conducted in groups in meetings of 40 minutes, with no silences and with instructions every 30-60 seconds to help PWP disengage from distressing hallucinations and delusions without the need to examine and challenge them14. MBIp can be delivered in the following fashions:

• Mindfulness-based-Stress-Reduction (MBSR) is an eight-week group therapy improving self-efficacy and acceptance during losses and limitations15.

• Mindfulness-based-Cognitive-Therapy (MBCT) is an eight-week group therapy teaching mindfulness skills16.

• Mindfulness-based-Psychoeducation-Programme (MBPP) helps PWP, especially those suffering from hallucinations, improve mindfulness and reduce rumination, stress and disruption17.

• Acceptance-and-Commitment-Therapy (ACT) helps PWP contact the present moment and their values and not take their thoughts literally18.

• Individual-Mindfulness-Therapy-for-Voices (iMTV) helps PWP who suffer from auditory hallucinations19.

• Mindfulness-Based-Crisis-Intervention (MBCI) reinforces PWP’s mindfulness during crises and goals setting20.

According to DSM-5, psychosis is a pathology of mind, such as schizophrenia, schizoaffective disorder or drug-induced psychosis, presenting one or more of the following symptoms for one month or more21-23:

• delusions or positive symptoms, as fixed and false beliefs held in the face of evidence to the contrary;

• perceptual hallucinations (auditory, visual, olfactory, tactile, taste and smell), as perceptions without an object;

• passivity experiences, as thoughts believed not to be originating from the self;

• thought (speech) disturbances, as disorganised and tangential thoughts with loose associations and neologisms or derailment;

• disorder of expression of emotions or negative symptoms with depression, flat affect, apathy, anhedonia, scarcity of speech and blunting;

• disorganised behaviour, including unusual, eccentric, aimless and restless activity or catatonia;

• psychomotor disturbances, including mutism, arousal, stupor, negativism, stiff posturing or flexibility.

Aim of the study

Previous research on psychotherapy overestimates its efficacy, while MBIp studies have some methodological issues and are based on few randomised controlled trials (RCT)6,14,24. The British Association for Counselling and Psychotherapy recommends using evidence-based practice (EBP) for intervention evaluation and new psychotherapy policies25. Therefore, the current systematic review (SR) will help endorse or reject the efficacy of MBIp and promote EBP by using an approach placed at the vertex of the evidence pyramid (SR, meta-analysis; figure 1) and apply statistically-based, bias-free, valid and verifiable methods to suggest conclusions and avoid procedures of little or uncertain efficacy for patients5,6,25-31.

Methods

Literature search

The Boolean search terms for secondary data were ‘mindfulness’, ‘psychosis’, ‘patient*’, ‘inpatient*’, ‘psychiatry’, ‘outcome*’, ‘randomised controlled trial/RCT’ and ‘psychotherapy’ (table 1).

The search engines queried were PubMed, Google Scholar, Medline, Embase, PsycInfo, CINAHL and Cochrane Central Register of Controlled Trials. The inclusion criteria were quantitative, qualitative and mixed-method peer-reviewed research on outcomes, RCT, studies no older than five years old, studies with statistical analysis, MBIp and English articles or abstracts (table 2a).

The exclusion criteria were other systematic reviews, meta-analyses of other studies, studies with no outcomes or statistics, pilot studies, studies older than five years old and ACT (table 2b).

The PRISMA32 flowchart summarises the search stages for the 260 studies found (figure 1). The research retrieved respected the higher levels of EBP (figure 1) with five RCT17,20,33-35, one controlled trial without randomisation19, two cohort studies36,37 and one qualitative and rigorous group study11.

Outcome measures

The outcome measures in the studies were as follows:

• Southampton-Mindfulness-Questionnaire (SMQ): a 16-item-self-report scale measuring the degree of the subject’s mindfulness during upsetting thoughts and hallucinations38;

• Positive-and-Negative-Symptoms-of-Schizophrenia (PANSS): a 30-item questionnaire assessing positive and negative symptoms of schizophrenia39;

• Psychotic-Symptom-Rating-Scale (PSYRATS): a 17-item scale assessing hallucinations and delusions over a range of ‘absent’ to ‘marked’40.

Methods of data summarisation

The PICO(s) (Population, Intervention, Comparison, Outcome and Study Design) summarised the articles41. An SR analyses the relevant research thoroughly to address a specific question and determine the validity of each study considered when stating conclusions42. Here, the intervention (I) is the independent variable – MBIp – while the outcome (O) represents the dependent variable – the effect of MBIp43. RCT are experimental studies that answer therapy questions and compare patients receiving the intervention or experimental group (e.g., MBIp) with a control group that does not receive the intervention or receives TAU or has no psychosis; through randomisation, the experimenter assigns participants on a random basis to a control or experimental group; outcomes are usually assessed at baseline/pre-test and post-test30. The author extracted the ES from each study (table 2a).

Statistical analysis

Effect Size (ES). Pearson’s r transformed the outcomes of the individual studies into ES measures to estimate the magnitude of the intervention (MBIp) on the outcomes30. ES is expressed as an interval scale from 0 to 1, with 0.20 or less representing a small effect, 0.24 to 0.33 an intermediate effect, 0.37 to 0.45 a large effect, and >0.80 a very large effect48,49. Online Campbell50 and Lenhard and Lenhard48 software calculated the ES.

A random-effect meta-analysis (M-A) performed by Medcalc Statistical Software (www.medcalc.org) merged the global ES from the individual studies, extracted the heterogeneity factor I2, and the risks of publication bias Egger’s and Begg’s test42,43,51,52.

Research hypothesis and alpha error. The Level I or alpha a error defines the statistical probability of committing a mistake by rejecting the null hypothesis Ho - no relationship between the dependent, independent and other variables - when it is instead true. It is here set at p=.05, indicating that the authors were willing to accept no more than a 5% chance of committing that error49,53.

The Kolmogorov-Smirnov (K-S) test verified the normality distribution of the ES of all studies to justify a parametric test and meta-analysis43,54.

Split-half Pearson’s r calculated internal reliability on the extracted ES55.

Research hypotheses

Hypothesis 1 Ho1: MBIp has no effect on PWP.

Hypothesis 2 Ho2: there is no difference between studies in terms of ES.

Assessment of risk of biases (RoB)

Cochrane GRADE software assessed the RoB for the RCT, while ROB-2 software assessed the RoB for non-RCT56,59.

Results

Major findings

The K-S test (D=.24; p=.63) reported a normality distribution for all ES, hence justifying the meta-analysis, while the split-half indicated significant reliability in the findings (r=0.65).

The authors rejected Ho1, as MBIp shows an intermediate effect on PWP when assessed at immediate follow-up and for a median period of two months after its discontinuation (r=0.34; 95%CI=0.26-0.42; p<.001). The authors also rejected Ho2 due to the significant heterogeneity of ES between studies (I2=68.94%; p=.002) (figure 2).

Characteristics of selected population

The global population comprised 170 PWP and 165 PSP.

Type of mindfulness intervention

MBIp included MBPP17,33, iMTV19, MBCI20, individual MBCT11, and group MBCT34-37.

Quantitative outcomes

MBIp produced the following ES on PWP’s psychotic symptoms:

• large effect on the absence of aversion and non-judgment at 2-month follow-up (r=0.39)19;

• large effect in reduction of positive and negative symptoms at 6-month follow-up (r=0.70-0.80)35,36;

• intermediate effect in observing and acting with awareness at 24-month follow-up (r=0.24)35;

• intermediate effect on subjective recovery rate at immediate follow-up (r=0.31)37;

• intermediate effect on reduced rumination at 3-month follow-up (r=0.31)17;

• small effect in reducing distress and disturbance from auditory hallucinations at 10-month follow-up (r=0.10)34;

• small effect on the readmission rate at 12-month follow-up (r=0.19)20.

Qualitative outcomes

CASP Checklist60 evaluated Böge et al.11. After a 4-week MBIp intervention, a thematic analysis extracted the effects of MBIp on 27 PWP with three major themes, such as emotions - higher acceptance of psychotic symptoms -, cognition - increased self-awareness and empowerment -, and symptoms changes - reduction of anxiety, depression, and PANSS11. However, the authors did not explain MBIp to participants, although they triangulated their impressions for the conclusions; research design, recruitment strategies, data collection were clearly stated11.

Follow-up assessment

The median follow-up period was two months, while the mode was zero months, mostly occurring in post-treatment immediately after completing the MBIp in all studies.

RoB assessment

RoB in RCTs was assessed by GRADE in five RCTs17,20,33-35, yielding moderate certainty in the results; the authors are convinced that the study’s ES is close to the actual effect61 (figure 3).

The certainty of the evidence for the SR relates to how confident the author is that effect size represents the actual effect, which is usually high for RCTs62. The results of the GRADE subcategories for RoB were as follows57,58,62:

• no RoB in design and implementation, as the RCTs had allocation concealment, blindness and follow-up, although studies mainly were single-blinded (therapist aware); exclusion criteria were reported;

• RoB in inconsistency and heterogeneity in ES and I2;

• no RoB in indirectness, as PICO was implemented;

• no RoB in imprecision, as MBIp had a moderate effect, while an adequate sample of patients was allocated to the experimental-intervention and control groups;

• RoB in publication bias as seen in the funnel plot (figure 4), although publication biases tested in M-A were not statistically significant (Egger’s test: p=.99; Begg’s test: p=.25).

ROB-256 assessed RoB for four non-RCTs11,19,36,37. The tool yielded an 80% low risk in supporting MBIp and a 20% of some concern linked to the absence of randomisation in these studies. No RoB was found in studies’ deviation from intents, missing outcome data or measurement, or selection results (figure 5).

Research objectivity

The current study improved its objectivity by using recent research with measurable outcomes and statistical analysis that was bias-free, predictable, transparent and verifiable29,63.

External validity

Cochrane AMSTAR-2 yielded a low quality for external validity for the current review because it ignored the RoB in the individual studies63,64. The points respected were: comprehensive literature search, list and description of included and excluded studies, PICO method, RoB assessment, RoB valuation in the meta-analysis (funnel plot; Egger’s and Begg’s test), meta-analysis and methods explaining studies’ heterogeneity and publication biases (GRADE and ROB-2)64.

Discussion and conclusions

The current review reports a moderate effect size of MBIp on psychosis with moderate certainty in the results. There is also an 80% low risk in supporting MBIp, although the current study has a low quality for external validity. In PWP’s psychotic symptoms, MBIp produced observing and acting with awareness, absence of aversion and non-judgment, reduction of positive and negative symptoms, increased subjective recovery rate, reduced distress and disturbance from auditory hallucinations. Qualitatively, MBIp increased acceptance of psychotic symptoms, self-awareness and empowerment with reduced anxiety and depression.

Limitations emerge from threats to external and population validity. The findings refer to PWP in the adult population and cannot be generalised to other PWP cohorts, while assessment scales and MBIp interventions differed for all studies65,66. However, a clinical validity can be claimed for the results regarding MBIp43.

The GRADEpro67 scale endorses a conditional recommendation for the suggestion of MBIp, mainly linked to the threats to validity and the risk of biases in outcome-based studies. Nonetheless, GRADEpro extracted a large desirable effect of MBIp, which has trivial undesirable effects, low costs of implementation, high impact on health equity, amounting to the acceptability of the intervention’s feasibility for major stakeholders67.

Future research shall standardise and control MBIp procedures and outcome measures, specify the required length and amount of MBIp needed, and identify required follow-up periods to stabilise PWP’s presentation14.

Conflict of interests: the authors have no conflict of interests to declare.

References

1. Kirkbride J, Errazuriz A, Croudace T, et al. Incidence of schizophrenia and other psychoses in England, 1950-2009: a systematic review and meta-analyses. PLoS ONE 2012; 7: e31660.

2. NICE (National Institute for Health and Care Excellence) (2014). CG178: Psychosis and schizophrenia in adults: treatment and management. Available at: https://bit.ly/3F3HGz5. [last accessed: 29 April 2022].

3. NICE (National Institute for Health and Care Excellence) (2020) Psychosis and schizophrenia: What is it? Available at: https://bit.ly/3kpm4Up [last accessed: 29 April 2022].

4. McManus S, Bebbington P, Jenkins R, Brugha T (eds). Mental health and wellbeing in England: adult psychiatric morbidity survey 2014. Leeds: NHS Digital. [Online] 2016. Available at: https://bit.ly/3MDDTer [last accessed: 29 April 2022].

5. Morrison A, Law H, Carter L, et al. Antipsychotic drugs versus cognitive behavioural therapy versus a combination of both in people with psychosis: a randomised controlled pilot and feasibility study. Lancet Psychiatry 2018; 5: 411-23.

6. Lewis S, Tarrier N, Drake R. Integrating non-drug treatments in early schizophrenia. Br J Psychiatry 2005; 187 (S48): s65-s71.

7. Aust J, Bradshaw T. Mindfulness interventions for psychosis: a systematic review of the literature. J Psychiatr Ment Health Nurs 2016; 24: 69-83.

8. Kabat-Zinn J. Mindfulness-based interventions in context: past, present, and future. Clinical Psychology: Science and Practice 2003; 10: 144-56.

9. Baer RA (ed). Mindfulness-based treatment approaches. London: Elsevier and Academic Press, 2006.

10. Hayes S, Hofmann S. The third wave of cognitive behavioral therapy and the rise of process-based care. World Psychiatry 2017; 16: 245-6.

11. Böge K, Karadza A, Fuchs L, et al. Mindfulness-based interventions for in-patients with schizophrenia spectrum disorders: a qualitative approach. Front Psychiatry 2020; 11: 600.

12. Teasdale J, Williams M, Segal Z. The Mindful Way Workbook. London, New York: The Guilford Press, 2014.

13. Ngô T. Les thérapies basées sur l’acceptation et la pleine conscience. Sante Ment Que 2014; 38: 35-63.

14. Chadwick P. Mindfulness for psychosis. Br J Psychiatry 2014; 204: 333-4.

15. Speca M, Carlson LE. Mackenzie MJ, Angen M. Mindfulness-based stress reduction (MBSR) as an intervention for cancer patients. In: Baer RA (ed). Mindfulness-based Treatment Approaches. London: Elsevier and Academic Press, 2006.

16. Coffman SJ, Dimidjan S, Baer RA. Mindfulness-based cognitive therapy for prevention of depressive relapse. In: Baer R (ed). Mindfulness-based treatment approaches. London: Academic Press, 2006.

17. Lam A, Leung S, Lin J, Chien W. The Effectiveness of a mindfulness-based psychoeducation programme for emotional regulation in individuals with schizophrenia spectrum disorders: a pilot randomised controlled trial. Neuropsychiatr Dis Treat 2020; 6: 729-47.

18. Bach PA, Gaudiano B, Pankey J, Herbert JD, Hayes SC. Acceptance, mindfulness, values and psychosis: Applying acceptance and commitment therapy (ACT) to the chronically mentally ill. In: Baer RA (ed). Mindfulness-based treatment approaches. London: Elsevier and Academic Press, 2006.

19. Louise S, Rossell S, Thomas N. The acceptability, feasibility and potential outcomes of an individual mindfulness-based intervention for hearing voices. Behav Cogn Psychother 2018; 47: 200-16.

20. Jacobsen P, Peters E, Robinson E, Chadwick P. Mindfulness-based crisis interventions (MBCI) for psychosis within acute inpatient psychiatric settings; a feasibility randomised controlled trial. BMC Psychiatry 2020; 20: 193.

21. Gaebel W, Zielasek J. Schizophrenia in 2020: trends in diagnosis and therapy. Psychiatry Clin Neurosci 2015; 69: 661-73.

22. Casey P, Kell B. Fish’s clinical psychopathology. 3rd edition. London: Gaskell, 2007.

23. American Psychiatric Association. DSM-5: Diagnostic and Statistical Manual of Mental Disorders. 5th edition. Washington: American Psychiatric Association Publishing, 2013.

24. Shonin E, Van Gordon W, Griffiths M. Do mindfulness-based therapies have a role in the treatment of psychosis? Aust N Z J Psychiatry 2013; 48: 124-7.

25. BACP (British Association of Counselling and Psychotherapy). The importance of evidence. [Online] 2018. Available at: https://bit.ly/3EXIxkY [last accessed: 29 April 2022].

26. Kendell R. Clinical validity. Psychol Med 1989; 19: 45-55.

27. Cook S, Schwartz A, Kaslow N. Evidence-based psychotherapy: advantages and challenges. Neurotherapeutics 2017; 14: 537-45.

28. Thomason T. The trend toward evidence-based practice and the future of psychotherapy. Am J Psychother 2010; 64: 29-38.

29. Cohen L, Manion L, Morrison K. Research methods in education. 8th edition. London and New York: Routledge, 2918.

30. Polit DF, Tatano Beck C. Essential of nursing research: appraising evidence for nursing practice. 10th edition. London: Lippincott Williams & Wilkins, 2022.

31. Lee C, Hunsley J. Evidence-based practice: separating science from pseudoscience. Can J Psychiatry 2015; 60: 534-40.

32. Moher D, Liberati A, Tetzlaff J, Altman D. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. PLoS Medicine 2009; 6: e1000097.

33. Chien W, Chow K, Chong Y, Bressington D, Choi K, Chan C. The role of five facets of mindfulness in a mindfulness-based psychoeducation intervention for people with recent-onset psychosis on mental and psychosocial health outcomes. Front Psychiatry 2020; 11: 177.

34. Chadwick P, Strauss C, Jones A, et al. Group mindfulness-based intervention for distressing voices: A pragmatic randomised controlled trial. Schizophr Res 2016; 175: 168-73.

35. Wang L, Chien W, Yip L, Karatzias T. A randomised controlled trial of a mindfulness-based intervention program for people with schizophrenia: 6-month follow-up. Neuropsychiatr Dis Treat 2016; 12: 3097-110.

36. Tong A, Lin J, Cheung V, et al. A low-intensity mindfulness-based intervention for mood symptoms in people with early psychosis: development and pilot evaluation. Clin Psychol Psychother 2015; 23: 550-60.

37. Jones A, Strauss C, Hayward M. A service evaluation of a group mindfulness-based intervention for distressing voices: how do findings from a randomised controlled trial compare with routine clinical practice? Behav Cogn Psychother 2020; 49: 76-90.

38. Chadwick P, Hember M, Symes J, Peters E, Kuipers E, Dagnan D. Responding mindfully to unpleasant thoughts and images: reliability and validity of the Southampton mindfulness questionnaire (SMQ). Br J Clin Psychol 2008; 47: 451-5.

39. Kay S, Fiszbein A, Opler L. The Positive and Negative Syndrome Scale (PANSS) for schizophrenia. Schizophr Bull 1987; 13: 261-76.

40. Woodward T, Jung K, Hwang H, et al. Symptom dimensions of the psychotic symptom rating scales in psychosis: a multisite study. Schizophr Bull 2014; 40(Suppl_4): S265-S274.

41. Coughlan M, Cronin P, Ryan F. Doing a literature review in nursing, Health and Social Care. London: SAGE, 2013.

42. Petticrew M, Roberts H. Systematic reviews in the social sciences: a practical guide. Oxford: Blackwell, 2006.

43. Bettany-Saltikov J, Whittaker V. Selecting the most appropriate inferential statistical test for your quantitative research study. J Clin Nurs 2013; 23: 1520-31.

44. Gaudiano B, Herbert J. Acute treatment of inpatients with psychotic symptoms using Acceptance and Commitment Therapy: pilot results. Behav Res Ther 2006; 44: 415-37.

45. Bach P, Hayes SC. The use of acceptance and commitment therapy to prevent the rehospitalisation of psychotic patients: a randomised controlled trial. J Consult Clin Psychol 2002; 70: 1129-39.

46. Potes A, Souza G, Nikolitch K, et al. Mindfulness in severe and persistent mental illness: a systematic review. Int J Psychiatry Clin Pract 2018; 22: 253-61.

47. Langer ÁI, Schmidt C, Mayol R, et al. The effect of a mindfulness-based intervention in cognitive functions and psychological well-being applied as an early intervention in schizophrenia and high-risk mental state in a Chilean sample: study protocol for a randomised controlled trial. Trials 2017; 18: 233.

48. Lenhard W, Lenhard A. Calculation of Effect Sizes. [Online] 2016. Available at: https://bit.ly/3LATjzX [last accessed: 29 April 2022].

49. Bowling A. Research methods in health. 3rd Edition. Maidenhead and New York: Open University Press, 2009.

50. Wilson DB. Practical meta-analysis effect size calculator [Online calculator]. Available at: https://bit.ly/3LCpFul [last accessed: 29 April 2022].

51. Higgins JPT, Green S (eds). Cochrane Handbook for Systematic Reviews of Interventions. Chichester: Wiley-Blackwell, 2008.

52. Lin L, Chu H. Quantifying publication bias in meta-analysis. Biometrics 2018; 74: 785-94.

53. McLeod SA. What are type I and type II errors? Simply Psychology [Webpage] 2019. Available at: https://bit.ly/3vzS9PI [last accessed: 29 April 2022].

54. Stangroom D. The Kolmogorov-Smirnov Test of Normality. [Online] 2021. Available at: https://bit.ly/375eDi4 [last accessed: 29 April 2022].

55. Dempster M, Hanna D. Research Methods in Psychology. Chichester: John Wiley & Sons, 2016.

56. Sterne JAC, Hernán MA, McAleenan A, Reeves BC, Higgins J PT. Assessing risk of bias in a non-randomised study. In: Higgins JPT, Thomas J, Chandler J, et al. (eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020) [Online]. Available at: https://bit.ly/3FaXpwy [last accessed: 29 April 2022].

57. Schünemann H, Broz˙ek J, Guyatt G, Oxman A (eds.). GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group. [Online] Available at: https://bit.ly/3kuWa1o [last accessed: 29 April 2022].

58. Guyatt G, Oxman A, Akl E, Kunz R, Vist G, Brozek J et al. GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011; 64: 383-94.

59. Higgins JPT, Savovic´ J, Page MJ, Elbers RG, Sterne JAC. Assessing risk of bias in a randomised trial. In: Higgins JPT, Thomas J, Chandler J, et al. (eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020) [Online]. Available at: https://bit.ly/3FaXpwy [last accessed: 29 April 2022].

60. CASP (Critical Appraisal Skills Programme). CASP: Qualitative Checklist. [Online] 2018. Available at: https://bit.ly/3vBJE6V [last accessed: 29 April 2022].

61. Goldet G, Howick J. Understanding GRADE: an introduction. J Evid Based Med 2013; 6: 50-4.

62. Granholm A, Alhazzani W, Møller M. Use of the GRADE approach in systematic reviews and guidelines. Br J Anaesth 2019; 123: 554-9.

63. Schünemann HJ, Oxman AA, Vist GE, et al. on behalf of the Chochrane Application and Recommendations Methods Group. Interpreting results and drawing conclusions. In: Higgins JPT, Green S (eds). Chochrane Handbook for Systematic Reviews of Interventions. Chichester: Wiley-Blackwell, 2018.

64. Shea B, Reeves B, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017; 358: j4008.

65. Bryman A. Social Research Methods. 5th edition. Oxford: Oxford University Press, 2016.

66. Bandhari P. Types of external validity [Webpage] 2021. Available at: https://bit.ly/3Face2s [last accessed: 29 April 2022].

67. McMaster University. GRADEpro GDT: GRADEpro Guideline Development Tool [Software] 2020. (Developed by Evidence Prime, Inc.). Available at: https://bit.ly/3OQLxnS [last accessed: 29 April 2022].