Systematic review and critical appraisal of guidance documents for extracorporeal membrane oxygenation.

Editor—Extracorporeal membrane oxygenation (ECMO) has been extensively used during both the H1N1 influenza pandemic and COVID-19 pandemic.1Barbaro R.P. MacLaren G. Boonstra P.S. et al.Extracorporeal membrane oxygenation support in COVID-19: an international cohort study of the Extracorporeal Life Support Organization registry.Lancet. 2020; 396: 1071-1078Abstract Full Text Full Text PDF PubMed Google Scholar,2Noah M.A. Peek G.J. Finney S.J. et al.Referral to an extracorporeal membrane oxygenation center and mortality among patients with severe 2009 influenza A(H1N1).JAMA. 2011; 306: 1659-1668Crossref PubMed Scopus (662) Google Scholar When the number of patients receiving ECMO support and ECMO centres sharply increased, many ECMO guidance documents were published.3Extracorporeal Life Support OrganizationECLS international summary of statistics.2022https://www.elso.org/Registry/InternationalSummaryandReports/InternationalSummary.aspxDate accessed: September 1, 2022Google Scholar High-quality guidance documents can provide explicit recommendations to healthcare professionals and improve the outcomes of patients, whereas low-quality guidance documents might mislead clinical practice or restrict the promotion of guidance documents.4Gorman E. Connolly B. Couper K. Perkins G.D. McAuley D.F. Non-invasive respiratory support strategies in COVID-19.Lancet Respir Med. 2021; 9: 553-556Abstract Full Text Full Text PDF PubMed Scopus (31) Google Scholar The rigour of the methodology, and an adequate dissemination and implementation strategy, are important components of high-quality guidance documents.5Cluzeau F. Littlejohns P. Grimshaw J.M. Appraising clinical guidelines: towards a “which” guide for purchasers.Qual Health Care. 1994; 3: 121-122Crossref PubMed Google Scholar,6Grimshaw J. Freemantle N. Wallace S. et al.Developing and implementing clinical practice guidelines.Qual Health Care. 1995; 4: 55-64Crossref PubMed Google Scholar However, the quality of published ECMO guidance documents has not been systematically reviewed, which could confuse healthcare professionals when choosing reliable guidance documents and recommendations in clinical practice. The objective of this systematic review was to use the Appraisal of Guidelines Research and Evaluation II (AGREE II) instrument to evaluate published ECMO guidance documents to help clinicians efficiently utilise the guidance documents and provide valuable information to develop and update ECMO guidance documents in the future. Five scientific databases and grey literature databases were searched from inception to December 18, 2022. We implemented the Population, clinical indication(s), and condition(s); Intervention(s); Comparator(s), comparison(s), and (key) content; Attributes of the CPG; Recommendation characteristics and “other’’ considerations (PICAR) framework to define guidance document (including guidelines, consensus, and statements) eligibility criteria.7Johnston A. Kelly S.E. Hsieh S.C. Skidmore B. Wells G.A. Systematic reviews of clinical practice guidelines: a methodological guide.J Clin Epidemiol. 2019; 108: 64-76Abstract Full Text Full Text PDF PubMed Scopus (95) Google Scholar After online training, three appraisers independently assessed the quality of guidance documents using the AGREE II instrument.8AGREE enterprise website. The AGREE II instrument. 2017https://www.agreetrust.org/Date accessed: October 25, 2021Google Scholar Double weight was assigned to domain 3 (rigour of development) and domain 5 (applicability).9Jiang M. Liao L.Y. Liu X.Q. et al.Quality assessment of clinical practice guidelines for respiratory diseases in China: a systematic appraisal.Chest. 2015; 148: 759-766Abstract Full Text Full Text PDF PubMed Scopus (23) Google Scholar A guidance document was ‘recommended’ if the overall score was >60%, whereas the scores of ‘recommended with modifications’ were between 30% and 60%. A guidance document was ‘not recommended’ if the overall score was <30%. In addition, we defined a guidance document as high quality if the scaled domain score was >70% for domain 3 and >50% for all other domains.10O'Shaughnessy S.M. Lee J.Y. Rong L.Q. et al.Quality of recent clinical practice guidelines in anaesthesia publications using the Appraisal of Guidelines for Research and Evaluation II instrument.Br J Anaesth. 2022; 128: 655-663Abstract Full Text Full Text PDF PubMed Scopus (5) Google Scholar Statistical significance among groups was determined by analysis of variance (anova) tests whenever data met normal distribution and equal variance assumptions. Otherwise, Mann–Whitney U-tests were applied. Full details on methods are provided in Supplementary files 2 and 3. We identified 81 guidance documents, of which 37 were updated versions. The AGREE II score of included guidance documents ranged from 17.5% to 77.3%, with a median of 42.6%. The domain with the highest score was domain 4 (‘clarity of presentation’; median 66.7 [inter-quartile range, IQR: 57.4–79.6]), followed by domain 1 (‘scope and purpose’; median 61.1 [IQR: 53.7–75.9]). The scores of the other four domains, including domain 6 (‘editorial independence’; median 50.0 [IQR: 33.3–77.8]), domain 2 (‘stakeholder involvement’; median 44.4 [IQR: 33.3–63.0]), domain 3 (median 33.3 [IQR: 20.8–53.5]), and domain 5 (median 30.6 [IQR: 25.0–41.7]), were <60%. (Fig 1). Fourteen guidance documents were recommended for use without modification, but only four met the criteria for high quality, suggesting a discrepancy between overall rating and high-quality status. Only one specialised ECMO guidance document was recommended for use without modification, but it was not high quality. Subgroup analysis showed that AGREE II scores of initial versions of guidance documents were significantly lower than those of the updated versions in domains 1 (59.3 vs 68.9), 2 (42.2 vs 52.1), 3 (31.0 vs 45.5), and 4 (63.5 vs 71.5). The guidance documents with funding had higher scores in domains 1 (70.3 vs 60.7), 2 (58.2 vs 41.6), 3 (48.5 vs 32.7), and 6 (69.0 vs 42.7) compared with those that were unfunded. Moreover, specialised ECMO guidance documents had lower scores in domains 1 (58.8 vs 65.6), 3 (27.0 vs 41.8), 4 (61.0 vs 69.5), and 5 (28.9 vs 35.0) compared with non-specialised ECMO guidance documents. In the subgroup analysis, domains 1 and 4 were still the top two domains in scoring, the results of which were similar to those of the primary analysis. The ICC was 0.81 (0.75–0.85), suggesting good consistency among the three appraisers. Detailed results are shown in Supplementary Figures S1 and S2 and Supplementary Tables S2–S6. ‘Rigour of development’, considered a pivotal domain, describes the methods used to synthesise evidence and formulate recommendations. More than half of the included guidance documents (64.2%) did not describe systematic literature search or selection methods, suggesting that the literature search strategy used in the included guidance documents was often unclear. Grading systems (e.g. GRADE [Grading of Recommendations Assessment, Development and Evaluation]), were developed to reduce inconsistency between the quality of evidence and the strength of recommendations during the process of guidance document development.11Guyatt G. Oxman A.D. Akl E.A. et al.GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables.J Clin Epidemiol. 2011; 64: 383-394Abstract Full Text Full Text PDF PubMed Scopus (5609) Google Scholar However, grading systems were only used in 31 out of 81 guidance documents. Subsequently, the unclear search strategy of the literature and the unclear link between the evidence and recommendations in most of the included guidance documents led to poor scores in rigour. Consistent with previous evaluations of guidance documents in other clinical areas, domain 5 scored the lowest (33%).12Molino C. Leite-Santos N.C. Gabriel F.C. et al.Factors associated with high-quality guidelines for the pharmacologic management of chronic diseases in primary care: a systematic review.JAMA Intern Med. 2019; 179: 553-560Crossref PubMed Scopus (41) Google Scholar,13Silberberg B. Aston S. Boztepe S. Jacob S. Rylance J. Recommendations for fluid management of adults with sepsis in sub-Saharan Africa: a systematic review of guidelines.Crit Care. 2020; 24: 286Crossref PubMed Scopus (8) Google Scholar These findings can be explained by the tendency for guideline development groups to focus more on methods compared with how their recommendations can be implemented and monitored, which can lead to limited compliance with their recommendations.14Lugtenberg M. Zegers-van Schaick J.M. Westert G.P. Burgers J.S. Why don’t physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners.Implement Sci. 2009; 4: 54Crossref PubMed Scopus (324) Google Scholar The scores of domains 2 and 6 were also low (<60%). Therefore, ECMO guidance document panels should explicitly disclose the information on conflicts of interest to ensure that readers can consider financial or academic conflicts of interest as a part of their assessment of the credibility of guidance documents. Moreover, attention must be paid to the views of the target population and engaging stakeholders (e.g. patients) in guidance documents development. To our knowledge, this is the first report to use the AGREE II instrument to evaluate systematically the methodologic quality of published ECMO guidance documents. We found that the methodologic quality of ECMO guidance documents evaluated with the AGREE II instrument was generally poor. Specifically, most guidance documents are deficient in domains 2, 3, 5, and 6. Hence, we suggest that ECMO guidance documents should be written and developed using the AGREE II instrument and GRADE. Furthermore, additional materials, such as summary documents, algorithms, and checklists, should be included to aid implementation. Study conception/design: MJ, HL. Data acquisition: HL, WY, ZL. Data analysis and interpretation: HL, WY, YX. Manuscript writing: all authors. Revision of the manuscript: all authors. Approved the final version of the manuscript for submission and have agreed to be accountable for all aspects of this work: all authors. MJ is guarantor. The authors declare that they have no competing interests. Emergency Key Program of Guangzhou Laboratory (EKPG21-29, EKPG21-31). The funding had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.