Effectiveness of Original vs. AI-Generated Plain Language Summaries of Systematic Reviews

Official title

Patients' Perceptions and Comprehension of Original Versus AI-Generated Plain Language Summaries of Cochrane Oncology Systematic Reviews

Overview

The study explores how patients with non-metastatic breast cancer perceive and comprehend original Plain Language Summary and AI-generated Plain Language Summary (PLS) of two systematic reviews from the Cochrane Library. The systematic reviews address exercise during adjuvant therapy and psychological interventions for non-metastatic breast cancer. The study, which is randomized and double-blind, aims to assess whether AI-generated PLS and original PLS differ in their influence on patients' intention to undertake and apply the recommendation presented in the summary, as well as their overall understanding of the study results.

Detailed description

Studies show that Plain Language Summaries (PLS), which are intended to improve accessibility to health information for laypeople, are lacking in terms of comprehension, readability, and user engagement. Further empirical research is needed to evaluate the comparative effectiveness of AI-generated summaries versus original Plain Language Summaries. This study, conducted at the Department of Oncology and Radiotherapy at the University Hospital of Split aims to explore whether large language models (LLMs), such as ChatGPT, can be leveraged to provide summaries of complex medical content, as those found in systematic reviews, to be more streamlined and adapted for laypeople. The prospective, randomized, single-center, parallel group, double-blind, controlled trial focuses specifically on patients diagnosed with non-metastatic breast cancer, histologically confirmed, and aims to help them make more evidence-based medical decisions regarding their health and recovery. The study's primary objective is to analyze patients' level of intention for changing their behavior and start applying the interventions described in the summary. The level of intention to change behavior will be measured using a visual analog scale ranging from 0 to 100 (0 - do not agree at all and 100 - fully agree). The secondary outcomes are patients' level of understanding calculated by the score received on a five-question knowledge survey (5 as a maximum score), the level of trust in (range 0-100) and clarity of (range 0-100) the information presented in the PLS, and their confidence in their decision to change behavior (range 0-100). The survey is based on the information presented in the PLS and the questions focus on patients' understanding of the benefits and risks of the intervention as well as the quality of evidence described in the systematic reviews. Recruitment for this study will take place over 6 months. Patients will be invited to participate in the study before or after their routine oncology appointment. The participants will read two summaries from two respective Cochrane reviews: exercise during adjuvant therapy and psychological interventions for non-metastatic breast cancer. They will then take the brief survey that tests the impact and likelihood of applying the recommendations described in the summary and their overall understanding of the medical information discussed in the reviews. The study includes female adult oncology patients diagnosed with non-metastatic breast cancer, histologically confirmed, following a modality of the treatment, which includes one or more of the following: neoadjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy); adjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy); adjuvant radiotherapy (including local and locoregional radiation treatment, with possible radiotherapy boost where indicated). Since participants in this study are limited to reading material and completing a survey, the study is non-invasive in nature and poses low-risk. Data will be analyzed using descriptive statistics to summarize participant characteristics and assess baseline comparability between the two study arms.

Interventions

• Behavioral AI-generated PLS
  Reading AI-generated Plain Language Summaries of two Cochrane systematic reviews covering the same two evidence-based topics: exercise during adjuvant therapy and psychological interventions for non-metastatic breast cancer.
• Behavioral Original PLS
  Reading original Plain Language Summaries of two Cochrane systematic reviews covering the same two evidence-based topics: exercise during adjuvant therapy and psychological interventions for non-metastatic breast cancer.

Primary outcome measures

• Patients' level of intention for changing their behavior [Time frame: Immediately after reading individual PLS]

Secondary outcome measures (4)

• Patients' comprehension of the information presented in the summary [Time frame: Immediately after reading individual PLS]
• Patients' level of trust in the information presented [Time frame: Immediately after reading individual PLS]
• Patients' level of perceived clarity in the information [Time frame: Immediately after reading individual PLS]
• Patient's level of confidence in their decision to change behavior [Time frame: Immediately after reading individual PLS]

Eligibility criteria

Inclusion criteria

• female patients with histologically confirmed non-metastatic breast cancer currently undergoing chemotherapy or radiotherapy:

• neoadjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy),
• adjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy),
• adjuvant radiotherapy (including local and locoregional radiation treatment with possible radiotherapy boost where indicated).
• patients need to be able to read and understand Croatian.

Exclusion criteria

• female patients with metastatic breast cancer.
• female patients receiving only hormonal treatment.

Criteria are shown verbatim from the registry (in English). Final eligibility is always assessed by the study center.

Healthy volunteers: No

Study design

Allocation

Randomized

Model

Parallel assignment

Masking

Triple blind

Primary purpose

Supportive care

Study locations

Croatia · 1 center

• Department of Oncology and Radiotherapy, University Hospital Split — Split

Publications

• Hasannejadasl H, Roumen C, Smit Y, Dekker A, Fijten R. Health Literacy and eHealth: Challenges and Strategies. JCO Clin Cancer Inform. 2022 Sep;6:e2200005. doi: 10.1200/CCI.22.00005. PMID 36194843
• Seiler A, Jenewein J. Resilience in Cancer Patients. Front Psychiatry. 2019 Apr 5;10:208. doi: 10.3389/fpsyt.2019.00208. eCollection 2019. PMID 31024362
• Goerling U, Faller H, Hornemann B, Honig K, Bergelt C, Maatouk I, Stein B, Teufel M, Erim Y, Geiser F, Niecke A, Senf B, Wickert M, Buttner-Teleaga A, Weis J. Information needs in cancer patients across the disease trajectory. A prospective study. Patient Educ Couns. 2020 Jan;103(1):120-126. doi: 10.1016/j.pec.2019.08.011. Epub 2019 Aug 12. PMID 31474389
• Diefenbach M, Turner G, Carpenter KM, Sheldon LK, Mustian KM, Gerend MA, Rini C, von Wagner C, Gritz ER, McQueen A, Prayor-Patterson HM, Miller SM. Cancer and patient-physician communication. J Health Commun. 2009;14 Suppl 1(Suppl 1):57-65. doi: 10.1080/10810730902814079. PMID 19449269
• Rosa WE, Levoy K, Doyon K, McDarby M, Ferrell BR, Parker PA, Sanders JJ, Epstein AS, Sullivan DR, Rosenberg AR. Integrating evidence-based communication principles into routine cancer care. Support Care Cancer. 2023 Sep 8;31(10):566. doi: 10.1007/s00520-023-08020-x. PMID 37682354
• Buljan I, Tokalic R, Roguljic M, Zakarija-Grkovic I, Vrdoljak D, Milic P, Puljak L, Marusic A. Framing the numerical findings of Cochrane plain language summaries: two randomized controlled trials. BMC Med Res Methodol. 2020 May 6;20(1):101. doi: 10.1186/s12874-020-00990-4. PMID 32375659
• Gainey K, Smith J, McCaffery K, Clifford S, Muscat D. Are plain language summaries published in health journals written according to instructions and health literacy principles? A systematic environmental scan. BMJ Open. 2024 Nov 27;14(11):e086464. doi: 10.1136/bmjopen-2024-086464. PMID 39608994
• Banic A, Fidahic M, Suto J, Roje R, Vuka I, Puljak L, Buljan I. Conclusiveness, linguistic characteristics and readability of Cochrane plain language summaries of intervention reviews: a cross-sectional study. BMC Med Res Methodol. 2022 Sep 10;22(1):240. doi: 10.1186/s12874-022-01721-7. PMID 36088293

Identifiers

NCT: NCT07140107 · 2181-198-03-04-25-0071 · 2181-147/01-06/LJ.Z.- 25-02

Primary sources (government registries)

• ClinicalTrials.gov (US NIH)

View this study on ClinicalTrials.gov ↗