Evaluating AI for the Translation of KM Resources

Background

Knowledge mobilization (KM) aims to ensure that research and evidence are effectively shared, applied and integrated into decision-making processes \[1, 2\]. KM resources serve different end-users (e.g., policymakers, healthcare providers, patients, caregivers) and can take various forms depending on their target audience (e.g., resources for patients include plain language summaries, videos, or infographics). By translating complex health information into user-friendly formats and evaluating dissemination strategies, this approach enhances awareness, engagement, and accessibility. In the context of child health, KM helps parents and caregivers access, understand, and use research findings to make informed healthcare decisions \[3\].

Our team has successfully co-created KM resources with families, integrating research, lived experience, and art to resonate with parents \[4-10\]. Our resources reflect diverse identities, including gender, family structure, ability, appearance, and socioeconomic status. The investigators have extensively evaluated accessible formats (e.g., video, interactive infographics) and effective dissemination strategies (e.g., social media) to maximize engagement \[11-13\]. Interactive infographics offer broad accessibility, easily shared across various platforms (e.g., websites, social media) to reach diverse audiences, regardless of geographic location (urban, rural, remote) or socioeconomic background (e.g., gender, race, culture, education, income) \[14\].

Communication challenges due to language barriers (specifically understanding or accessing healthcare information) have been shown to impact quality of care and lead to poor health outcomes \[15\]. While KM resources are meant to be widely accessible, they can be challenging for those whose first language is not English. According to the 2021 Canadian census, immigrants make up 23% of the population. This proportion is growing and is expected to reach 29-34% by 2041 \[16, 17\]. The proportion in Alberta is consistent with Canada overall at 23%, with the top places of birth of immigrants being the Philippines, India, and China. Therefore, the communities of interest in this proposal also represent common non-English languages spoken at home in Alberta: Mandarin, Punjabi, Tagalog, Urdu \[18\].

With increasing diversity in Alberta, the complexities associated with supporting groups with different ethnocultural backgrounds also increases. In situations where immigrants may face challenges in adjusting to new sociopolitical environments and health systems, KM in health is critical to address their knowledge and information needs \[19, 20\]. Translating KM resources into multiple languages can mitigate disparities in healthcare access and quality, ensuring that immigrants receive the information they need to navigate and engage with the health system effectively \[15\]. However, language translation of KM resources by a professional (i.e., human) translator can be resource intensive, costly, and time consuming. Artificial intelligence (AI) applications (e.g., ChatGPT, DeepL, Google Translate) have the potential for more efficient translation \[21\]. Yet, there may be disadvantages with using AI for translation. For instance, translations involve not only the literal meaning of words but also the cultural implications and social connotations attached to them \[22, 23\]. AI may lack the ability to recognize cultural subtleties, which could result in insensitive or inappropriate interpretations \[22\]. Nevertheless, as technology rapidly advances, AI is likely to be used for healthcare communications to enhance access and quality of care for those who are not proficient in English \[24\]. Empirical assessments of AI translation are therefore essential for understanding its capabilities and limitations.

Our previous work has demonstrated that parent-informed and parent-mediated KM resources can increase knowledge and influence decision-making and expectations regarding healthcare seeking; \[25, 26\] in turn, expectations can impact the care delivered by health professionals. A key tenet of KM is that resources are accessible to the intended end users. An important barrier to accessing KM resources is difficulty understanding the information which may arise due to language limitations. With nearly 25% of Alberta's population reporting an immigrant mother tongue, it is essential to assess the utility of English versus non-English resources to promote equitable access to research and support informed healthcare decisions. Evaluating professional versus AI translated resources will provide crucial information to support efficiencies in KM. This project has the potential for substantial impact in dissemination and uptake of research.

Objective

The study objective is to explore two key research questions (RQs) regarding the language translation of child health KM resources:

RQ1: How does language translation of a pediatric KM resource influence the understanding, accessibility, usability, satisfaction, and preferences of parents whose first language is not English?

RQ2: How do outcomes (understanding, usability, readability, and acceptability) compare between professional and AI translated versions?

Methods

Trial design: This study will use a multi-methods approach, combining both quantitative (non-pharmacological randomized controlled trial \[RCT\]) and qualitative (interview) data to provide a comprehensive understanding of our research questions \[27\]. RCTs are the most rigorous design to compare interventions and will provide estimates of differences between the versions of a KM resource. The qualitative interviews will help us gain a greater understanding of any observed differences and parent preferences.

Trial setting: The trial will be conducted entirely online within Canada. Participants will complete the study remotely using their personal electronic devices (e.g., computers, tablets) with internet access. All materials (study link, instructions, questionnaires, interviews) will be delivered electronically.

Patient and public involvement: The investigators will involve parents as partners in this work to help refine the study design, interpret results, and disseminate study findings. This will consist of seeking feedback from our Pediatric Parents' Advisory Group (P-PAG) that was established by our research team in 2016 \[28\].

Reporting guidelines: The study protocol is reported using the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) \[29\]. The investigators will follow established guidance for reporting RCTs and qualitative studies: CONSORT (Consolidated Standards of Reporting Trials) \[30\] and SRQR (Standards for Reporting Qualitative Research) \[31\].

Recruitment: The recruitment approach will be through convenience sampling. The investigators will co-develop a recruitment strategy with our target end users through our established P-PAG. Parents will be recruited from the communities of interest using traditional and non-traditional approaches (e.g., newsletters, posters, list serv, social media) in partnership with our collaborators (Our Kids Health, Translating Emergency Knowledge for Kids).

Study screening, enrollment and consent: Parents will begin the screening and study enrollment process by clicking the study link (hosted by the REDCap platform) provided on the recruitment materials \[32, 33\]. After accessing the study link, parents will complete a short set of eligibility questions to self-identify as eligible to participate in the study, and review the study information letter. Parents can read the letter and will have the option to download a copy. They will then be stratified by language, randomized and directed within the same study link to complete the online study tasks (e.g., review KM resource\[s\], questionnaire). Implied consent by overt action will be sought for this portion of the study.

Participants will self-identify as being interested in participating in the optional one-on-one interview at the end of the questionnaire. Interested participants will then be contacted to arrange an interview. Verbal consent will be obtained on Zoom prior to starting the interview and before any data collection. This will be documented in the consent log.

Design and study flow: After enrollment (as described above), participants will be stratified by language (Mandarin, Punjabi, Tagalog, Urdu). Within each language stratum, they will be randomized (computer-generated sequence developed by a statistician) in a 2:1 ratio into one of two groups aligned with the study's research questions:

* RQ1 Group: English vs. Professional Translation * RQ2 Group: Professional Translation vs. AI Translation

Within each RQ group, participants will undergo a second randomization (1:1 ratio, using a random number generator) to determine which version of the KM resource they receive.

RQ1 Group: Participants will be shown either the English or the professionally translated online KM resource in their language via REDCap. They will then be directed to complete a series of 16 questions that will be used to measure the outcomes: understandability, accessibility, usability, satisfaction. After this, they will be shown the alternate version (English or professionally translated) and asked a series of 3 questions about their preference, followed by a series of 10-12 demographic questions to describe the study population (age, education level, country of birth, time in Canada, including self-reported health literacy).

RQ2 Group: Participants will receive either the professionally translated or the AI-translated online KM resource in their language via REDCap. They will then be directed to complete a series of 20 questions that will be used to measure the outcomes: understandability, readability, acceptability, followed by a series of 10-12 demographic questions to describe the study population (age, education level, country of birth, time in Canada, including self-reported health literacy).

Reviewing the resources and completing the questionnaire is expected to take 10-20 minutes via REDCap, depending on whether participants are randomized to RQ1 or RQ2. Data will be collected and managed using the REDCap survey platform for this portion of the study.

Blinding: Participants will be blinded to the study hypothesis and to their group allocation. Team members involved in data collection and analysis, as well as the Investigators, will remain blinded during data interpretation and manuscript drafting. General group labels will be used until group allocation is revealed.

Intervention (KM resource): For the KM resource, the investigators selected an interactive infographic that was originally created in English with funding from Women and Children's Health Research Institute (WCHRI) and the Stollery Children's Hospital Foundation (SCHF). The topic was identified as a priority by local interest holders at the Stollery Children's Hospital. Further, the investigators have already been asked by the Multicultural Health Brokers Cooperative in Edmonton to translate this resource into Ukrainian and Russian due to interest from these communities (translations completed in 2024). The infographic provides information about what to expect when taking an ill or injured child to the emergency department (ED). The infographic contains 750 words and takes on average 3 minutes to read. For this study, the infographic will be professionally and AI-translated into Mandarin, Punjabi, Tagalog and Urdu. ChatGPT (the latest version available at time of translation) will be used, as it is widely used and easily accessible, increasing the generalizability of our results. ChatGPT has also demonstrated strong performance metrics, process efficiency, and cost-effectiveness \[34\].

Interviews (optional): Participants will be invited to take part in a one-on-one semi-structured onlin

• Acceptability/Usability [Срок оценки: Baseline]
• Satisfaction Level [Срок оценки: Baseline]
• Preference [Срок оценки: Baseline]
• Readability [Срок оценки: Baseline]
• Method of Translation and Acceptability [Срок оценки: Baseline]