Usability Evaluation of Gen AI-based Nutrition Chatbot for Pregnant Women

Objectives: This study primarily aims to evaluate the usability of a nutrition AI chatbot for pregnant women by comparing System Usability Scale (SUS) scores between intervention and control groups. Secondary objectives include assessing technology acceptance (Technology Acceptance Model), engagement patterns, information quality (accuracy, comprehensibility, consistency), and changes in nutritional knowledge.

Design: A quasi-experimental design with two parallel groups (n=50 each) will be employed. Using self-selection, participants will choose to enroll in the intervention group (access to an AI chatbot plus routine care) or the control group (access to a standardized WeChat information service plus routine care). Routine care for all participants includes standard prenatal clinic visits and printed nutritional materials.

The WeChat service for the control group will be operated by a trained research assistant using a pre-defined script during two scheduled windows daily, providing information quoted from the official nutritional leaflets. This isolates the mode of information delivery (AI versus human-facilitated messaging) as the primary variable.

Participants: Inclusion criteria: pregnant women aged ≥18 years, able to consent, owning a smartphone with internet access. Exclusion criteria: enrollment in other nutrition interventions or severe mental health conditions impairing technology use. A purposive subsample of 20 participants (10 per group) will complete qualitative interviews.

Sample Size: Based on an expected mean SUS score of 78 (SD=12) in the intervention group and 65 (SD=15) in the control group (Cohen's d=0.95), 23 participants per group are required for 90% power at alpha=0.05. Accounting for 50% attrition, 50 participants per group will be recruited. Propensity score matching will be applied to reduce selection bias using variables including age, gestational age, parity, education, and baseline technology use.

Measurements: The primary outcome, usability, will be measured using the System Usability Scale (SUS) and the Chatbot Usability Questionnaire (CUQ) at 12 weeks. Technology acceptance will be assessed using the Technology Acceptance Model (TAM). Nutritional knowledge will be evaluated at baseline and 12 weeks using a 15-item questionnaire and the FIGO Nutrition Checklist. Information accuracy and consistency will be assessed by an expert panel rating 150 chatbot responses and repeated submission of 20 test questions. Engagement will be analyzed via application usage logs measuring adherence, intensity, and persistence. Semi-structured interviews will explore user experiences in depth.

Data Analysis: Quantitative data will be analyzed using intention-to-treat principles. Primary analysis will compare mean SUS scores between groups using independent samples t-tests, with effect sizes calculated as Cohen's d. Secondary outcomes will be analyzed using similar approaches, with chi-square tests for proportions and linear mixed models for nutritional knowledge change scores. Missing data will be addressed through multiple imputation. Qualitative interview transcripts will be analyzed using thematic analysis with dual independent coding.

Study Timeline: Participants will be enrolled over a 6-month period, with each participant completing a 12-week intervention and follow-up period.

• Mean Score on the Technology Acceptance Model (TAM) Questionnaire [Time frame: 12 weeks]
• Proportion of Participants Achieving Adequate Engagement Adherence [Time frame: 12 weeks]
• Mean Number of Platform Logins per Week [Time frame: 12 weeks]
• Mean Number of Queries Submitted per Participant [Time frame: 12 weeks]
• Proportion of Chatbot Responses Rated as Accurate by Clinical Expert Panel [Time frame: 12 weeks]