Research and Application of an AI Agent-Based MBBS Course Assistant

Overview

Prior to the quasi-experiment, this study performed semi-structured interviews among Bachelor of Medicine and Bachelor of Surgery(MBBS) undergraduate students from the Belt and Road International Medical College, Zhejiang University. Based on learning science, cognitive load theory and causal inference frameworks, the interview outline focused on system usability, feedback quality, learning reasoning processes, cognitive burden and instructional optimization advice. Each 20-30 minute individual interview was audio-recorded with participants' informed consent and verbatim transcribed for standardized qualitative analysis. The quasi-experiment recruited no fewer than 60 MBBS students. The sample size was determined by intergroup statistical power analysis to guarantee around 30 participants per group for valid intergroup comparison. All eligible students were randomly assigned into two groups with balanced demographic and academic baseline characteristics. The experimental group (n=30) received structured epidemiological learning assisted by the (Socratic Agent for Guided Epidemiology) SAGE (Artificial Intelligence)AI agent with professional Socratic cognitive guidance. The control group (n=30) adopted general large language model (LLM)-based learning without systematic thinking intervention. Participants with clinically diagnosed severe mental disorders, cognitive dysfunction or inability to finish the complete research process were excluded. A baseline epidemiological knowledge pre-test confirmed no significant academic differences between the two groups via independent samples t-test, and all participants had no prior experience of AI-assisted medical learning. The semi-structured interviews were conducted to collect students' authentic learning experiences, interactive perceptions and cognitive characteristics during the use of SAGE agent and general LLMs, providing qualitative evidence for the iterative optimization of AI teaching tools. All interviewees completed baseline assessments and preliminary AI learning trials, ensuring qualified professional foundation and genuine interactive experience. Conducted by trained researchers, the standardized interviews centered on three core themes: system usability and feedback clarity; AI-induced changes in information extraction, hypothesis formulation and causal inference; and common learning barriers including interactive obstacles, comprehension difficulties, cognitive overload and potential AI over-reliance. Transcribed interview data were analyzed through thematic analysis to summarize typical user experience patterns. Qualitative outcomes were triangulated with quantitative experimental results to revise the SAGE teaching protocol, optimize agent prompt chains and improve the interpretation of experimental findings. The quasi-experiment consisted of three standardized stages. In the pre-test stage, all participants signed informed consent, completed a 25-item clinical epidemiology knowledge scale, an 11-item reasoning ability test and a demographic questionnaire to establish consistent baseline levels. In the intervention stage, the experimental group received standardized training in confounder identification and causal inference construction in strict accordance with the SAGE teaching protocol. The SAGE agent improved students' advanced epidemiological reasoning ability through continuous multi-round Socratic questioning and targeted cognitive guidance. The control group received equal-duration learning in the same experimental environment, only using conventional search engines and unguided LLMs for basic information retrieval without any cognitive and thinking intervention. All participants submitted screenshots to record their accurate AI tool usage duration after completing learning tasks. In the post-test stage, all participants finished parallel-version epidemiological knowledge assessments and unified reasoning ability tests. Validated scales were adopted to evaluate students' cognitive load, system usability, learning satisfaction and academic self-confidence. Students' final scores of the Epidemiology course were collected as supplementary indicators of long-term learning effectiveness. Upon the completion of data collection, backend AI interaction logs were summarized and strictly screened. Invalid samples with insufficient interaction rounds or incomplete responses were excluded to ensure high data quality and reliable experimental conclusions.

Interventions

• Behavioral Socratic Agent for Guided Epidemiology
  A total of 30 participants are assigned to the experimental group receiving the SAGE teaching model, namely the Socratic Agent for Guided Epidemiology.
• Behavioral General LLM
  30 participants are allocated to the control group with teaching assistance from a general large language model (General LLM).

Primary outcome measures

• Clinical Epidemiology Knowledge Assessment Scale (Utrecht questionnaire on knowledge on clinical epidemiology for evidence-based practice) [Time frame: Before the intervention (One to seven days before starting the epidemiology course) and After the intervention(Within half a month after completing the epidemiology course)]
• Epidemiological Reasoning Test [Time frame: Before the intervention (One to seven days before starting the epidemiology course ) and After the intervention(Within half a month after completing the epidemiology course)]

Secondary outcome measures (6)

• System Usability Scale(SUS) [Time frame: Immediately after intervention(One to seven days after completing the epidemiology course)]
• Needs and perceptions questionnaire (AI-powered simulation-based teaching agent) [Time frame: Qualitative research stage (one to five months before starting the epidemiology course]
• Student's final exam score in the Epidemiology course [Time frame: Immediately after intervention(One to seven days after completing the epidemiology course)]
• Demographic information questionnaire [Time frame: Before the intervention (One to seven days before starting the epidemiology course)]
• Learning Satisfaction and Self-confidence Scale [Time frame: Immediately after intervention(One to seven days after completing the epidemiology course)]
• Cognitive Load Scale [Time frame: After the intervention(Within half a month after completing the epidemiology course)]

Eligibility criteria

Inclusion criteria

• 1: The undergraduate students of the "Belt and Road Initiative" International Medical School of Zhejiang University who have officially registered for the MBBS program
• 2: Voluntary signing of the informed consent form

Exclusion criteria

• 1: Having severe mental illnesses (such as severe depression, bipolar disorder, acute phase of schizophrenia), serious physical diseases or cognitive impairments
• 2: Is currently participating in other studies that may affect the outcome indicators of this research
• 3: Have systematically studied or participated in research that is highly similar to this study
• 4: No experience of AI agent learning

Criteria are shown verbatim from the registry (in English). Final eligibility is always assessed by the study center.

Healthy volunteers: Yes

Study design

Allocation

Randomized

Model

Parallel assignment

Masking

Single blind

Primary purpose

Basic science

Study locations

China · 1 center

• Zhejiang University International School of Medicine — Yiwu

Identifiers

NCT: NCT07752160 · KY-2025-374

Primary sources (government registries)

• ClinicalTrials.gov (US NIH)

View this study on ClinicalTrials.gov ↗