Effect of a Localized ICU AI Teaching Agent on Rotating ICU Residents

1. Research Background Early-stage intensive care unit (ICU) clinical rotation imposes substantial cognitive challenges for newly incoming resident physicians. Novice residents must adapt to unfamiliar ward workflows, manage critically ill patients with unstable vital status under severe time constraints, and master a large volume of hospital-specific institutional clinical protocols covering routine and emergency ICU management. Rooted in Cognitive Load Theory, unnecessary extraneous cognitive workload originating from repeated information searching and cross-verification severely occupies limited working memory capacity and hinders clinical learning efficiency during high-intensity residency training.

General-purpose large language model (LLM) AI tools have gained widespread popularity as supplementary learning resources among medical trainees worldwide. Nevertheless, conventional off-the-shelf LLMs lack embedded access to hospital-specific, site-validated ICU clinical protocols, which leads to generalized, decontextualized recommendations inconsistent with local institutional practice requirements. Consequently, residents are forced to spend extra working memory to validate AI-generated suggestions against internal hospital guidelines, generating avoidable cognitive burden that impedes on-the-job protocol learning and bedside task execution.

Retrieval-augmented generation (RAG) framework enables customized institutional AI agents bounded exclusively within locally approved ICU protocols, delivering source-cited, site-compliant clinical guidance without requiring post-hoc manual verification by trainees. Existing medical education AI research predominantly evaluates model performance on generalized medical knowledge examinations rather than real-world on-site protocol application within authentic ICU working environments. This prospective randomized controlled trial is designed to fill this research gap by comparing the educational benefits of a hospital-customized protocol-locked AI agent (ICU-Tutor) versus unrestricted free access to commercial general LLMs among first-time ICU rotating residents during a standardized 14-day observation window. The core research hypothesis specifies that ICU-Tutor will lower resident cognitive load and improve protocol-based task performance relative to open-access general AI. 2. Predefined Study Objectives Primary Objectives To prospectively evaluate whether ICU-Tutor reduces standardized task completion time and self-reported Paas-scale cognitive load on study Day 7 compared with unrestricted general-purpose AI access in first-time ICU rotating residents. These two metrics serve as co-primary trial endpoints for formal statistical comparison.

Confirmatory Secondary Objectives

Three pre-specified secondary endpoints will undergo formal statistical testing with pre-defined multiple-testing correction rules:

Compare composite standardized practical task performance scores between the two study arms at post-randomization Day 7; Quantify inter-group differences of institutional ICU protocol knowledge retention via closed-book written assessment administered on Day 14; Measure participant satisfaction differences toward assigned AI support tools using a 0-100 visual analogue scale (VAS) collected on Day 7.

Exploratory Objective Separate individual score analysis across three independent practical assessment stations including airway management, hemodynamic assessment and ventilator troubleshooting will be implemented as exploratory analyses only. No formal statistical correction will be applied for these sub-station outcomes, whose aggregated total score constitutes the composite Day7 task performance primary secondary endpoint. All exploratory findings are intended for hypothesis generation rather than confirmatory conclusion. 3. Trial Design and Study Setting This is a prospective, single-center, pragmatic, parallel-group superiority randomized controlled trial currently in active participant recruitment, scheduled to run from March 2026 through June 2026 at ICU clinical teaching units of Peking Union Medical College Hospital, Beijing, China, a large tertiary academic medical center. The pragmatic trial design preserves residents' routine daily ICU clinical responsibilities throughout the entire research period without modification to regular clinical rotation arrangements; all assigned AI interventions are integrated into authentic daily clinical workflow consistent with standard real-world residency training practice. The trial protocol complies with core methodological and reporting standards from CONSORT 2010 and the CONSORT extension statement for pragmatic randomized trials. Institutional Review Board (IRB) approval (approval code I-26ZM0024) has been fully obtained prior to recruitment launch. 4. Participant Eligibility, Screening and Enrollment Plan Inclusion Criteria First-time ICU rotation participants with zero prior formal ICU clinical rotation experience; Scheduled to complete a minimum of four consecutive weeks of ICU training after trial enrollment; Capable of completing all scheduled multi-timepoint assessments (Day3 baseline, Day7 core endpoint tests, Day14 retention evaluation); Possess basic digital device operation competency to access assigned AI platforms during daily clinical work; Voluntarily provide written informed consent for full trial participation. Exclusion Criteria Accumulated prior formal ICU clinical experience exceeding seven calendar days; Regular daily clinical utilization of various commercial AI learning tools within three months preceding enrollment screening; Physical or cognitive impairment prohibiting full completion of all required trial assessments across pre-defined follow-up timepoints.

Screening \& Randomization Target Sample Size The research team plans to consecutively screen a total of 44 incoming rotating residents who initiate ICU rotation within the March-June 2026 recruitment window. Per pre-trial protocol estimation, all screened candidates are anticipated to satisfy pre-set eligibility criteria without exclusion, leading to full randomization of 44 qualified participants at a fixed 1:1 allocation ratio: 22 subjects assigned to the ICU-Tutor intervention group and another 22 participants allocated to the unrestricted general-AI control group. Recruitment activities are completed during centralized departmental resident orientation sessions hosted by the hospital's Division of Critical Care Medicine. 5. Randomization Strategy and Blinding Arrangement Eligible enrolled participants receive computer-generated variable-block randomization with stratified allocation to guarantee balanced baseline distribution across study groups. Pre-specified stratification covariates include postgraduate training year and baseline AI proficiency, with AI experience graded via a validated 1-5 Likert scale split into low experience (score 1-2) and high experience (score 3-5) subgroups. The fixed 1:1 grouping ratio yields planned n=22 per trial arm as defined in the trial statistical plan.

Complete participant blinding is not feasible due to distinct functional differences between the customized ICU-Tutor system and heterogeneous commercial general AI software products, and all residents are aware of their assigned AI tool category after randomization. To minimize assessment bias, all attending physicians responsible for grading standardized practical station assessments remain fully blinded to individual participants' group assignment throughout endpoint evaluation. Independent research coordinators exclusively manage randomization codes and participant grouping information, entirely separated from clinical assessors responsible for outcome scoring. 6. Detailed Description of Study Interventions All participants from both intervention and control cohorts attend a uniform 15-minute pre-intervention orientation covering standardized safe AI application specifications. The unified training explicitly emphasizes that all AI-derived content serves only as auxiliary clinical reference and cannot replace attending physician clinical judgment or mandatory institutional ICU policies.

Intervention Arm (ICU-Tutor, Planned n=22) ICU-Tutor is built on a strict RAG architecture anchored to a curated institutional knowledge repository containing 247 ICU-specific clinical protocols comprehensively peer-reviewed and validated by senior attending intensivists from the host hospital's critical care department. The covered protocol spectrum includes mechanical ventilation management, hemodynamic support, sedation and analgesia regulation, continuous renal replacement therapy (CRRT), antimicrobial stewardship codes, in-hospital emergency response algorithms, ICU enteral/parenteral nutrition guidance, delirium screening and intervention, venous thromboembolism prophylaxis, and end-of-life institutional care protocols.

A conservative safety constraint is embedded within ICU-Tutor's core algorithm: all system-generated responses are strictly limited to information retrieved exclusively from pre-approved internal protocol documents, with each output attached with traceable source links referencing original institutional guideline files. The hospital's ICU clinical governance committee assumes ongoing responsibility for periodic repository updates to maintain protocol currency throughout trial implementation.

Control Arm (Unrestricted General-Purpose AI, Planned n=22) Control-group participants receive full unrestricted permission to select any commercially available mainstream general LLMs per individual personal preference, matching routine off-protocol resident AI usage in real clinical practice. No constraints are imposed on tool type, daily access frequency or usage duration for control subjects. Consistent with pragmatic trial design goals, centralized backend usage log collection is not pre-scheduled for control participants given the heterogeneous assortment of self-selected third-party AI platforms. 7. Pre-specified Outcome Metrics and Assessment Timetable All assessments follow fixed pre-defined post-randomization timelines: baseline testing on study Day3, core primary plus confirmatory secondary endpoint measurements on Day7, and delayed knowledge retention written examination on Day14.

Co-Primary Endpoints (Uniform Day7 Assessment) Total task completion time (in minutes): cumulative time spent finishing a composite standardized ICU protocol task battery consisting of three independent practical assessment stations; Paas 9-point cognitive load scale score: validated single-item subjective rating ranging from 1 (extremely low mental workload) to 9 (extremely high mental workload), measuring overall perceived cognitive burden generated during standardized protocol task completion.

Confirmatory Secondary Endpoints Day7 composite practical task performance score (0-100 total points): aggregated total score combined from three separate practical station evaluations graded by blinded attending physicians using unified standardized checklists; Day14 institutional protocol knowledge retention score (percentage scoring): closed-book written examination covering core content of the hospital's verified ICU protocols; Day7 AI satisfaction VAS (0-100): continuous visual analogue scale measuring participants' overall satisfaction with their allocated AI auxiliary resource.

Exploratory Endpoints Independent discrete scoring results for three individual practical sub-stations: airway management (max 40 points), hemodynamic assessment (max 30 points), ventilator troubleshooting (max 30 points). These discrete sub-station results are solely for exploratory post-hoc analysis and are not defined as formal confirmatory trial endpoints. Supplementary self-reported learning efficiency questionnaires will be collected descriptively without pre-planned inter-group statistical compariso

• Composite practical task performance score [Срок оценки: Day 7 post-randomization]
• AI tool satisfaction visual analogue scale (VAS) score [Срок оценки: Day 7 post-randomization]