Concordance Between Large Language Model and Multidisciplinary Team Recommendations in Rectal Cancer

Management of rectal cancer often requires multidisciplinary decision-making based on tumor location, pelvic magnetic resonance imaging findings, clinical stage, mesorectal fascia or circumferential resection margin status, extramural vascular invasion, lateral lymph node status, metastatic status, previous treatment, surgical feasibility, organ preservation considerations, and patient preferences. Large language models have shown potential in medical information processing and clinical decision support, but their performance in complex oncologic decision-making has not been fully validated.

This study is designed as a prospective, single-center, observational concordance study. Consecutive patients with rectal adenocarcinoma who are scheduled for routine rectal cancer multidisciplinary team discussion at the study center will be screened. Before the multidisciplinary team meeting, investigators will prepare a standardized de-identified case summary using a predefined template. The summary will include relevant demographic information, clinical status, endoscopic findings, pathological and molecular information, key imaging findings, previous treatments, and patient preferences or practical constraints when available.

The same standardized case summary will be used as the input for a locally deployed large language model. A fixed prompt, fixed model version, and fixed inference parameters will be used throughout the study. Each case will be processed in an independent session, without additional interactive prompting or manual correction. The model will not use internet access, external knowledge retrieval, or retrieval-augmented generation during the study.

Routine multidisciplinary team discussion will proceed independently according to standard clinical workflow. The large language model output will not be provided to the multidisciplinary team and will not be used to guide patient treatment. Actual treatment decisions will be made by the treating physicians and multidisciplinary team according to routine clinical practice.

After both recommendations have been generated, the large language model recommendation and the multidisciplinary team recommendation will be transformed into a structured format. The structured recommendations will include the preferred treatment pathway, specific treatment plan, acceptable alternative options, key rationale, and need for additional examinations or information. De-identified and randomly ordered recommendations will then be evaluated using predefined concordance rules and blinded expert review.

The primary objective is to estimate the complete concordance rate between the large language model recommendation and the multidisciplinary team reference recommendation for the preferred treatment pathway. Secondary objectives include evaluation of concordance in specific treatment implementation, acceptable alternative options, identification of additional examinations or information needs, and the rate of major discordance. Exploratory analyses will assess patterns of major discordance and clinical features associated with concordant or discordant recommendations.

• Percentage of Cases With Concordant Specific Treatment Implementation Assessed Using a Structured Concordance Adjudication Form [Срок оценки: From enrollment to completion of recommendation adjudication for each case, up to 12 months.]
• Percentage of Applicable Cases With Concordant Alternative Treatment Options Assessed Using a Structured Concordance Adjudication Form [Срок оценки: From enrollment to completion of recommendation adjudication for each case, up to 12 months.]
• Percentage of Cases With Concordant Identification of Additional Examination or Information Needs Assessed Using a Structured Concordance Adjudication Form [Срок оценки: From enrollment to completion of recommendation adjudication for each case, up to 12 months.]
• Percentage of Cases With Major Discordance Assessed by Blinded Expert Review [Срок оценки: From enrollment to completion of recommendation adjudication for each case, up to 12 months.]
• Cohen's Kappa Coefficient for Inter-Rater Agreement in Blinded Expert Assessment [Срок оценки: At completion of blinded expert review, up to 12 months.]