Large Language Model-Assisted cTNM Annotation From Chinese PSMA PET/CT Reports

This is a multicenter observational diagnostic accuracy validation study based on Chinese PSMA PET/CT report texts from patients with prostate cancer or suspected prostate cancer. The study is not designed to evaluate a drug, device, surgical procedure, or imaging intervention. PSMA PET/CT examinations will be performed as part of routine clinical care, and the study will only analyze de-identified report texts and necessary baseline information after the reports have been finalized.

The study consists of retrospective and prospective components. Retrospectively, approximately 4,000 PSMA PET/CT reports from the First Affiliated Hospital of Wenzhou Medical University will be systematically annotated to construct a research database. An internal independent validation set of 300 reports, not used for model development or prompt optimization, will be used to evaluate the performance of the large language model. The reference standard for this 300-report validation set will be established by two experienced urologists through joint annotation, with adjudication by a nuclear medicine expert when needed. External validation will be performed using 110 de-identified reports from the First Affiliated Hospital of Ningbo University and 102 de-identified reports from Liuzhou People's Hospital. In addition, after ethics approval, 100 consecutive routine PSMA PET/CT reports from the First Affiliated Hospital of Wenzhou Medical University will be prospectively included to evaluate the accuracy and operational stability of the frozen model and prompt versions.

The large language model workflow will be deployed locally or in an institutionally controlled environment. The model will be instructed to generate structured JSON outputs, including cT\_report, cN\_report, cM\_report, cT\_uncertain, cN\_uncertain, cM\_uncertain, evidence\_T, evidence\_N, and evidence\_M. The target task is report-derived imaging cTNM staging annotation, not pathological TNM staging or overall AJCC stage grouping. The model output will be used only for research evaluation and methodological analysis and will not be used for clinical diagnosis, treatment decision-making, or patient notification.

A human-AI comparison will be conducted on the 300-report internal validation set. Eight human evaluators from urology and imaging-related specialties, including trainees, residents, attending physicians, and associate chief physicians, will independently annotate the reports before and after learning the annotation manual. Annotation time will be recorded for each round. The performance of human evaluators and the large language model will be compared against the expert consensus reference standard.

The main outcome will be the accuracy of the large language model in identifying cT, cN, and cM categories from Chinese PSMA PET/CT reports. Secondary outcomes will include precision, recall, F1-score, macro-F1, micro-F1, complete cTNM triplet matching rate, uncertainty recognition performance, evidence extraction quality, human-AI comparison results, annotation time, external validation performance, prospective validation performance, and error type distribution. Error analysis will focus on local tumor extent, regional versus non-regional lymph node boundaries, bone and visceral metastasis recognition, equivocal wording, treatment-related context, benign or inflammatory alternatives, and lesions not attributable to prostate cancer.

• Component-Level Accuracy of LLM-Based Uncertainty Recognition [Срок оценки: After freezing the model and prompt versions, through completion of all validation analyses, up to 18 months.]
• Complete cTNM Triplet Matching Rate for Human Evaluators and the LLM [Срок оценки: During pre-training and post-training human annotation rounds and LLM batch inference, up to 18 months.]
• Annotation Time per Report for Human Evaluators and the LLM [Срок оценки: During pre-training and post-training human annotation rounds and LLM batch inference, up to 18 months.]