Artificial Intelligence Triage of Patients Requiring Emergency or Unscheduled Care Who Need to be Referred to Hospital

1. Introduction - scientific justification 1.1. Context As the population ages and the prevalence of chronic diseases increases, the number of general practitioners is declining. This growing imbalance between medical needs and supply is leading to difficulties in accessing primary care, particularly urgent and unscheduled care.

To address this issue, France has set up a healthcare access service (SAS). In the event of urgent or unscheduled care needs, the SAS allows individuals whose primary care physician is unavailable to access a healthcare professional. Depending on the situation, the service can provide medical advice, offer a teleconsultation, refer the person to a local doctor or emergency department, or even call out a mobile emergency and resuscitation unit (SMUR) or ambulance. The SAS functions as a triage system, providing a response tailored to each individual's needs.

The aim of this project is to study certain aspects of the validity of an artificial intelligence (AI) triage model for patients seeking urgent or unscheduled care, enabling the identification of patients who will require hospital treatment.

This project follows on from a previous study that developed a model using AI to triage patients using SOS Médecins Grand Paris. As in the SAS, people who call SOS Médecins present their problem to a medical dispatch assistant (MDA), who in most cases arranges a home visit.

The SOS Médecins databases contain audio recordings of care requests and the medical decisions made by doctors after their home visits, including whether or not the patient was referred to hospital (the data on patient outcomes is exhaustive). A predictive model of patients' use of hospital services following a home visit by an SOS doctor was therefore developed using AI. The model "listens" to the exchanges between the patient and the MDA and predicts the use of hospital services following the doctor's home visit.

The model and its development are the subject of an article currently submitted for publication in an international journal. During the test phase, it achieved an area under the ROC (receiver operating characteristic) curve of 0.80. Ultimately, the AI-based model will need to operate in real time and provide information to MDAs to assist them in their work. For the time being, the aim of this project is to complete the validation of the system. It will then need to be implemented and tested in real-world conditions in order to determine the benefits of patient triage by MDAs using the model's predictions.

It should be noted that while SOS Médecins is not the same as SAS, it is similar. First, the two structures operate in a similar way: an MDA contacted by telephone attempts to respond to a request for urgent or unscheduled care. The range of possible responses is the same for both structures (SOS does not only offer home visits, it also conducts face-to-face or video consultations and transfers certain calls to emergency medical service), as is the use of a doctor's expertise if necessary (medical regulation). Secondly, SOS Médecins is one of the downstream effectors of the SAS. Older and better known to the general public and healthcare professionals than the SAS (which currently covers only 60% of the national territory), SOS could have a recruitment process very similar to that of this new service. To our knowledge, there are not yet any statistics on the SAS that would allow a comparison of the activities of the two structures. In this context, validation within the SAS of the model developed with SOS data will therefore be necessary. If, in the future, data show that the two structures are similar, the SAS could benefit from this project developed at SOS. SOS data have a significant advantage over SAS data: they contain information on patient outcomes, including whether or not they went to the hospital emergency department. This labeling of calls enabled supervised learning of an AI model.

This project, which to our knowledge is unique, paves the way for triage assistance that can be used in the SAS (and perhaps to some extent in emergency departments).

1.2. Benefits for patients and the healthcare system Early identification of patients at high risk of hospitalization, some of whom have conditions requiring emergency care in a specialized setting, should facilitate clinical decision-making and ultimately improve patient health. This primarily concerns patients with conditions whose prognosis is directly linked to the speed of care delivery, such as severe ischemic, respiratory, traumatic, or infectious conditions, who will benefit from being hospitalized without waiting for a doctor to visit them at home. The benefit for patients will therefore be a reduction in lost opportunities due to triage errors.

In addition, triage upstream of emergency departments (as proposed by the previous Minister of Health and implemented in some understaffed emergency departments and is routine practice in Denmark) will improve the functioning of hospital emergency departments, where excess activity has an impact on the mortality of elderly patients receiving care. In terms of public health, the introduction of an effective triage system could therefore improve access to urgent and unscheduled care. 2. Research objectives and evaluation criteria The AI-based triage model used to identify patients who need to be referred to hospital has already been developed and tested using calls to SOS Médecins Grand Paris. An article is currently being published. Therefore, this project does not focus on the development or internal validation of the triage model, but on other aspects related to its validation: prospective and external validation; comparison of its predictions with those of a medical expert; investigation of social discrimination that could be associated with its use; and the determination of the actual conditions of its use in healthcare settings.

2.1. Objective and primary endpoint The prospective evaluation of the performance of predictive models on real-world operational data is a crucial step towards integrating these models into the clinical context and assessing their impact on patient care.

\- Primary objective (PO): Prospective validation of the triage model used to identify patients who will be referred to hospital following a home visit by a doctor.

The area under the ROC curve (AROC) provides an overall measure of the performance of a classification system.

It should be noted that all the data used in this study are retrospective. The model was designed using data from 2023 and earlier (up to and including 2017), and the "prospective" data are those subsequent to its design and will be the data for 2024.

\- Primary endpoint (PE): Comparison of AROCs obtained from representative samples of retrospective and prospective data from SOS Médecins Paris.

2.2. Secondary objectives and endpoints External validation on patient data from separate geographic sites is necessary to understand how models developed at one site can be safely and effectively implemented at other sites.

\- Secondary Objective #1 (SO1): External validation of the triage model * Secondary Endpoint #1 (SEP1): Comparison of AROCs obtained from representative samples of retrospective data from SOS Médecins Grand Paris and Ajaccio. * SEP2: Comparison of the triage model's predictions with those of an expert physician. * Secondary Endpoint #2 (SEP2): Comparison of correct prediction rates between the triage model and an expert physician on a balanced non-representative sample (i.e., comprising 50% of patients referred to the hospital following a home visit by the physician, rather than 3% as is the case in reality) of data from SOS Médecins Grand Paris.

Several AI-based triage systems have been found to be discriminatory. Two well-known examples illustrate this point: * the selection of computer engineers applying for jobs based on their resumes: the system had been trained using existing engineers, who were predominantly male, and it disadvantaged women; * The detection of melanomas from dermatoscope images: the system had been trained using light skin tones and was not as effective on dark skin tones. * OS3: Study of potential discrimination phenomena associated with the model, in relation to the social characteristics of patients. * Secondary endpoint #3 (CJS3): Comparison of AROCs obtained from representative samples of SOS Médecins Grand Paris data for both sexes.

The transition from a validated predictive model to an efficient device in real-world conditions is a well-known challenge in AI.

\- SO4: Determination of the real-world conditions of use of the model by MDAs.

\- Secondary endpoint #4 (SEP4): Drafting of specifications for the implementation of the triage system by MDAs 3. Selection and exclusion of individuals from the study 3.1. Inclusion criteria Individuals of any age who received a home visit from an SOS doctor following a call in 2024 or earlier.

3.2. Exclusion criteria Individuals who do not speak French. Individuals without health insurance. 4. Description of the study 4.1. Type of study This is essentially a study to validate an AI-based patient triage model. It will be conducted using observational data.

4.2. Research process 4.2.1. Location of the research The study will be conducted at the SOS Médecins premises. Medical experts from SOS Médecins Grand Paris will be recruited to carry out the SO2.

4.2.2. Recruitment methods Data collected by SOS Médecins in the course of its activities will be used.

4.2.3. Patient care procedures The project focuses on the need to use hospitals to meet the demand for urgent or unscheduled care from patients in the community. However, it will not interfere with patient care.

4.2.4. Study exit procedures Patients who request to be excluded will be excluded.

4.2.5. Data collection * PO (prospective validation), SO1 (external validation) and SO3 (discrimination):

The data used will be that already collected in the course of SOS Médecins' activities. Both sites (Paris and Ajaccio, Corsica) use the same software to manage telephone calls from the moment they are received and recorded until the end of the SOS care process (i.e., the end of the home visit in our case), including the entry of administrative and medical data (SOS Médecins file). Here is the list of data that will be used and how it will be processed for analysis:

\- Transcripts of audio recordings of conversations between patients and MDAs. AI predictions will be made based on these transcripts (and only these transcripts). * Data from the SOS Médecins file: * day of the week, month, year, and time of the phone call; * the census area number of the place of visit will be obtained from the address provided by the patient. Census area are the smallest spatial units for which statistical data is available while preserving the anonymity of its inhabitants. Several socio-economic variables from the INSEE (National Institute of Statistics and Economic Studies) census are available at this level, such as unemployment rates, high school graduates, workers, median income, the social disadvantage index (also known as FDep for French DEPrivation index) and EDI (for Ecological Deprivation Index). These different variables provide ecological (as opposed to individual) measures of the social position of the individuals residing there; * reasons for the call, defined by the MDA, based on a predefined list but not corresponding to an international classification. Several reasons may be reported for the same call; * level of urgency (4 levels), defined by the MDA; * date and time of the end of the home visit, defined by the doctor at the end of the visit * Consultation diagnoses, defined by the doctor at the end of the visit, based on a predefined list but not co

• The area under the ROC (receiver operating characteristic) curve (AROC) [Срок оценки: From the telephone call to the end of the home visit (maximum 24 hours)]
• Accuracy [Срок оценки: From the telephone call to the end of the home visit (maximum 24 hours)]
• The area under the ROC (receiver operating characteristic) curve (AROC) [Срок оценки: From the telephone call to the end of the home visit (maximum 24 hours)]