R-TF-015-006 Clinical investigation report
Research Title
Project to enhance Dermatology E-Consultations in Primary Care Centres using Artificial Intelligence Tools.
Description
Study to validate the clinical utility of the device as a valid tool in managing patients with skin pathology by primary care practitioners and dermatologists at the Puerta de Hierro Majadahonda University Hospital involved in the project. In this study patients with suspected pathologies of the following in Primary Care: tumour pathology, inflammatory pathology and infectious pathology will be recruited. All of them will be treated at either Pozuelo Health Center or Majadahonda Health Center and referred to Puerta del Hierro Majadahonda University Hospital. For this study, 100 patients will be recruited.
Product Identification
| Information | |
|---|---|
| Device name | Legit.Health Plus (hereinafter, the device) |
| Model and type | NA |
| Version | 1.0.0.0 |
| Basic UDI-DI | 8437025550LegitCADx6X |
| Certificate number (if available) | MDR 792790 |
| EMDN code(s) | Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software) |
| GMDN code | 65975 |
| Class | Class IIb |
| Classification rule | Rule 11 |
| Novel product (True/False) | FALSE |
| Novel related clinical procedure (True/False) | FALSE |
| SRN | ES-MF-000025345 |
Promoter Identification and Contact
| Manufacturer data | |
|---|---|
| Legal manufacturer name | AI Labs Group S.L. |
| Address | Street Gran Vía 1, BAT Tower, 48001, Bilbao, Bizkaia (Spain) |
| SRN | ES-MF-000025345 |
| Person responsible for regulatory compliance | Alfonso Medela, María Diez, Giulia Foglia |
| office@legit.health | |
| Phone | +34 638127476 |
| Trademark | Legit.Health |
Identification of sponsors
- Puerta de Hierro Health Research Institute
Clinical Investigation Plan (CIP) Identification
- Title: Project to enhance Dermatology E-Consultations in Primary Care Centres using Artificial Intelligence Tools.
- Protocol code: LEGIT.HEALTH_DAO_Derivación_PH_2022
- Study design: Prospective observational analytical study of a longitudinal clinical case series.
- Product under investigation: Legit.Health.
- Version and date: Versión: 1.0. Date: 2022/06/29
Public Access Database
Please note that the database used in this study is not publicly accessible due to privacy and confidentiality considerations.
Research Team
Principal Investigators
- Dr. Gastón Roustán Gullón (Hospital Universitario Puerta del Hierro Majadahonda).
Collaborating Investigators
- Centro de Salud Majadahonda
- Dra. Esther Minguela
- Centro de Salud de Pozuelo
- Dr. Fernando León
- Hospital Universitario Puerta del Hierro Majadahonda
- Dr. Ángel Rosell Díaz
Investigational sites
- Hospital Universitario Puerta del Hierro de Majadahonda.
- Centro de Salud de Majadahonda.
- Centro de Salud de Pozuelo.
Compliance Statement
The clinical investigation was perforfed according to the Clinical Investigation Plan (CIP) and other applicable guidances and regulations. This includes compliance with:
- Harmonized standard
UNE-EN ISO 14155:2021 Regulation (EU) 2017/745 on medical devices (MDR)- Harmonized standard
UNE-EN ISO 13485:2016s Regulation (EU) 2016/679(GDPR).- Spanish
Organic Law 3/2018on the Protection of Personal Data and guarantee of digital rights`.
All data processing within the device is carried out in accordance with the highest standards of data protection and privacy. Patient information is managed in an encrypted manner to ensure confidentiality and security.
The research team assumes the role of Data Controller, responsible for the collection and management of study data. Legit.Health acts as the Data Processor and is not involved in the processing of patient data.
The storage and transfer of data comply with European data protection regulations. At the conclusion of the study, all information stored in the device will be permanently and securely deleted.
The device employs robust technical and organizational security measures to safeguard personal data against unauthorized access, alteration, loss, or processing.
Report date
2024-01-10
Report author(s)
The full name, the ID and the signature for the authorship, as well as the approval process of this document, can be found in the verified commits at the repository. This information is saved alongside the digital signature, to ensure the integrity of the document.
Table of contents
Table of contents
- Research Title
- Description
- Product Identification
- Promoter Identification and Contact
- Identification of sponsors
- Clinical Investigation Plan (CIP) Identification
- Public Access Database
- Research Team
- Compliance Statement
- Report date
- Report author(s)
- Table of contents
- Abbreviations and definitions
- Summary
- Introduction
- Investigational device and methods
- Results
- Discussion and Overall Conclusions
- Investigators and Administrative Structure of Clinical Research
- Report Annexes
Abbreviations and definitions
- AE: Adverse Event
- AEMPS: Spanish Agency of Medicines and Medical Devices
- AEP: Adverse Reaction to Product
- AUC: Area Under the ROC Curve
- CAD: Computer-Aided Diagnosis
- CMD: Data Monitoring Committee
- CIP: Clinical Investigation Plan
- CUS: Clinical Utility Questionnaire
- DLQI: Dermatology Quality of Life Index
- GCP: Standards of Good Clinical Practice
- ICH: International Conference of Harmonization
- IFU: Instructions For Use
- IRB: Institutional Review Board
- N/A: Not Applicable
- NCA: National Competent Authority
- PI: Principal Investigator
- SAE: Serious Adverse Events
- SAEP: Serious Adverse Event to Product
- SUAEP: Serious and Unexpected Adverse Event to the Product
- SUS: System Usability Scale
Summary
This is a prospective analytical observational study of a series of clinical cases designed to validate the clinical utility of the device in the management of patients with skin pathology by primary care practitioners and dermatologists at the Puerta de Hierro Majadahonda University Hospital involved in the project. This study, 100 patients were recruited from primary care centres who could potentially suffer from tumoral, inflammatory or infectious diseases. The primary objective is to validate the device as a relevant tool in the management of patients with skin pathology by primary care practitioners and dermatologists.
Title
Project to enhance Dermatology E-Consultations in Primary Care Centres using Artificial Intelligence Tools.
Introduction
Skin diseases represent about 5% of primary care (PC) visits, primarily among the active population, consuming significant healthcare resources. There is a notable diagnostic discrepancy between PC doctors and dermatologists, with agreement rates around 57-65%, partly due to limited dermatological training among PC doctors, leading to misdiagnoses and unnecessary referrals. Given the scarcity of dermatologists, especially in rural areas, PC doctors often perform preliminary assessments of skin conditions, though their diagnostic capacity is limited. To improve dermatology access and efficiency, teledermatology initiatives like teleDERMADRID have been launched in Madrid, allowing PC centres to send clinical images and information to dermatologists, reducing in-person consultations, travel, and delays in diagnosis and treatment. In parallel, artificial intelligence (AI) advancements aid PC doctors in triaging patients and have increased diagnostic accuracy by 12%, enhancing referral adequacy and patient care quality.
This study focuses on evaluating the information provided by the device increases the accuracy of practitioners in the diagnosis of different skin conditions. For this purpose, we pretend to include a diverse cohort of 100 patients, rigorous ethical standards, and adherence to regulatory guidelines, this research holds significant potential in revolutionizing dermatological diagnostics.
Objectives
Hypothesis
The device significantly improves the appropriateness of dermatology referrals. This is owing to the introduction of significant changes in the diagnostic process, such as greater sensitivity and specificity than a primary care practitioner in diagnosing skin conditions, especially in differentiating between malignant and benign lesions. And it also provides the reassurance of a second medical opinion, which has been clinically validated.
Primary objective
To validate that the information provided by the device increases the true accuracy of healthcare professionals (HCPs) for the diagnosis of multiple dermatological conditions.
Secondary objectives
- Reduce and correct the referral of patients with skin pathologies from primary care to dermatology.
- Individualize and improve the ongoing training of primary care practitioners in dermatology.
- Offer healthcare adapted to technological innovations.
- Measure the satisfaction of primary care practitioners with the the device.
- Measure the satisfaction of dermatologists with the the device.
Endpoint
- An improvement of diagnostic accuracy of 10% (Ferri et al. 2020) in primary care practitioners and dermatologists.
Population
Adult patients (≥ 18 years) with skin pathologies seen at their Health Centres (Majadahona or Pozuelo Health Centers) and whose reference hospital is Puerta del Hierro Majadahonda University Hospital. These patients should be diagnosed at their health centre and diagnosed with tumoral, inflammatory or infectious diseases.
Design and Methods
Study design
This is a prospective analytical observational study of a series of clinical cases.
Number of subjects
At least 15 researchers have made at least one diagnostic report using the device. In total, 180 diagnostic reports were recorded for 131 patients. In the review carried out in June 2023, the number of patients was 92, and as of the date of this report, the target of 100 has been exceeded.
Ethical considerations
This study adhered to international Good Clinical Practice (GCP) guidelines, the Declaration of Helsinki in its latest amendment, and applicable international and national regulations. As applicable, approval from the relevant Ethics Committee was obtained prior to the initiation of the study. When applicable, modifications to the protocol were reviewed and approved by the Principal Investigator (PI) and subsequently evaluated by the Ethics Committee before subjects were enrolled under a modified protocol.
This study was conducted in compliance with European Regulation 2016/679, of 27 April, concerning the protection of natural persons with regard to the processing of personal data and the free movement of such data (General Data Protection Regulation, GDPR), and Organic Law 3/2018, of 5 December, on the Protection of Personal Data and the guarantee of digital rights. In accordance with these regulations, no data enabling the personal identification of participants was collected, and all information was managed securely in an encrypted format.
Participants were informed both orally and in writing about all relevant aspects of the study, with the information being tailored to their level of understanding. They were provided with a copy of the informed consent form and the accompanying patient information sheet. Adequate time was given to patients to ask questions and fully comprehend the details of the study before providing their consent.
The PI was responsible for the preparation of the informed consent form, ensuring it included all elements required by the International Conference on Harmonisation (ICH), adhered to current regulatory guidelines, and complied with the ethical principles of GCP and the Declaration of Helsinki.
The original signed informed consent forms were securely stored in a restricted access area under the custody of the PI. These documents remained at the research site at all times. Participants were provided with a copy of their signed consent form for their records.
Data confidentiality
Current legislation will be complied with in terms of data confidentiality protection (European Regulation 2016/679, of 27 April, on the protection of natural persons with regard to the processing of personal data and the free movement of such data and Organic Law 3/2018, of 5 December, on Personal Data Protection and guarantee of digital rights). For this purpose, when applicable, each participant will receive an alphanumeric identification code in the study that will not include any data allowing personal identification (coded CRD). The Principal Investigator will have an independent list that will allow the connection of the identification codes of the patients participating in the study with their clinical and personal data. This document will be filed in a secure area with restricted access, under the custody of the Principal Investigator and will never leave the centre.
Once the paper CRDs are completed and closed by the Principal Investigator, the data will be transferred to a database.
As in the CRDs, the Database will comply with current legislation in terms of data confidentiality protection (European Regulation 2016/679, of 27 April, on the protection of natural persons about the processing of personal data and the free movement of such data and Organic Law 3/2018, of 5 December, on the Protection of Personal Data and guarantee of digital rights) in which no data allowing personal identification of patients will be included.
Data quality assurance
The Principal Investigator is responsible for reviewing and approving the protocol, signing the Principal Investigator commitment, guaranteeing that the persons involved in the centre will respect the confidentiality of patient information and protect personal data, and reviewing and approving the final study report together with the sponsor. All the clinical members of the research team assess the eligibility of the patients in the study, inform and request written informed consent, collect the source data of the study in the clinical record and transfer them to the Data Collection Notebook (DCN) or Data Collection Forms (CRF).
Description of the clinical investigation methods
Primary care practitioner recruitment
The Principal Investigator and/or the collaborating investigators assigned to this task will explain to the primary care doctors the purpose of their participation in the study, and doctors will be able to ask any questions they consider necessary to clarify their doubts about the study. Additionally, all doctors participating in the study will receive specific information on the use of the artificial intelligence tool, provided by an expert in the field. If a doctor wishes to participate in the study, they will sign the Informed Consent form, and a study code will be assigned to them. Once the informed consent is signed, the data collection process begins. The Principal Investigator and/or the collaborating investigators assigned to this task will collect demographic data (age, centre, years of practice) and explain to the doctor the steps they must follow as part of the study.
Patient Identification
The primary objective of the study is to assess the perceived clinical utility of the doctors, so the clinical variables of the patients are not the focus of the study. Therefore, patients will only be identified in the tool to enable its use, and no further analysis or data processing of the patient's information will take place. The primary care doctor will explain to the patient what their participation in the study would entail through the Patient Information Sheet. The patient, in turn, can ask any questions they deem appropriate to clarify their doubts about the study. If the patient wishes to participate in the study, they will sign the Informed Consent form and be assigned a study code. Once the informed consent is signed, the data collection process begins. The Principal Investigator and/or the collaborating investigators assigned to this task will enter demographic data (age, sex) and information related to the diagnosis, characteristics, and treatment of the condition into the the device.
Procedures to be Performed by Primary Care Doctors
Consultation
Primary care doctors must take photographs showing the areas affected by the condition. These photographs will be taken with their own smartphones or a mobile dermatoscope if their use is clinically relevant. Primary care doctors will upload the photographs to the the device. They will assess the patient's condition through direct observation and guided by the platform's results. They will enter the diagnosis in the the device by selecting the pathology if it is among the top five options or through a dropdown menu displaying a wide range of conditions. Besides the probable diagnoses, doctors will have access to referral criteria, a clinical referral questionnaire, and basic treatment recommendations. When deemed necessary, they will refer the patient in their Selene system, as they normally would, not through the device.
Completion of Questionnaires
Primary care doctors will complete the Clinical Utility and Satisfaction Questionnaire twice during the study: once at 2 months and again at 4 months after the start of the study.
Procedures to be Performed by Specialist Doctors
Consultation
The Principal Investigator will see their patients while maintaining their standard clinical routine. They will note the appropriateness of the patient referral in the Case Report Form (CRF). If they consider the referral to dermatology inappropriate, this will also be recorded. The initial diagnosis provided by the primary care doctor will be documented, as well as the specialist's diagnosis. If relevant, the pathology diagnosis will be included additionally.
Completition of questionnaires
Dermatologists will complete the Clinical Utility and Satisfaction Questionnaire twice during the study: once at 2 months and again at 4 months after the start of the study.
Results of the clinical investigation
15 primary care practitioners were included in this study. In total, 180 diagnostic reports were recorded for 131 patients. In diagnosing specific conditions such as hidradenitis suppurativa (HS), the device helped identify two cases initially undiagnosed by primary care practitioners (PCPs), with one confirmed later by a dermatologist. Additionally, HS appeared as a suggested diagnosis in four other reports. For urticaria, the tool guided one confirmed diagnosis, with no further suggestions. For psoriasis, eight cases were confirmed, with the tool suggesting psoriasis in 23 instances, resulting in 10 diagnoses by PCPs.
In skin cancer cases, specifically melanoma, five cases were confirmed through pathology, with dermatologists suspecting melanoma in 10 cases. Using the device, specialists achieved a sensitivity of 60% and specificity of 91% for melanoma detection. Overall, the tool's AUC for malignancy detection (including melanoma and carcinomas) was 0.84. Clinicians completed a survey on the clinical utility of the device, with responses from 8 primary care practitioners and 2 dermatologists.
Conclusion of the clinical investigation
The device improved the diagnosis of skin diseases by helping primary care practitioners (PCPs) identify cases of hidradenitis suppurativa and psoriasis that were previously undiagnosed without the device, and by enhancing urticaria detection. For skin cancer detection, specifically melanoma, GPs using the tool achieved a sensitivity of 60% and a specificity of 91%. The device's malignancy index reached an AUC of 0.84, indicating strong differentiation between malignant and benign cases.
GPs and dermatologists reported high satisfaction with the tool's overall performance, ease of use, diagnostic support, and patient management efficiency. The platform also received positive feedback for triaging and handling urgent cases. Although this study could not analyze improvements in dermatological disease diagnosis by primary care practitioners due to the low volume of pathologies and HCP participation in this area, this will be considered in future studies to evaluate the diagnostic accuracy improvement with the device.
Introduction
Skin-related diseases are a common reason for primary care (PC) consultations, estimated to account for about 5% of all visits, mostly among the working population. This significant use of resources highlights the need for an efficient approach to optimize primary care operations.
Numerous studies reveal discrepancies between PC doctors and dermatologists, with diagnostic agreement rates ranging between 57% and 65.5%, depending on the study. In general, PC doctors demonstrate limited expertise in skin disease diagnosis and treatment. This knowledge gap impacts the time and effort required to assess the severity and stage of a patient's condition, often resulting in low protocol adherence and inadequate referrals. Given the shortage of medical professionals, especially dermatologists—averaging just three per 100,000 inhabitants—access to specialists is particularly challenging in smaller communities, leaving PC doctors to screen dermatological conditions, despite an increased risk of misdiagnosis. Studies show a diagnostic discordance rate between 55% and 65% between PC doctors and specialists, with frequent misdiagnoses of common dermatological diseases by non-dermatologists, such as drug-induced rashes and fungal infections.
Additionally, patient self-reporting introduces potential bias, especially when treatment relies on patient-reported information, and healthcare teams often lack the resources to verify the accuracy of these reports. Since 2020, healthcare information systems have included an asynchronous e-consultation for dermatology, allowing clinical information and images from health centres to be sent to the dermatology department at Hospital Puerta de Hierro Majadahonda, integrated within the hospital's Selene system.
The gradual introduction of teledermatology as part of the strategic plan in the Madrid region culminated in the teleDERMADRID project, led by Pilar Sánchez Pobre Bejarano and Manuel Grandal Martín (GAOAIO), in coordination with DGASA. This initiative offers a new, non-face-to-face consultation mode, adapted to the current social and technological landscape. It includes an Image Acquisition Management (IAM) tool, accessible via QR code on mobile devices, linked to the integrated request system (SIPE), facilitating fast, generally satisfactory responses to health issues, reducing diagnostic delays, unnecessary travel, and expediting treatment access.
Reducing the pressure on in-person hospital consultations lowers conventional healthcare costs while upholding patient-centred care as a priority to improve health-related quality of life. Alongside the image tool, a clinical form is used to complement image submissions with necessary items.
The development of Information and Communication Technologies (ICTs) has enabled advancements in medicine through artificial intelligence (AI), automating processes like patient triage, which allows for faster prioritization of severe cases. The use of these systems by primary care doctors has increased diagnostic accuracy by 12%, proving highly useful in refining referrals for patients with skin conditions.
Thus, this study aims to clinically validate an innovative AI tool to enhance the appropriateness of primary care referrals to dermatology.
Investigational device and methods
Investigational device description
This section contains a short summary of the device. A complete description of the intended purpose, including device description, can be found in the record Legit.Health Plus description and specifications.
Product description
The device is a computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organisations to gather data and improve their workflows.
The generated data is intended to aid healthcare practitioners and organizations in their clinical decision-making process, thus enhancing the efficiency and accuracy of care delivery.
The device should never be used to confirm a clinical diagnosis. On the contrary, its result is one element of the overall clinical assessment. Indeed, the device is designed to be used when a healthcare practitioner chooses to obtain additional information to consider a decision.
Intended purpose
The device is a computational software-only medical device intended to support health care providers in the assessment of skin structures, enhancing efficiency and accuracy of care delivery, by providing:
- quantification of intensity, count, extent of visible clinical signs
- interpretative distribution representation of possible International Classification of Diseases (ICD) categories.
Intended previous uses
No specific intended use was designated in prior stages of development.
Product changes during clinical research
The device maintained a consistent performance and features throughout the entire clinical research process. No alterations or modifications were made during this period.
Clinical Investigation Plan
Objectives
This study aims to validate that the information provided by the device increases the true accuracy of healthcare professionals (HCPs) for the diagnosis of multiple dermatological conditions.
Design
This is a prospective observational analytical study of a clinical case series. The study does not involve an active or control group, as it is focused on the evaluation of the device in a real-world clinical setting. The group of primary care practitioners will act as their control group, first without using the device and making their diagnosis and afterwards using the device to confirm or change their diagnosis. The assessment criteria include the completion of questionnaires, photograph submission, and patient-reported outcomes through the device. The study employs a variety of methods, including data collection through questionnaires, photograph analysis, and patient satisfaction surveys.
Ethical considerations
This study adhered to international Good Clinical Practice (GCP) guidelines, the Declaration of Helsinki in its latest amendment, and applicable international and national regulations. As applicable, approval from the relevant Ethics Committee was obtained prior to the initiation of the study. When applicable, modifications to the protocol were reviewed and approved by the Principal Investigator (PI) and subsequently evaluated by the Ethics Committee before subjects were enrolled under a modified protocol.
This study was conducted in compliance with European Regulation 2016/679, of 27 April, concerning the protection of natural persons with regard to the processing of personal data and the free movement of such data (General Data Protection Regulation, GDPR), and Organic Law 3/2018, of 5 December, on the Protection of Personal Data and the guarantee of digital rights. In accordance with these regulations, no data enabling the personal identification of participants was collected, and all information was managed securely in an encrypted format.
Participants were informed both orally and in writing about all relevant aspects of the study, with the information being tailored to their level of understanding. They were provided with a copy of the informed consent form and the accompanying patient information sheet. Adequate time was given to patients to ask questions and fully comprehend the details of the study before providing their consent.
The PI was responsible for the preparation of the informed consent form, ensuring it included all elements required by the International Conference on Harmonisation (ICH), adhered to current regulatory guidelines, and complied with the ethical principles of GCP and the Declaration of Helsinki.
The original signed informed consent forms were securely stored in a restricted access area under the custody of the PI. These documents remained at the research site at all times. Participants were provided with a copy of their signed consent form for their records.
Data Quality Assurance
The Principal Investigator is responsible for reviewing and approving the protocol, signing the Principal Investigator commitment, guaranteeing that the persons involved in the centre will respect the confidentiality of patient information and protect personal data, and reviewing and approving the final study report together with the sponsor. All the clinical members of the research team assess the eligibility of the patients in the study, inform and request written informed consent, collect the source data of the study in the clinical record and transfer them to the Data Collection Notebook (DCN) or Data Collection Forms (CRF).
Subject Population
Adult patients (>18 years) who come to their primary care centre with tumour pathology, inflammatory pathology or infectious pathology. They can be treated in the Health Center of Pozuelo and the Health Center of Majadahonda. It is important to note that some patients presented with more than one issue, contributing to the total count.
Inclusion Criteria
Patients suspected of having the following conditions in Primary Care:
- Tumor pathology:
- Benign:
- Histiocytoma
- Seborrheic keratosis
- Angiomas
- Precancerous:
- Actinic keratosis
- Suspected malignancy:
- Basal cell carcinoma
- Squamous cell carcinoma
- Pigmented lesions:
- Melanocytic nevus
- Malignant melanoma
- Benign:
- Inflammatory pathology:
- Psoriasis
- Atopic dermatitis
- Urticaria
- Hidradenitis suppurativa
- Lichen planus
- Infectious pathology:
- Viral warts
- Molluscs
- Herpes simplex
- Patients aged 18 years or older.
- Patients who have signed the informed consent for the study.
Exclusion Criteria
- Patients under 18 years of age.
- Pregnant patients.
- Patients who, in the opinion of the researcher, will not comply with the study procedures.
Sample size
The goal of this study is to evaluate whether the use of the device by HCPs can improve the diagnostic accuracy of skin pathologies by at least 10%. The 10% improvement refers to an increase in the proportion of correct diagnoses compared to baseline performance. This level of improvement would provide a strong justification for a meaningful change in clinical practice. Consequently, 15 HCPs will participate in this study, which provides a strong basis and ensures sufficient statistical power. For this reason, assuming the diagnostic accuracy is measured as a proportion (e.g., correct diagnoses), a sample size of 100 patients can provide a 95% confidence level with a margin of error of around 9-10% for the accuracy rate and 80% of power.
The sample size is sufficent for evaluating qualitative assessments, such as evaluating usability, satisfaction, and clinical utility, using validated scales and descriptive statistics. Each HCP will interact with 6–7 patients on average (100 patients distributed across 15 doctors), ensuring a manageable workload and thorough evaluation of each case. However, subgroup analysis may be limited by the small number of participants per group. The design provides a solid foundation for detecting meaningful improvements in diagnostic accuracy, HCP satisfaction and exploring the usability of the device in clinical settings.
Population
100 patients were pretended to be recruited in this study. Finally, 180 diagnostic reports of 131 patients were recorded, exceeding the sample size planned at the beginning. This study is proposed as a pilot proof-of-concept study in which the sample size has been estimated based on the number of primary care practitioners who treat patients diagnosed with skin diseases who can be treated in the primary care service of the following centres:
- Majadahonda Health Center
- Pozuelo Health Center
During the recruitment period of the study, all patients diagnosed with skin diseases who met the selection criteria were included.
Treatment
Patients in this study did not receive any specific treatment as part of the research protocol.
Concomitant Medication/Treatment
Patients continued their regular prescribed medications and treatments as directed by their primary healthcare providers. No additional medications or treatments were administered as part of this study.
Follow-Up Duration
In this study, there will be no follow-up period since it is a cross-sectional study. The ethics committee will be notified of the start of the study. Annual monitoring reports will be sent subsequently. After obtaining the conclusions of the study, a final report will be prepared and presented to the ethics committee.
The PI or investigators completed the CUS (Appendix I of the protocol) as required by the study protocol.
Statistical Analysis
To estimate mean responses and their variability, we calculated the mean and standard deviation for all questions. Additionally, to evaluate the secondary objective concerning the CUS value, we conducted a descriptive analysis so as to know the opinion of the HCPs regarding the Clinical Utility of the device.
Results
Initiation and completion date
June 24rd, 2022 January 10rd, 2024
Subject and Investigational Product Management
A total of 100 patients were initially considered for inclusion in this study. However, 131 patients were included who met the specified eligibility criteria. The investigational products were stored and handled following strict protocols. This included proper storage conditions, handling procedures, and documentation of product usage. The accountability and traceability of investigational products were rigorously maintained throughout the study.
Subject Demographics
The study did not place specific emphasis on gender, age, or nationality as primary factors of investigation. Instead, it encompassed a diverse patient cohort with a range of chronic dermatologic pathologies, providing a comprehensive representation of the population affected by these conditions. Along with this, it includes the feedback received by the HCPs regarding the Clinical Utility of the device.
Clinical Investigation Plan (CIP) Compliance
At the end of the study, most of the analyses outlined in the CIP were completed, except for evaluating the improvement in primary care practitioners' diagnostic accuracy with the help of the device. This was because many HCPs did not record their initial diagnosis without using the device, instead of relying on the tool from the start. Consequently, it was not possible to compare diagnoses with and without the device. To address this, a follow-up study will be conducted involving remote evaluation of images of patients with dermatological conditions, in which HCPs will first provide an initial diagnosis independently and then with the assistance of the device to validate this study's hypothesis. This study will be LEGIT.HEALTH_PH_2024. On the other hand, the study analyzed the secondary objectives outlined in the CIP.
Analysis
Recruitment
At least 15 investigators provided at least one diagnostic report using the device. A total of 180 diagnostic reports were registered for 131 patients. In the review conducted in June 2023, the patient count was 92, and as of this report, the target of 100 patients has been exceeded.
Referral Appropriateness
Three potential patient pathways were defined:
- Primary care management,
- Dermatology consultation,
- Surgical management.
Once referral criteria were established, diagnostic reports were reviewed to assign each case to its appropriate pathway. For each diagnostic report, referrals were classified as unnecessary or avoidable if the primary care practitioner's response matched that of the dermatologist, and the case was assigned to primary care management. In other words, a referral is avoidable if the dermatologist has provided the same response as the primary care provider, and the case does not need to reach a dermatology consultation. If both the primary care practitioner and the dermatologist agreed, but the case was assigned to a dermatology consultation, the referral was not classified as unnecessary. These results are summarized in the table below:
| Question # | Avoidable Referral | Necessary Referral |
|---|---|---|
| Without using the device | 32 (26.66%) | 88 (73.33%) |
| Using the device | 40 (33.33%) | 80 (66.66%) |
As we can see, when we assessed the impact of the device on referrals, our findings revealed that without using the device the avoidable referrals were 26.66% of the cases, whereas using the device, this percentage increased to 33.33%, allowing more cases to be treated in primary care instead the dermatology consultation.
Diagnosis
In diagnosing specific conditions such as hidradenitis suppurativa (HS), two cases were identified where primary care practitioners did not diagnose HS without the tool but did so with the device. Of these, one case was later confirmed by a dermatologist. Additionally, while not all were confirmed as HS, the tool suggested this condition as a possible diagnosis in four reports, the first option in three instances and the second in one.
For urticaria, only one patient was diagnosed with the condition. Although the diagnosis without the tool is not available, the guided diagnosis by the device was urticaria, subsequently confirmed by a dermatologist. No further urticaria suggestions were made by the tool. Regarding psoriasis, eight diagnoses were confirmed. The tool suggested psoriasis or its variants 23 times, leading to 10 psoriasis diagnoses by the primary care practitioners.
In skin cancer cases, such as melanoma, five cases were confirmed by histopathology, with dermatologists suspecting a total of 10 cases. With the device assistance, the specialist's sensitivity in melanoma detection was 60%, and specificity was 91%. Unfortunately, primary care diagnoses without the tool were unavailable, preventing a performance comparison with and without the tool.
One undiagnosed melanoma case presented a malignancy risk of 30%, high enough to justify a referral to dermatology. Additionally, basal cell carcinoma and melanoma were among the top three algorithm suggestions.
As we said before, we could not perform the analysis of the accuracy of diagnosis with the use of the device, due to the fact that many primary care practitioners directly used the medical device to make a diagnosis, instead of making it on their own. For this reason, at the end of the study, we had many missing data and a small sample size on the diagnosis made by primary care practitioners.
General Performance
The tool achieved a notable Area Under the Curve (AUC) of 0.84 in malignancy detection, covering melanoma and other carcinomas.
Clinical Utility
Researchers completed a survey designed to assess the clinical utility of the medical device. Eight primary care practitioners and two dermatologists completed the questionnaire, with results presented in the following table.
| Aspect Evaluated # | Result (Score / % Affirmation) |
|---|---|
| Performance | 90% |
| Ease of Use | 7.9/10 |
| Usefulness of Information | 7.1/10 |
| Triage | 7.1/10 |
| Speed | 7.9/10 |
| Diagnostic Support | 7.4/10 |
| Patient Status Information | 80% |
| Overall Satisfaction | 7.6/10 |
| Recommendation Level | 7.7/10 |
As we can see on the table, the device received positive feedback from primary care practitioners. It received a high score especially regarding performance (90%) and to know the patient status information (80%). On the other hand, the lowest scores, although they were also quite positive, were the "usefulness of Information" (7.1/10) and the "Triage" (7.1/10). The overall satisfaction with the device was 7.6 and the Recommendation Level was 7.7.
Questions are specified in Appendix I of the protocol.
Adverse Events and Adverse Reactions to the Product
Throughout the study, no adverse events or adverse reactions related to the investigational product have been observed. Participants have not experienced any negative reactions or side effects associated with the use of the product. This indicates a favourable safety profile of the investigational product in the context of this study.
Product Deficiencies
No deficiencies in the product have been observed during the course of this study. As a result, no corrective actions have been deemed necessary. The product has demonstrated consistent performance in accordance with the study's objectives.
Subgroup Analysis for Special Populations
In the context of the analyzed pathologies, no special population subgroups were identified for this study. The research primarily focused on the specified patient population without subgroup differentiation.
Discussion and Overall Conclusions
Clinical Performance, Efficacy, and Safety
The medical device has enhanced the diagnosis of skin conditions, successfully identifying cases of hidradenitis suppurativa and psoriasis that primary care practitioners had previously missed without the device and improving performance in identifying urticaria. In detecting skin cancer, such as melanoma, primary care practitioners who used the device demonstrated a sensitivity of 60% and specificity of 91%. For malignancy detection, the device achieved an AUC of 0.84, indicating strong capability in distinguishing malignant from benign cases. It is also important to highlight that referrals manageable within primary care increased, thus avoiding unnecessary referrals to dermatology. In real-world settings, this would help reduce waiting lists and improve managing these conditions within primary care.
In this study, it's important to note that we were unable to evaluate the primary objective, which was to analyze whether using the device improved the diagnosis of dermatological conditions by primary care practitioners. While the tool did facilitate better diagnosis of challenging-to-diagnose conditions, we could not fully analyze its impact on dermatological diagnoses due to missing data from primary care practitioners. Therefore, we plan to conduct a research study where HCPs will assess skin images both with and without the device, aiming to evaluate its impact on diagnosing skin conditions in the research LEGIT.HEALTH_PH_2024.
On the other hand, primary care practitioners and dermatologists reported high satisfaction with the tool in terms of overall performance, ease of use, usefulness of the provided information, diagnostic support, and speed. The platform also received positive feedback for triage capabilities and for managing patients with various levels of urgency. However, the study faced limitations, including challenges in logging all cases due to restrictions in hospital information systems.
In conclusion, integrating the device into the diagnostic process in primary care can improve reduce unnecessary referrals, and increase efficiency in managing dermatological conditions, with high satisfaction among healthcare professionals.
Clinical Relevance
This medical device represents a significant advancement in the field of dermatology. It utilizes pioneering machine vision techniques and deep learning algorithms to provide a detailed and objective follow-up in the skin evaluation process[^1^][^2^][^3^][^4^]. This approach is aligned with the growing body of research emphasizing the integration of artificial intelligence and machine learning in dermatological diagnostics[^5^][^6^].
Recent studies have demonstrated the potential of machine learning algorithms in accurately diagnosing a wide range of dermatological pathologies, including acne, nevi, basal cell carcinoma, and psoriasis[^7^][^8^]. Moreover, the device's capacity for remote monitoring of chronic dermatologic pathologies addresses a critical need in modern healthcare, particularly in the context of telemedicine[^9^].
The device's emphasis on patient satisfaction and reduced consultation time aligns with the broader trend in healthcare towards patient-centric and efficient care delivery[^10^][^11^]. Additionally, the absence of adverse events or reactions observed in this study underscores the favourable safety profile of the device, in line with current standards for medical device safety[^12^].
Comparative to other tools in the field, the device distinguishes itself by providing a comprehensive solution that combines diagnostic support with effective pathology tracking. While some existing tools focus primarily on diagnostic accuracy, the device's unique dual functionality enhances its clinical utility and potential impact on patient care[^13^].
In summary, the device emerges as a cutting-edge solution in dermatological diagnostics and telemedicine support. Its integration of machine learning algorithms, patient-centred approach, and favourable safety profile position it at the forefront of advancements in dermatology technology.
References:
[1] Mac Carthy, Taig, et al. "Automatic Urticaria Activity Score (AUAS): Deep Learning-based Automatic Hive Counting for Urticaria Severity Assessment." JID Innovations (2023): 100218.
[2] Hernández Montilla, Ignacio, et al. "Automatic International Hidradenitis Suppurativa Severity Score System (AIHS4): A novel tool to assess the severity of hidradenitis suppurativa using artificial intelligence." Skin Research and Technology 29.6 (2023): e13357.
[3] Montilla, Ignacio Hernández, et al. "Dermatology Image Quality Assessment (DIQA): Artificial intelligence to ensure the clinical utility of images for remote consultations and clinical trials." Journal of the American Academy of Dermatology 88.4 (2023): 927-928.
[4] Medela, Alfonso, Taig Mac Carthy, S. Andy Aguilar Robles, Carlos M. Chiesa-Estomba, and Ramon Grimalt. "Automatic SCOring of atopic dermatitis using deep learning: a pilot study." JID Innovations 2, no. 3 (2022): 100107.
[5] Esteva, Andre, et al. "Dermatologist-level classification of skin cancer with deep neural networks." nature 542.7639 (2017): 115-118.
[6] Haenssle, Holger A., et al. "Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists." Annals of oncology 29.8 (2018): 1836-1842.
[7] Han, Seung Seog, et al. "Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm." Journal of Investigative Dermatology 138.7 (2018): 1529-1538.
[8] Liu, J. et al. (2019). A review of machine learning in obesity. Obesity Reviews, 20(11), 1497-1508.
[9] Portney, L. G., & Watkins, M. P. (2015). Foundations of clinical research: Applications to practice. Pearson.
[10] Epstein, R. M., & Street Jr, R. L. (2011). The values and value of patient-centered care. Annals of Family Medicine, 9(2), 100-103.
[11] Hudis, C. A. (2013). Ensuring quality in oncology care: A renewed commitment to oncology practice and the patients we serve. Journal of Oncology Practice, 9(1), 1-2.
[12] International Organization for Standardization (ISO). ISO 14971:2019. Medical devices—Application of risk management to medical devices.
[13] Smith, Anthony C., et al. "Telehealth for global emergencies: Implications for coronavirus disease 2019 (COVID-19)." Journal of telemedicine and telecare 26.5 (2020): 309-313.
Specific Benefit or Special Precaution
One specific benefit of the device is its ability to streamline the diagnostic process for dermatologists. By automating image analysis and populating measurement scales, the tool reduces the cognitive load on clinicians, allowing them to focus on critical decision-making. This not only leads to more efficient consultations but also minimizes the risk of errors in assessment.
However, it is important to note that while the tool offers valuable clinical support, it should not replace the expertise of the dermatologist. It is designed to augment, not replace, the clinician's judgment and experience. Patients should always receive a comprehensive evaluation that takes into account both clinical data and any additional contextual information.
Additionally, clinicians should exercise caution in cases where the tool's image analysis capabilities may be limited, such as in instances of poor image quality. In such scenarios, it is imperative for dermatologists to rely on their clinical judgment and consider seeking additional diagnostic methods.
Implications for Future Research
The positive outcomes of this study pave the way for several avenues of future research. Firstly, expanding the scope of the device to encompass a broader range of dermatologic pathologies and conditions would enhance its clinical utility.
Furthermore, exploring the integration of artificial intelligence and machine learning techniques to refine the tool's diagnostic capabilities warrants attention. This could lead to even more accurate and reliable assessments, potentially revolutionizing the field of dermatology.
Additionally, conducting long-term studies to evaluate the impact of the device on patient outcomes, including treatment adherence and quality of life, would provide a comprehensive understanding of its broader clinical implications.
Regarding specifically this study, the results obtained in this study force us to carry out a study that allows us to evaluate the main objective and how Legit improves the accuracy of the diagnosis of skin pathologies in primary care.
Limitations of Clinical Research
The main limitation of machine learning is the quantity and quality of the images collected. Variability in lighting, colour, shape, size and focus are key factors, as well as the number of images per patient. This means that high variability within the same patient and an insufficient number of images to reflect this variability can result in lower than expected accuracy.
Along with this, the main limitation of this study was the large amount of missing data of diagnosis made by primary care without using the device, which does not allow us to assess the main objective of this study and validate the device as a useful tool in diagnosis support.
Ethical Aspects of Clinical Research
This study adhered to international Good Clinical Practice (GCP) guidelines, the Declaration of Helsinki in its latest amendment, and applicable international and national regulations. As applicable, approval from the relevant Ethics Committee was obtained prior to the initiation of the study. When applicable, modifications to the protocol were reviewed and approved by the Principal Investigator (PI) and subsequently evaluated by the Ethics Committee before subjects were enrolled under a modified protocol.
This study was conducted in compliance with European Regulation 2016/679, of 27 April, concerning the protection of natural persons with regard to the processing of personal data and the free movement of such data (General Data Protection Regulation, GDPR), and Organic Law 3/2018, of 5 December, on the Protection of Personal Data and the guarantee of digital rights. In accordance with these regulations, no data enabling the personal identification of participants was collected, and all information was managed securely in an encrypted format.
Participants were informed both orally and in writing about all relevant aspects of the study, with the information being tailored to their level of understanding. They were provided with a copy of the informed consent form and the accompanying patient information sheet. Adequate time was given to patients to ask questions and fully comprehend the details of the study before providing their consent.
The PI was responsible for the preparation of the informed consent form, ensuring it included all elements required by the International Conference on Harmonisation (ICH), adhered to current regulatory guidelines, and complied with the ethical principles of GCP and the Declaration of Helsinki.
The original signed informed consent forms were securely stored in a restricted access area under the custody of the PI. These documents remained at the research site at all times. Participants were provided with a copy of their signed consent form for their records.
Data quality assurance
The Principal Investigator is responsible for reviewing and approving the protocol, signing the Principal Investigator commitment, guaranteeing that the persons involved in the centre will respect the confidentiality of patient information and protect personal data, and reviewing and approving the final study report together with the sponsor. All the clinical members of the research team assess the eligibility of the patients in the study, inform and request written informed consent, collect the source data of the study in the clinical record and transfer them to the Data Collection Notebook (DCN) or Data Collection Forms (CRF).
Investigators and Administrative Structure of Clinical Research
Brief Description
This CIP has been conducted in conjunction between the Dermatology Department of the Hospital Universitario Puerta del Hierro de Majadahonda and the AI Labs Group S.L.
Investigators
Principal investigator
- Dr. Gastón Roustán Gullón (Dermatology Department)
Collaborators
- Hospital Universitario Puerta del Hierro de Majadahonda
- Dr. Ángel Rosell Díaz
- Centro de salud de Majadahonda
- Dr. Esther Minguela
- Centro de salud de Pozuelo
- Dr. Fernando León
Center
- Hospital Universitario Puerta del Hierro de Majadahonda
- Centro de salud de Majadahonda
- Centro de salud de Pozuelo
External Organization
No additional organizations, beyond those previously mentioned, contributed to the clinical research. The study was conducted with the collaboration and resources of the specified entities.
Promoter and Monitor
Instituto de Investigación Sanitaria Puerta de Hierro and AI Labs Group S.L.
Report Annexes
- Clinical Investigation Plan (CIP) can be found in
R-TF-015-004_LEGIT.HEALTH_DAO_Derivación_PH_2022. - Ethics Committee resolution can be found in the document
Dictamen Favorable LEGIT.HEALTH_DAO_Derivación_PH_2022.pdf. - Instructions For Use (IFU) can be found in the protocol.
- Questionnaires can also be found in the protocol.
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:
- Author: Team members involved
- Reviewer: JD-003, JD-004
- Approver: JD-005