R-TF-015-006 Clinical investigation report LEGIT.HEALTH_BI_2024
Research Title
Non-Invasive Prospective Pilot in a Live Environment for the improvement of the diagnosis of Generalized Pustular Psoriasis
Product Identification
| Information | |
|---|---|
| Device name | Legit.Health Plus (hereinafter, the device) |
| Model and type | NA |
| Version | 1.0.0.0 |
| Basic UDI-DI | 8437025550LegitCADx6X |
| Certificate number (if available) | MDR 792790 |
| EMDN code(s) | Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software) |
| GMDN code | 65975 |
| Class | Class IIb |
| Classification rule | Rule 11 |
| Novel product (True/False) | FALSE |
| Novel related clinical procedure (True/False) | FALSE |
| SRN | ES-MF-000025345 |
Promoter Identification and Contact
| Manufacturer data | |
|---|---|
| Legal manufacturer name | AI Labs Group S.L. |
| Address | Street Gran Vía 1, BAT Tower, 48001, Bilbao, Bizkaia (Spain) |
| SRN | ES-MF-000025345 |
| Person responsible for regulatory compliance | Alfonso Medela, María Diez, Giulia Foglia |
| office@legit.health | |
| Phone | +34 638127476 |
| Trademark | Legit.Health |
Identification of sponsors
- Boehringer Ingelheim
Clinical Investigation Plan (CIP) Identification
- Title: Non-Invasive Prospective Pilot in a Live Environment for the improvement of the diagnosis of Generalized Pustular Psoriasis.
- Protocol code: LEGIT.HEALTH_BI_2024.
- Study design: Prospective observational analytical and cross-sectional study.
- Product under investigation: Legit.Health Plus.
- Version and date: Version 1.0, 2024-06-01.
Public Access Database
Please note that the database used in this study is not publicly accessible due to privacy and confidentiality considerations.
Research Team
Principal investigator
- Dr. Antonio Martorell Calatayud
Collaborators
- Medical staff
- Dr. Mari Carmen Galindo
- Dr. Paco García Tolosa
- Dr. Laura Yuste Hidalgo
- Dr. Nuria Comabella
- Dr. Marta Vázquez
- Dr. David Palacios
- Dr. Norma Alejandra Doria Carlin
- Dr. Francisco José Esteban González
- Dr. Alfonso José Valcarce Leonisio
- Dr. José Antonio Arjona Sevilla
- Dr. Javier Melgosa
- Dr. Manuel Ballesteros Redondo
- Dr. Esmeralda Silva
- Dr. Ana Llull Ramos
- Dr. Angela Patricia Guzmán
- AI Labs Group S.L.
- Mr. Alfonso Medela
- Dr. Alberto Sabater
- Mrs. Alba Rodríguez
Center
- This study was conducted remotely and sending the images to the participating dermatologists.
Compliance Statement
The clinical investigation was perforfed according to the Clinical Investigation Plan (CIP) and other applicable guidances and regulations. This includes compliance with:
- Harmonized standard
UNE-EN ISO 14155:2021 Regulation (EU) 2017/745 on medical devices (MDR)- Harmonized standard
UNE-EN ISO 13485:2016s Regulation (EU) 2016/679(GDPR).- Spanish
Organic Law 3/2018on the Protection of Personal Data and guarantee of digital rights`.
All data processing within the device is carried out in accordance with the highest standards of data protection and privacy. Patient information is managed in an encrypted manner to ensure confidentiality and security.
The research team assumes the role of Data Controller, responsible for the collection and management of study data. Legit.Health acts as the Data Processor and is not involved in the processing of patient data.
The storage and transfer of data comply with European data protection regulations. At the conclusion of the study, all information stored in the device will be permanently and securely deleted.
The device employs robust technical and organizational security measures to safeguard personal data against unauthorized access, alteration, loss, or processing.
Report Date
September 15, 2024
Report Author(s)
Table of contents
- Research Title
- Product Identification
- Promoter Identification and Contact
- Identification of sponsors
- Clinical Investigation Plan (CIP) Identification
- Public Access Database
- Research Team
- Compliance Statement
- Report Date
- Report Author(s)
- Abbreviations and Definitions
- Summary
- Introduction
- Material and methods
- Results
- Discussion and Overall Conclusions
- Investigators and Administrative Structure of Clinical Research
- Report Annexes
Abbreviations and Definitions
- CAD: Computer-Aided Diagnosis
- CIP: Clinical Investigation Plan
- CUS: Clinical Utility Questionnaire
- SUS: System Usability Scale
- GCP: Standards of Good Clinical Practice
- ICH: International Conference of Harmonization
- PI: Principal Investigator
- DLQI: Dermatology Quality of Life Index
- ICH: International Conference of Harmonization
- AUC: Area Under the ROC Curve
Summary
Title
Non-Invasive Prospective Pilot in a Live Environment for the improvement of the diagnosis of Generalized Pustular Psoriasis.
Introduction
Dermatological conditions represent a significant portion of primary care consultations, constituting approximately 5% of all visits. In these consultations can be discrepancies between primary care physicians and dermatologists, specially in rare and severe conditions, such as Generalized Pustular Psoriasis (GPP) or Hidradenitis Suppurativa (HS), which can lead to misdiagnoses. This investigation pretends to assess whether the use of the Legit.Health medical device can improve the diagnosis of complicated pathologies with a low incidence but a great impact on the patient's quality of life, such as GPP or also HS. 15 HCPs will be recruited and each one of them will be presented with 100 images of patients with GPP, HS or pathologies that can be consufed with these. This research follows rigorous ethical standards, and adherence to regulatory guidelines, this research holds significant potential in revolutionizing dermatological diagnostics.
Objectives
Primary objective
To validate that the information provided by device increases the true accuracy of healthcare professionals (HCPs) in the diagnosis of generalized pustular psoriasis (GPP).
Secondary objectives
To validate that the information provided by device increases the true accuracy of healthcare professionals (HCPs) in the diagnosis of other dermatological skin conditions, such as hidradenitis suppurativa.
Population
In this study the population will consist of primary care physicians and dermatologists. A minimum of 15 physicians will be selected.
Design and methods
Design
The study proceeded as follows:
Healthcare practicioners recruitment and image presentation
We developed a website to conduct the experiment. Participants, including dermatologists and primary care physicians, were required to log in to the website. They were then presented with a series of two questions structured as follows:
- Based on the provided image, what diagnosis do you consider most appropriate? This question was accompanied by anamnesis inquiries regarding allergies, ongoing treatments, and other relevant medical history, such as systemic symptoms that could be related to conditions like GPP.
- Considering both the image and the analysis provided by the AI, what diagnosis do you deem most appropriate? In this instance, the same information from question 1 was supplemented with the top 5 diagnoses and their respective confidence levels, calculated by Legit.Health's diagnosis support algorithm based on the image.
Each participant was presented with a total of 100 cases or images to review. These images had been previously confirmed by dermatologists and were extracted from dermatology atlases or from previous researches of Boehringer Ingelheim. The conditions were distributed as follows:
| Condition | ICD-11 code | Number of images |
|---|---|---|
| Generalised pustular psoriasis | EA90.40 | 10 |
| Eczematous dermatitis | EA89 | 10 |
| Acute generalized exanthematous pustulosis | EH67.0 | 5 |
| Acne | ED80 | 5 |
| Acné conglobata | ED80.41 | 10 |
| Severe inflammatory acne | ED80.4 | 5 |
| Seborrheic keratosis | 2F21.0 | 5 |
| Seborrheic dermatitis | EA81 | 5 |
| Palmoplantar pustulosis | EA90.42 | 5 |
| Plaque psoriasis | EA90.0 | 5 |
| Pemphigus vulgaris | EB40.0 | 5 |
| Impetigo | 1B72 | 10 |
| Hidradenitis suppurativa | ED92.0 | 10 |
| Subcorneal pustular dermatosis | EB2Y | 5 |
| Tinea corporis | 1F28.Y | 5 |
The list of conditions includes the target condition GPP, as well as similar conditions such as subcorneal pustular dermatosis and palmoplantar pustulosis. It also covers common conditions treated in primary care that can be confused with GPP, such as dermatitis. Pigmented lesions are not relevant to this experiment.
All this information was recorded in a database and exported to a CSV file, which was subsequently used for further analysis. The analysis was conducted using the Python programming language and statistical libraries such as numpy and pandas. Statistical measures, including the P-Value, were calculated to either accept or reject the hypothesis. Additionally, metrics were calculated based on pathology and medical specialization.
Number of subjects
A total of 15 HCPs (11 primary care physicians and 4 dermatologists) were recruited in this study.
Initiation Date
June 01rd, 2024
Completion Date
September 15rd, 2024
Duration
The study spanned a total duration of 4 months, encompassing the time needed for tasks such as the creation of the website, database closure and editing, data analysis, and the preparation of the final study report after the recruitment of the last subject.
Methods
The study employed a prospective observational analytical design to evaluate whether the use of Legit.Health improved the accuracy on the diagnosis of pathologies that are difficult to diagnoses, such as GPP or HS. This investigation encompassed a diverse cohort of 15 HCPs (dermatologists and primary care physicians). Data collection included the accuracy on the diagnosis of different pathologies with and without the use of Legit.Health and specially for the target pathologies, GPP and HS. The study adhered to strict ethical guidelines, ensuring patient confidentiality and compliance with international standards.
Results
For this study, 15 HCPs (11 primary care physicians and 4 dermatologists) were included. Among them, 9 completed the entire process, while the remaining reviewed a partial number of images, specifically 99, 93, 93, 80, 77 and 68 respectively.
When HCPs did not use Legit.Health they showed an accuracy of 47.91% on their diagnosis. On the other hand, when they integrated Legit.Health on their diagnosis, the accuracy increased to 62.81%. This increase was higher for primary care physicians, who showed an accuracy of 44.29% rising to 61.71%. Meanwhile for dermatologists. The diagnostic accuracy increased from 57.25% to 65.65%.
In relation with the targeted pathologies, for GPP the use of Legit.Health improved the doctor's diagnoses in 32.32% of cases. For HS, Legit.Health improved doctor's diagnoses in 10.11% of cases.
Conclusions
Legit.Health's medical device significantly enhanced the diagnostic accuracy for generalized pustular psoriasis (GPP) among healthcare professionals, doubling the diagnosis rate and increasing it by 120% in primary care. On average, diagnostic accuracy improved by 31%, from 47.91% to 62.81%, with primary care doctors experiencing a 40% increase in correct diagnoses. Similar positive results were observed in the diagnosis of other conditions like hidradenitis suppurativa and palmoplantar pustulosis, with a notable 146% improvement in primary care diagnoses for the latter.
In the dermatologists' data, results were not statistically significant due to a limited participant pool and their expertise level, especially in treating hidradenitis suppurativa. This suggests that the device may be particularly beneficial for primary care settings, where dermatological expertise is lower, and in diagnosing rarer conditions that general dermatologists may encounter less frequently.
Introduction
Dermatological conditions represent a significant portion of primary care consultations, constituting approximately 5% of all visits. However, discrepancies between diagnoses made by primary care physicians and dermatologists remain substantial, with concordance rates between 57% and 65.52%. This gap in expertise often leads to misdiagnoses, incorrect referrals, and delays in appropriate treatment, particularly in rare and severe conditions like generalized pustular psoriasis (GPP) and hidradenitis suppurativa (HS). The limited availability of dermatologists, especially in rural areas, further complicates patient care, underscoring the need for innovative solutions to optimize resource allocation and improve diagnostic accuracy.
Teledermatology has shown promise in reducing the pressure on in-person consultations by enabling remote assessments. However, the use of artificial intelligence (AI) presents a transformative opportunity to enhance the diagnostic capabilities of primary care physicians. Legit.Health, an AI-based medical device, has already been validated in the diagnosis of skin conditions and offers advanced tools, such as the automatic scoring of GPP severity using the AGPPGA system. This pilot study aims to evaluate whether the use of the Legit.Health medical device can improve the diagnosis of complicated pathologies with a low incidence but a great impact on the patient's quality of life, such as GPP or also HS.
Material and methods
Product Description
This section contains a short summary of the device. A complete description of the intended purpose, including device description, can be found in the record Legit.Health Plus description and specifications.
Product description
The device is computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organisations to gather data and improve their workflows.
The generated data is intended to aid healthcare practitioners and organizations in their clinical decision-making process, thus enhancing the efficiency and accuracy of care delivery.
The device should never be used to confirm a clinical diagnosis. On the contrary, its result is one element of the overall clinical assessment. Indeed, the device is designed to be used when a healthcare practitioner chooses to obtain additional information to consider a decision.
Intended purpose
The device is a computational software-only medical device intended to support health care providers in the assessment of skin structures, enhancing efficiency and accuracy of care delivery, by providing:
- quantification of intensity, count, extent of visible clinical signs
- interpretative distribution representation of possible International Classification of Diseases (ICD) classes.
Intended previous uses
No specific intended use was designated in prior stages of development.
Product changes during clinical research
The device maintained a consistent performance and features throughout the entire clinical research process. No alterations or modifications were made during this period.
Clinical Investigation Plan
Objectives
This study aims to validate that the information provided by device increases the true accuracy of healthcare professionals (HCPs) in the diagnosis of generalized pustular psoriasis (GPP).
Design (type of research, assessment criteria, methods, active group, and control group)
This is a prospective observational and cross-sectional study. The study does not involve an active or control group, as the physicians will be their own control group, firstly without using Legit.Health and after making their diagnosis, using Legit.Health to assess if they want to change their diagnosis or keep it. The assessment criteria include the assessment of different images with pathologies such as GPP or HS and their diagnosis. The study employs a variety of methods, including data collection through websites or photograph analysis.
Ethical considerations
The conduct of this study adhered to international Good Clinical Practice (GCP) guidelines, the Declaration of Helsinki in its latest amendment, and applicable international and national regulations. Approval from the relevant Ethics Committee was obtained prior to the initiation of the study. Any modifications to the protocol were reviewed and approved by the Principal Investigator (PI) and subsequently evaluated by the Ethics Committee before subjects were enrolled under a modified protocol.
This study was conducted in compliance with European Regulation 2016/679, of 27 April, concerning the protection of natural persons with regard to the processing of personal data and the free movement of such data (General Data Protection Regulation, GDPR), and Organic Law 3/2018, of 5 December, on the Protection of Personal Data and the guarantee of digital rights. In accordance with these regulations, no data enabling the personal identification of participants was collected, and all information was managed securely in an encrypted format.
Patients were informed both orally and in writing about all relevant aspects of the study, with the information being tailored to their level of understanding. They were provided with a copy of the informed consent form and the accompanying patient information sheet. Adequate time was given to patients to ask questions and fully comprehend the details of the study before providing their consent.
The Principal Investigator was responsible for the preparation of the informed consent form, ensuring it included all elements required by the International Conference on Harmonisation (ICH), adhered to current regulatory guidelines, and complied with the ethical principles of GCP and the Declaration of Helsinki.
The original signed informed consent forms were securely stored in a restricted access area under the custody of the PI. These documents remained at the research site at all times. Patients were provided with a copy of their signed consent form for their records.
Data Quality Assurance
The Principal Investigator is responsible for reviewing and approving the protocol, signing the Principal Investigator commitment, guaranteeing that the persons involved in the centre will respect the confidentiality of patient information and protect personal data, and reviewing and approving the final study report together with the sponsor. All the clinical members of the research team assess the eligibility of the patients in the study, inform and request written informed consent, collect the source data of the study in the clinical record and transfer them to the Data Collection Notebook (DCN) or Data Collection Forms (CRF).
Subject Population
This study enrolled both primary care physicians and dermatologists. At the end of the study, 15 HCPs were enrolled. These doctors should evaluate 100 images of different skin pathologies and diagonse them.
Inclusion Criteria
- Board-certified primary care physicians and dermatologists, regardless of their professional experience.
- Good quality images of patients with GPP.
- Good quality images of patients with HS.
- Good quality images of patients with pathologies that can be confused with GPP or HS leading to a wrong diagnosis.
Exclusion Criteria
- Images of patients with pathologies different from GPP or HS that can be easily identified.
Statistical Analysis
For the purpose of estimating the correlation between diagnoses with and without using Legit.Health, we analyzed the concordance between diagnoses for both primary care physicians and dermatologists. We analyzed if Legit.Health reinforced the diagnosis of the physicians (after observing the results of Legit.Health, the doctor maintains the diagnosis when his answer matches that of the solution), if Legit.Health improved the practicioner's diagnosis (the doctor changes the diagnosis when his answer does not match that of the solution), if it has no impact (The doctor does not change the diagnosis even though his answer does not match that of the solution) and if it has a negative impact (The doctor changes the diagnosis for an answer that does not match that of the solution).
Results
Initiation and Completion Date
June 06rd, 2024 September 15rd, 2024
Subject and Investigational Product Management
A total of 15 HCPs were recruited for this study, 11 primary care physicians and 4 dermatologists. Each physician was presented with 100 images in order to review them and make a diagnosis of the pathology. This meticulous selection process ensures the integrity and validity of the study's findings. The investigational products were stored and handled following strict protocols. This included proper storage conditions, handling procedures, and documentation of product usage. The accountability and traceability of investigational products were rigorously maintained throughout the study.
Subject Demographics
The study did not place specific emphasis on gender, age, or nationality as primary factors of investigation. Instead, it encompassed a diverse physician cohort with different specialties.
Clinical Investigation Plan (CIP) Compliance
The study adhered to all aspects outlined in the CIP. This ensured that the research was conducted in accordance with established protocols, procedures, and ethical standards. Any deviations from the CIP were duly documented and appropriately addressed. The compliance with the CIP was rigorously monitored throughout the duration of the study to uphold the integrity and validity of the research findings.
Analysis
Primary Analyses
Diagnosis support
In this study, a total of 15 healthcare professionals (HCPs) participated, comprising 11 primary care doctors and 4 dermatologists. Among them, 9 completed the entire process, while the remaining reviewed a partial number of images, specifically 99, 93, 93, 80, 77 and 68 respectively.
We conducted a McNemar test in order to analyze if the information provided by Legit.Health impacts on the healthcare professionals diagnostics'. Overall, the HCPs demonstrated an accuracy of 47.91%, which notably increased to 62.81% with the integration of Legit.Health. Our analysis, supported by a very low p-value, revealed the following key findings:
- Legit.Health reinforces practitioners' diagnostics in 46.13% of the cases
- Legit.Health improves practitioners' diagnostics in 16.61% of the cases
- Legit.Health does not have any impact on practitioners' diagnostics in 35.42% of the cases
- Legit.Health has a negative impact on practitioners' diagnostics in 1.77% of the cases
When focusing on primary care physicians, the disparity was even more pronounced, with an initial accuracy of 44.29% rising to 61.71% with Legit.Health. Consequently:
- Legit.Health reinforces practitioners' diagnostics in 42.22% of the cases
- Legit.Health improves practitioners' diagnostics in 19.39% of the cases
- Legit.Health does not have any impact on practitioners' diagnostics in 36.22% of the cases
- Legit.Health has a negative impact on practitioners' diagnostics in 2.07% of the cases
For dermatologists, the diagnostic accuracy increased from 57.25% to 65.65%:
- Legit.Health reinforces practitioners' diagnostics in 56.23% of the cases
- Legit.Health improves practitioners' diagnostics in 9.41% of the cases
- Legit.Health does not have any impact on practitioners' diagnostics in 33.33% of the cases
- Legit.Health has a negative impact on practitioners' diagnostics in 1.02% of the cases
The results of diagnostic accuracy are summarized in the table below:
| HCP | Accuracy (%) | Accuracy with Legit.Health (%) | Difference (%) |
|---|---|---|---|
| All specialties | 47.91 | 62.81 | 14.90 |
| Primary care | 44.29 | 61.71 | 17.42 |
| Dermatologist | 57.25 | 65.65 | 8.40 |
An analysis by pathology identified significant impacts for certain conditions, as detailed in the table below:
| Condition | Accuracy (%) | Accuracy with Legit.Health (%) | Difference (%) | p-value |
|---|---|---|---|---|
| Generalised pustular psoriasis | 23.70 | 46.67 | 22.97 | 0.00001 |
| Eczematous dermatitis | 71.34 | 73.17 | 1.83 | 0.62906 |
| Acute generalized exanthematous pustulosis | 5.00 | 5.00 | 0.00 | 1.00000 |
| Acne | 37.50 | 54.69 | 17.19 | 0.00739 |
| Acné conglobata | 18.40 | 37.60 | 19.20 | 0.00000 |
| Severe inflammatory acne | 10.61 | 43.94 | 33.33 | 0.0000 |
| Seborrheic keratosis | 94.67 | 96.00 | 1.33 | 1.00000 |
| Seborrheic dermatitis | 75.34 | 90.41 | 15.07 | 0.00098 |
| Palmoplantar pustulosis | 45.31 | 79.69 | 34.38 | 0.00002 |
| Plaque psoriasis | 91.89 | 97.30 | 5.41 | 0.12500 |
| Pemphigus vulgaris | 28.77 | 56.16 | 27.39 | 0.0000 |
| Impetigo | 57.43 | 75.68 | 18.25 | 0.0000 |
| Hidradenitis suppurativa | 85.48 | 93.60 | 8.12 | 0.00195 |
| Subcorneal pustular dermatosis | 2.67 | 2.67 | 0.00 | 1.00000 |
| Tinea corporis | 35.96 | 62.50 | 26.54 | 0.00004 |
We separated the results per pathology into two tables, one for primary care doctors and another for dermatologists.
Primary care doctors
| Condition | Accuracy (%) | Accuracy with Legit.Health (%) |
|---|---|---|
| Generalised pustular psoriasis | 20.20 | 44.44 |
| Eczematous dermatitis | 68.33 | 70.83 |
| Acute generalized exanthematous pustulosis | 0.00 | 0.00 |
| Acne | 36.36 | 59.09 |
| Acné conglobata | 21.18 | 47.06 |
| Severe inflammatory acne | 10.87 | 50.00 |
| Seborrheic keratosis | 92.73 | 94.55 |
| Seborrheic dermatitis | 69.81 | 88.68 |
| Palmoplantar pustulosis | 32.61 | 80.43 |
| Plaque psoriasis | 88.89 | 96.30 |
| Pemphigus vulgaris | 22.64 | 43.40 |
| Impetigo | 50.00 | 71.30 |
| Hidradenitis suppurativa | 82.02 | 92.22 |
| Subcorneal pustular dermatosis | 0.00 | 0.00 |
| Tinea corporis | 29.23 | 59.38 |
Dermatologists
Because of the quantity of images per pathology and the total number of dermatologists involved, the evidence is inconclusive and may be biased.
| Condition | Accuracy (%) | Accuracy with Legit.Health (%) |
|---|---|---|
| Generalised pustular psoriasis | 33.33 | 52.78 |
| Eczematous dermatitis | 79.55 | 79.55 |
| Acute generalized exanthematous pustulosis | 18.75 | 18.75 |
| Acne | 40.00 | 45.00 |
| Acné conglobata | 12.50 | 17.50 |
| Severe inflammatory acne | 10.00 | 30.00 |
| Seborrheic keratosis | 100.00 | 100.00 |
| Seborrheic dermatitis | 90.00 | 95.00 |
| Palmoplantar pustulosis | 77.78 | 77.78 |
| Plaque psoriasis | 100.00 | 100.00 |
| Pemphigus vulgaris | 45.00 | 90.00 |
| Impetigo | 77.50 | 87.50 |
| Hidradenitis suppurativa | 94.29 | 97.14 |
| Subcorneal pustular dermatosis | 10.00 | 10.00 |
| Tinea corporis | 54.17 | 70.83 |
Target pathologies
If we focus on primary care doctors and target conditions like generalized pustular psoriasis (GPP), hidradenitis suppurativa, and palmoplantar pustulosis, we found the following results.
For GPP, with a p-value of 0.00015:
- Legit.Health supports doctors' diagnoses in 12.12% of cases.
- Legit.Health improves doctors' diagnoses in 32.32% of cases.
- Legit.Health has no impact on doctors' diagnoses in 47.47% of cases.
- Legit.Health negatively affects doctors' diagnoses in 8.08% of cases.
For hidradenitis suppurativa, with a p-value of 0.00391:
- Legit.Health supports doctors' diagnoses in 82.02% of cases.
- Legit.Health improves doctors' diagnoses in 10.11% of cases.
- Legit.Health has no impact on doctors' diagnoses in 7.87% of cases.
- Legit.Health does not affect negatively doctors' diagnoses.
For palmoplantar pustulosis, with a p-value of 0:
- Legit.Health supports doctors' diagnoses in 32.61% of cases.
- Legit.Health improves doctors' diagnoses in 47.83% of cases.
- Legit.Health has no impact on doctors' diagnoses in 19.57% of cases.
- Legit.Health does not affect negatively doctors' diagnoses.
Adverse Events and Adverse Reactions to the Product
Throughout the study, no adverse events or adverse reactions related to the investigational product have been observed. Participants have not experienced any negative reactions or side effects associated with the use of the product. This indicates a favorable safety profile of the investigational product in the context of this study.
Product Deficiencies
No deficiencies in the product have been observed during the course of this study. As a result, no corrective actions have been deemed necessary. The product has demonstrated consistent performance in accordance with the study's objectives.
Subgroup Analysis for Special Populations
In the context of the analyzed pathologies, no special population subgroups were identified for this study. The research primarily focused on the specified physicians population without subgroup differentiation.
Discussion and Overall Conclusions
Conclusions
Legit.Health significantly improved healthcare professionals' diagnosis of generalized pustular psoriasis (GPP), doubling the diagnosis rate with the use of the medical device. In primary care, this effect was even greater, with a 120% increase in GPP diagnoses.
On average, diagnostic accuracy increased from 47.91% to 62.81%, a relative increase of 31%. For primary care doctors, the improvement was even more pronounced, with a 40% relative increase in correct diagnoses.
For other conditions like hidradenitis suppurativa and palmoplantar pustulosis, the results were also impressive. Primary care doctors correctly diagnosed 12.43% more cases, while dermatologists showed similar improvements with the device. For palmoplantar pustulosis, primary care doctors demonstrated an outstanding 146% increase, with dermatologists maintaining their performance.
Regarding the dermatologist's data, the results were not statistically significant at the pathology level due to two factors we identified. First, the lower number of participants. We had only four dermatologists, while primary care had eleven participants. This is not a design flaw because the focus was on primary care, but it is a limitation when conducting a detailed analysis. Additionally, the dermatologists had a high level of expertise, especially in hidradenitis suppurativa (HS). This, combined with the relatively average complexity of the HS cases in the experiment, may explain why the tool was not useful for dermatologists in the case of HS. However, this does not mean it is not useful for other dermatologists or dermatologists in general.
In summary, Legit.Health had a substantial impact on the diagnostic capabilities of healthcare professionals, particularly primary care doctors who have less specialization in dermatology. It also significantly benefited dermatologists when diagnosing rare conditions like GPP, which they encounter infrequently in their practice.
Implications for Future Research
The positive outcomes of this study pave the way for several avenues of future research. Firstly, helping to improve the diagnosis of difficult-to-diagnose pathologies such as GPP or HS, which significantly impacts the quality of life of patients who suffer from them.
On the other hand, exploring the integration of artificial intelligence and machine learning techniques to refine the tool's diagnostic capabilities warrants attention. This could lead to even more accurate and reliable assessments, potentially revolutionizing the field of dermatology.
Additionally, conducting long-term studies to evaluate the impact of the device on patient outcomes, including treatment adherence and quality of life, would provide a comprehensive understanding of its broader clinical implications.
Limitations of Clinical Research
The main limitations of the pilot included several factors that may influence the perception and effectiveness of the AI-based device. Firstly, the acceptance and trust of healthcare professionals in these emerging technologies can vary significantly. The device's effectiveness may be compromised if users are not fully convinced of its accuracy or usefulness, thereby affecting the overall perception of its performance.
Additionally, image quality is crucial for the device's performance. Issues such as low-quality photographs, errors in cropping lesions, or variations in lighting and focus could deteriorate the quality of the data received by the system, which may negatively influence the evaluation and perception of its effectiveness by the researchers.
Variability in image conditions is also an important aspect to consider. Differences in lighting, color, shape, size, and focus of the images, along with the number of images available for each patient, can affect the accuracy of the results. High variability in images of the same patient or an insufficient number of representative images can lead to a decrease in the expected diagnostic accuracy of the device.
Additionally, the consistency of investigators in using Legit.Health is crucial. Variations in how diligently investigators use the device can impact the pilot's findings. If the investigators are not consistent in their use of the device, it can lead to unreliable results and affect the overall assessment of its efficacy.
Another limitation could be the Hawthorne effect, where pilot subjects may change their behavior simply because they know they are being observed. This awareness can influence their decisions and actions within the pilot, potentially skewing the results and not accurately reflecting how the device would be used in a non-study environment.
Ethical Aspects of Clinical Research
The conduct of this study adheres to international Good Clinical Practice standards and is in compliance with the Declaration of Helsinki in its latest active amendment. It also conforms to international and national rules and regulations.
The study did not require the approvation by an Ethics Committee due to its observational character and not allowing patients' identification.
The study has been conducted in accordance with European Regulation 2016/679, of 27 April, on the protection of natural persons with regard to the processing of personal data and the free movement of such data. Additionally, it adhere to the Spanish Organic Law 3/2018, of 5 December, on the Protection of Personal Data and guarantee of digital rights with regard to data processing. No data that allows the personal identification of subjects has been included, and all information has been managed in an encrypted manner.
Physicians have received comprehensive oral and written information about the study, tailored to their level of understanding. The main investigator has ensured that the pariticipants had sufficient time to ask questions and clarify any doubts regarding the study details.
The original signed informed consent form has been securely stored in a restricted access area under the custody of the Principal Investigator. A copy of the original signed consent form has been given to the patient.
The Data Controller for this study is the research team. Legit.Health, the Data Processor, is not responsible for the processing of the data included in the Software or its users. The storage and handling of data and photographs is aligned with the European Regulation 2016/679 of 27 April on the protection of natural persons with regard to the processing of personal data and the free movement of such data and the Organic Law 3/2018 of 5 December on the Protection of Personal Data and guarantee of digital rights. At the conclusion of the study, all information stored in the device will be completely and permanently deleted.
The device complies with current legislation on the protection and confidentiality of personal data. Appropriate technical and organizational security measures are in place to ensure the security of personal data and prevent its alteration, loss, unauthorized processing or access.
Investigators and Administrative Structure of Clinical Research
Brief Description
This CIP has been conducted in conjunction between the participating medical staff, AI Labs Group S.L. (Legit.Health) and Boehringer Ingelheim.
Investigators
Principal investigator
- Dr. Antonio Martorell Calatayud
Collaborators
- Medical staff
- Dr. Mari Carmen Galindo
- Dr. Paco García Tolosa
- Dr. Laura Yuste Hidalgo
- Dr. Nuria Comabella
- Dr. Marta Vázquez
- Dr. David Palacios
- Dr. Norma Alejandra Doria Carlin
- Dr. Francisco José Esteban González
- Dr. Alfonso José Valcarce Leonisio
- Dr. José Antonio Arjona Sevilla
- Dr. Javier Melgosa
- Dr. Manuel Ballesteros Redondo
- Dr. Esmeralda Silva
- Dr. Ana Llull Ramos
- Dr. Angela Patricia Guzmán
- AI Labs Group S.L.
- Mr. Alfonso Medela
- Dr. Alberto Sabater
- Mrs. Alba Rodríguez
External Organization
No additional organizations, beyond those previously mentioned, contributed to the clinical research. The study was conducted with the collaboration and resources of the specified entities.
Promoter and Monitor
Boehringer Ingelheim
Report Annexes
- Instructions For Use (IFU) can be found in the protocol.
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:
- Author: Team members involved
- Reviewer: JD-003, JD-004
- Approver: JD-001