R-TF-015-006 Clinical investigation report LEGIT_COVIDX_EVCDAO_2022
Research Title
Clinical Validation of a Computer-Aided Diagnosis (CAD) System Utilizing Artificial Intelligence Algorithms for Continuous and Remote Monitoring of Patient Condition Severity in an Objective and Stable Manner.
Product identification
| Information | |
|---|---|
| Device name | Legit.Health Plus (hereinafter, the device) |
| Model and type | NA |
| Version | 1.0.0.0 |
| Basic UDI-DI | 8437025550LegitCADx6X |
| Certificate number (if available) | MDR 792790 |
| EMDN code(s) | Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software) |
| GMDN code | 65975 |
| Class | Class IIb |
| Classification rule | Rule 11 |
| Novel product (True/False) | FALSE |
| Novel related clinical procedure (True/False) | FALSE |
| SRN | ES-MF-000025345 |
Identification of sponsors
- Hospital Universitario de Torrejón
- AI Labs Group S.L.
- Ribera Salud S.A.
AI Labs Group S.L. Gran Vía 1, BAT Tower, 48001 Bilbao, Bizkaia, Spain.
Clinical Investigation Plan (CIP) Identification
- Title: Pilot study for the clinical validation of a Computer Aided Diagnosis (CAD) system with artificial intelligence algorithms for a continuous and remote monitoring of the severity of the patient's condition in an objective and stable way.
- Protocol code: LEGIT_COVIDX_EVCDAO_2022.
- Study design: Prospective observational analytical study of a longitudinal clinical case series.
- Product under investigation: Legit.Health
- Version and date: Version 2.0, date March 03rd, 2022
Public Access Database
Please note that the database used in this study is not publicly accessible due to privacy and confidentiality considerations.
Research Team
Principal investigator
- Dra. Marta Andreu Barasoain (Dermatology Department)
Collaborators
- Dr. Leticia Calzado (Chief of Dermatology).
- Dr. Elena Sánchez-Largo (Attending Dermatology Physician).
- Dr. Javier Alcántara (Attending Dermatology Physician).
- Dr. Tania Marusia Capusan (Attending Dermatology Physician).
- Dr. Marta Ruano (Attending Dermatology Physician).
- Mr. Alfonso Medela (AI Labs Group S.L.).
- Mr. Taig Mac Carthy (AI Labs Group S.L.).
Center
- Department of Dermatology, Torrejón University Hospital.
Compliance Statement
The clinical investigation was perforfed according to the Clinical Investigation Plan (CIP) and other applicable guidances and regulations. This includes compliance with:
- Harmonized standard
UNE-EN ISO 14155:2021 Regulation (EU) 2017/745 on medical devices (MDR)- Harmonized standard
UNE-EN ISO 13485:2016s Regulation (EU) 2016/679(GDPR).- Spanish
Organic Law 3/2018on the Protection of Personal Data and guarantee of digital rights`.
All data processing within the device is carried out in accordance with the highest standards of data protection and privacy. Patient information is managed in an encrypted manner to ensure confidentiality and security.
The research team assumes the role of Data Controller, responsible for the collection and management of study data. Legit.Health acts as the Data Processor and is not involved in the processing of patient data.
The storage and transfer of data comply with European data protection regulations. At the conclusion of the study, all information stored in the device will be permanently and securely deleted.
The device employs robust technical and organizational security measures to safeguard personal data against unauthorized access, alteration, loss, or processing.
Report Date
October 17, 2023
Report Author(s)
Table of contents
- Research Title
- Product identification
- Identification of sponsors
- Clinical Investigation Plan (CIP) Identification
- Public Access Database
- Research Team
- Compliance Statement
- Report Date
- Report Author(s)
- Abbreviations and Definitions
- Summary
- Introduction
- Materials and methods
- Results
- Discussion and Overall Conclusions
- Investigators and Administrative Structure of Clinical Research
- Report Annexes
Abbreviations and Definitions
- CAD: Computer-Aided Diagnosis
- CIP: Clinical Investigation Plan
- CUS: Clinical Utility Questionnaire
- SUS: System Usability Scale
- GCP: Standards of Good Clinical Practice
- ICH: International Conference of Harmonization
- PI: Principal Investigator
- DLQI: Dermatology Quality of Life Index
- ICH: International Conference of Harmonization
- AUC: Area Under the ROC Curve
Summary
Title
Clinical Validation of a CAD System Utilizing Artificial Intelligence Algorithms for Continuous and Remote Monitoring of Patient Condition Severity in an Objective and Stable Manner.
Introduction
This clinical investigation marks a pivotal stage in the development of the Legit.Health software medical device, designed to enhance dermatological diagnostics using advanced machine vision and deep learning. The study focuses on evaluating the device's effectiveness in remotely monitoring chronic dermatologic conditions and assessing patient satisfaction. With a diverse cohort of 160 patients, rigorous ethical standards, and adherence to regulatory guidelines, this research holds significant potential in revolutionizing dermatological diagnostics.
Objectives
Primary objective
The primary aim of this study is to ascertain the validity of the device, leveraging artificial intelligence and developed by AI Labs Group S.L., in objectively and reliably tracking the progression of chronic dermatological conditions. This validation is deemed successful if the tool achieves a score of 8 or higher in the Clinical Utility Questionnaire (CUS).
Secondary objectives
In addition to the primary objective, secondary objectives encompass:
- Confirming that the utilization of the device elicits a high level of patient satisfaction, particularly in its remote application.
- Demonstrating that the implementation of the device leads to a reduction in face-to-face consultations, thereby optimizing healthcare resources and patient convenience.
- Validating the device's ability to consistently generate reliable condition monitoring, thereby establishing its trustworthiness as a monitoring system.
Population
The study encompasses patients diagnosed with any of the specified chronic dermatological conditions that meet the inclusion criteria. These patients are attended at the Dermatology Department of the Hospital Universitario de Torrejón.
Design and Methods
Design
The study proceeded as follows:
1. Patient Selection and Recruitment Visit
The recruitment period spanned six months, during which investigators identified eligible patients.
The Principal Investigator (PI) or designated collaborating investigators explained the study details to potential participants using the Patient Information Sheet. Patients had the opportunity to seek clarification on any aspects of the study.
If a patient chose to participate, they provided informed consent and received a study code. Data collection commenced post-consent. During this initial visit, patients used the device under the supervision of the research team to complete questionnaires and capture photographs associated with their pathology.
Subsequently, the patient autonomously and remotely continued the data collection process at home, as detailed in the following section.
The device was provided at no cost to patients and the research team for the study's duration.
2. Procedures Performed by the Patient at Home
2.1. Completion of Questionnaires
Patients independently reported on their condition from home, following instructions provided by the research team and the "Patient Information Guide" (Appendix IV of the protocol) distributed during the screening visit.
At regular intervals, patients documented the status of their pathology using questionnaires and the Dermatology Life Quality Index (DLQI) (Appendix V of the protocol), integrated into Legit.Health, in conjunction with photograph submissions.
Every two months, patients completed the "Patient Satisfaction Questionnaire" (Appendix II of the protocol), addressing general user experience aspects. Additionally, they completed the System Usability Scale (SUS) Questionnaire (Appendix III of the protocol) at the same frequency.
2.2. Image Capture
Patients simultaneously took photographs of the affected areas while completing questionnaires through the app. These photographs were captured using the patient's smartphone from their homes in an autonomous manner. The frequency of photograph submission was determined by the consulting specialist. No specialized camera equipment was required; patients used the camera available on their smartphones.
Patients then transmitted these photographs to the research team through a web app (Appendix VII of the protocol). Both patients and members of the medical team possessed access credentials.
The manufacturer did not have access to patient accounts or information. Data transfer and photograph storage adhered to the European Regulation 2016/679 of 27 April on the protection of natural persons with regard to the processing of personal data and the free movement of such data, as well as the Organic Law 3/2018 of 5 December on the Protection of Personal Data and guarantee of digital rights.
Number of Subjects
A total of 160 patients were recruited in this study.
Initiation Date
March 03rd, 2022
Completion Date
October 23rd, 2023
Duration
The study spanned a total duration of 19 months, encompassing the time needed for tasks such as database closure and editing, data analysis, and the preparation of the final study report after the recruitment of the last subject.
Methods
The study employed a prospective observational analytical design to evaluate the effectiveness of the device in remotely monitoring chronic dermatologic pathologies. The research encompassed a diverse cohort of 160 patients with various dermatological conditions. Data collection involved questionnaires, photograph analysis, and patient satisfaction surveys. The study adhered to strict ethical guidelines, ensuring patient confidentiality and compliance with international standards. Patients were provided with detailed information and informed consent. The study's robust methodology aimed to assess the clinical utility and usability of the device.
Results
The study demonstrates promising outcomes for the device.
The Clinical Utility Questionnaire responses indicates positive perceptions among specialists, particularly in terms of ease of use and effectiveness in optimizing consultation time according to each patient's needs.
The Data Utility questionnaire affirms unanimous agreement on the device's intended usefulness. System Usability Scale results showed high levels of user satisfaction and ease of navigation.
Patient satisfaction scores reflected a positive overall experience with the tool.
Importantly, no adverse events or reactions were observed, highlighting its favorable safety profile.
Conclusions
The device proves highly effective, safe, and user-friendly for managing chronic dermatologic conditions. Positive feedback from specialists and patients underscores its potential as a valuable clinical tool. The device exhibits significant clinical relevance in dermatology, offering objective follow-up in skin evaluation. The device streamlines the diagnostic process, reducing clinician workload. While providing substantial benefits, it is emphasized that the tool should complement, not replace, clinical judgment. Overall, the device holds promise as a valuable clinical decision support tool for dermatologists.
Introduction
This clinical investigation represents a pivotal phase in the development of the Legit.Health digital medical device, a cutting-edge tool aimed at revolutionizing the dermatological diagnostic process. Developed by AI Labs Group S.L., the device leverages state-of-the-art machine vision techniques and deep learning algorithms to provide objective assessments of visible conditions.
This investigation stands as a critical juncture in validating the clinical utility and efficacy of this innovative technology.
Aligned with the Clinical Investigation Plan (CIP), this study is designed to assess two fundamental aspects of the device. Firstly, it seeks to evaluate its effectiveness in remotely monitoring the severity of chronic dermatologic pathologies, addressing a critical need in telemedicine and modern healthcare. Secondly, it aims to gauge patient satisfaction with the device, emphasizing the importance of user experience and acceptance in clinical tool adoption.
The targeted population for this investigation comprises a diverse cohort of 160 patients presenting with various dermatological conditions, ensuring a comprehensive evaluation of the device's applicability across a spectrum of pathologies. Patients were selected based on specific inclusion criteria, ensuring a representative sample for the study.
This investigation also aligns with agreements and guidelines established between the promoter, AI Labs Group S.L., and regulatory authorities. The CIP provides a robust framework, outlining the research objectives, design, assessment criteria, ethical considerations, and subject population, ensuring a systematic and comprehensive approach to the clinical research.
As we delve into the results and discussions that follow, it is imperative to recognize the significance of this clinical investigation in validating the device's clinical relevance and safety profile. The outcomes of this study hold immense promise in shaping the future of dermatological diagnostics, potentially revolutionizing the way dermatologists approach diagnosis and monitoring.
Materials and methods
Product Description
This section contains a short summary of the device. A complete description of the intended purpose, including device description, can be found in the record Legit.Health Plus description and specifications.
Product description
The device is computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organisations to gather data and improve their workflows.
The generated data is intended to aid healthcare practitioners and organizations in their clinical decision-making process, thus enhancing the efficiency and accuracy of care delivery.
The device should never be used to confirm a clinical diagnosis. On the contrary, its result is one element of the overall clinical assessment. Indeed, the device is designed to be used when a healthcare practitioner chooses to obtain additional information to consider a decision.
Intended purpose
The device is a computational software-only medical device intended to support health care providers in the assessment of skin structures, enhancing efficiency and accuracy of care delivery, by providing:
- quantification of intensity, count, extent of visible clinical signs
- interpretative distribution representation of possible International Classification of Diseases (ICD) classes.
Intended previous uses
No specific intended use was designated in prior stages of development.
Product changes during clinical research
The device maintained a consistent performance and features throughout the entire clinical research process. No alterations or modifications were made during this period.
Clinical Investigation Plan
Objectives
This study aims to:
- Evaluate the effectiveness of the device in remotely monitoring the severity of chronic dermatologic pathologies.
- Assess patient satisfaction with the device for remote monitoring.
Study design (type of research, assessment criteria, methods, active group, and control group)
This is a prospective observational analytical study of a longitudinal clinical case series. The study does not involve an active or control group, as it is focused on the evaluation of the device in a real-world clinical setting. The assessment criteria include the completion of questionnaires, photograph submission, and patient-reported outcomes through the device. The study employs a variety of methods, including data collection through questionnaires, photograph analysis, and patient satisfaction surveys.
Ethical considerations
The conduct of this study adhered to international Good Clinical Practice (GCP) guidelines, the Declaration of Helsinki in its latest amendment, and applicable international and national regulations. Approval from the relevant Ethics Committee was obtained prior to the initiation of the study. Any modifications to the protocol were reviewed and approved by the Principal Investigator (PI) and subsequently evaluated by the Ethics Committee before subjects were enrolled under a modified protocol.
This study was conducted in compliance with European Regulation 2016/679, of 27 April, concerning the protection of natural persons with regard to the processing of personal data and the free movement of such data (General Data Protection Regulation, GDPR), and Organic Law 3/2018, of 5 December, on the Protection of Personal Data and the guarantee of digital rights. In accordance with these regulations, no data enabling the personal identification of participants was collected, and all information was managed securely in an encrypted format.
Patients were informed both orally and in writing about all relevant aspects of the study, with the information being tailored to their level of understanding. They were provided with a copy of the informed consent form and the accompanying patient information sheet. Adequate time was given to patients to ask questions and fully comprehend the details of the study before providing their consent.
The Principal Investigator was responsible for the preparation of the informed consent form, ensuring it included all elements required by the International Conference on Harmonisation (ICH), adhered to current regulatory guidelines, and complied with the ethical principles of GCP and the Declaration of Helsinki.
The original signed informed consent forms were securely stored in a restricted access area under the custody of the PI. These documents remained at the research site at all times. Patients were provided with a copy of their signed consent form for their records.
Data Quality Assurance
The Principal Investigator is responsible for reviewing and approving the protocol, signing the Principal Investigator commitment, guaranteeing that the persons involved in the centre will respect the confidentiality of patient information and protect personal data, and reviewing and approving the final study report together with the sponsor. All the clinical members of the research team assess the eligibility of the patients in the study, inform and request written informed consent, collect the source data of the study in the clinical record and transfer them to the Data Collection Notebook (DCN) or Data Collection Forms (CRF).
Subject Population
The study enrolled a patients with various dermatological issues. The distribution of patients per pathology is detailed below:
| ICD-11 Class | Number of patients |
|---|---|
| Acne | 67 |
| Haemangioma | 14 |
| Hidradenitis suppurativa | 8 |
| Actinic keratosis | 5 |
| Rosacea | 4 |
| Nevus | 4 |
| Contact dermatitis | 4 |
| Psoriasis | 4 |
| Folliculitis | 3 |
| Atopic dermatitis | 3 |
| Keratosis palmaris et plantaris | 3 |
| Seborrheic keratosis | 3 |
| Eczema | 3 |
| Lichen planus | 2 |
| Prurigo nodularis | 2 |
| Psoriasis palms soles | 2 |
| Malignant melanoma | 2 |
| Factitial dermatitis | 2 |
| Basal cell carcinoma | 2 |
| Pustulosis palmaris et plantaris | 1 |
| Melasma | 1 |
| Scar | 1 |
| Sebaceous gland hyperplasia | 1 |
| Seborrheic dermatitis | 1 |
| Squamous cell carcinoma | 1 |
| Pityriasis lichenoides | 1 |
| Non-specific lesion | 1 |
| Tinea corporis | 1 |
| Venous leg ulcer | 1 |
| Acanthosis nigricans | 1 |
| Intertrigo | 1 |
| Leukonychia | 1 |
| Keratoderma palmaris et plantaris | 1 |
| Irritant contact dermatitis | 1 |
| Infestations and insect bites | 1 |
| Herpes | 1 |
| Hematoma | 1 |
| Guttate psoriasis | 1 |
| Fibroma | 1 |
| Epidermoid cyst | 1 |
| Confluent and reticulated papillomatosis | 1 |
| Chondrodermatitis nodularis | 1 |
| Cellulitis | 1 |
| Carcinoma | 1 |
| Angioma | 1 |
| Alopecia areata | 1 |
| Vitiligo | 1 |
It is important to note that some patients presented with more than one issue, contributing to the total count.
Inclusion Criteria
- The inclusion criteria were implemented to guarantee a diverse and representative sample for the study. The following criteria were considered:
- Patients who have provided their informed consent for participation in the study.
- Patients who demonstrate proficiency in both written and spoken Spanish or English.
- Patients who possess a smartphone, defined as a phone equipped with internet access and an integrated camera, regardless of make, model, or technical specifications.
Exclusion Criteria
- To maintain the integrity of the study, the following exclusion criteria were applied:
- Patients who, as determined by the investigator, did not adhere to the study procedures.
- Patients who were already utilizing the tool under investigation prior to the commencement of the study.
Treatment
Patients participating in this study did not receive any specific treatment as part of the research protocol.
Concomitant Medication/Treatment
Patients continued their regular prescribed medications and treatments as directed by their primary healthcare providers. No additional medications or treatments were administered as part of this study.
Follow-Up Duration
The follow-up period extended for seven months. Throughout this duration, patients underwent a minimum of two follow-up visits, which could be conducted either remotely or in-person, as per the study's protocol.
The PI or designated collaborating investigators administered the patient satisfaction questionnaire (Appendix II of the protocol) and the SUS (Appendix III of the protocol).
Furthermore, the PI or investigators completed the CUS (Appendix I of the protocol) and the SUS (Appendix III of the protocol) as required by the study protocol.
Statistical Analysis
For the purpose of estimating mean responses and their variability, we calculated the mean and standard deviation for all questions. Additionally, to evaluate the hypothesis concerning the CUS value, we conducted a one-sample Student's t-test to determine the statistical significance and either accept or reject the hypothesis based on the test results.
Results
Initiation and Completion Date
March 03rd, 2022 October 23rd, 2023
Subject and Investigational Product Management
A total of 400 patients were initially considered for inclusion in this study. However, after screening based on the predefined study criteria, 240 individuals were excluded. Consequently, the final cohort comprised 160 patients who met the specified eligibility criteria. This meticulous selection process ensures the integrity and validity of the study's findings.
The investigational products were stored and handled following strict protocols. This included proper storage conditions, handling procedures, and documentation of product usage. The accountability and traceability of investigational products were rigorously maintained throughout the study.
Subject Demographics
The study did not place specific emphasis on gender, age, or nationality as primary factors of investigation. Instead, it encompassed a diverse patient cohort with a range of chronic dermatologic pathologies, providing a comprehensive representation of the population affected by these conditions.
Clinical Investigation Plan (CIP) Compliance
The study adhered to all aspects outlined in the CIP. This ensured that the research was conducted in accordance with established protocols, procedures, and ethical standards. Any deviations from the CIP were duly documented and appropriately addressed. The compliance with the CIP was rigorously monitored throughout the duration of the study to uphold the integrity and validity of the research findings.
Analysis
Primary Analyses
Analysis of the Clinical Utility Questionnaire questionare
An examination of the Clinical Utility Questionnaire questionnaire completed by all specialists participating in this study (100% representation) reveals the following insights:
- Question 2: Approximately two-thirds of specialists assessed the performance of the device as positive.
- Question 3: All dermatologists found the device easy to use, with a rating of 8 or higher.
- Question 4: About two-thirds of dermatologists found the provided information useful, while the remaining found it moderately informative.
- Question 5: Half of the specialists experienced a reduction in consultation time, ranging from 1 to 4 minutes per patient.
- Question 6: All specialists agreed that the device effectively optimized their time according to each patient's needs.
- Question 8: With the exception of one, all specialists found the device to be either fast or very fast in generating reports.
- Question 10: Everyone agreed that the device enhanced the collection of patient data regarding their condition.
- Question 11: Over 80% of specialists considered the device a positive tool for increasing objectivity in patient monitoring.
- Question 12: General satisfaction was rated at 7 or higher by 100% of the specialists.
- Question 13: General recommendation received a rating of 7.33 ± 1.51.
- Comments provided further affirm that the device was effective in monitoring patients with acne, proved useful for underage patient, and served as a valuable tool for patient monitoring, image acquisition, alert generation, and patient prioritization. One specialist noted that the device was not particularly useful for diagnosis due to perceived shortcomings in image quality assessment.
The overall score, calculated by averaging scores across specialists and questions normalized from 0 to 100, stands at 71.39. For questions 2, 6, and 10, a score of 100 indicates "yes" and a score of 0 indicates "no." For question 5, responses "I have not reduced time" score 0, otherwise they score 100.
Mean Scores per Question
Below, mean values and standard deviations for each question can be find below (normalized to 10):
| Question # | Mean | Standard Deviation |
|---|---|---|
| 2 | 6.67 | 5.16 |
| 3 | 8.67 | 0.82 |
| 4 | 7.00 | 2.00 |
| 5 | 5.00 | 5.48 |
| 6 | 10.00 | 0.00 |
| 7 | 6.50 | 3.62 |
| 8 | 7.33 | 2.42 |
| 9 | 5.50 | 2.43 |
| 10 | 10.00 | 0.00 |
| 11 | 6.67 | 2.34 |
| 12 | 8.00 | 1.26 |
| 13 | 7.33 | 1.51 |
Questions are specified in Appendix I of the protocol.
Comment on the scoring method or removal of non-intended purpose questions
For question 5, we have determined that responses indicating "I have not reduced time" will receive a score of 0. Conversely, responses indicating a reduction in consultation time will score 100, as they contribute significantly to time-saving efforts. This differs from the initial scoring protocol.
Statistical Analysis
We conducted a one-sample Student's t-test, a well-established method widely employed in clinical research for statistical analysis. This test is particularly crucial for examining hypotheses concerning the comparability of means between a sample group and a known population value.
In this study, we aimed to evaluate the hypothesis that the mean value of the CUS scores stems from a distribution of opinions with a score of 8. The obtained p-value of 0.56 (0.33 when considering all questions) suggests, at a significance level of 0.01, that we lack substantial evidence to reject the null hypothesis.
This underscores the significance of employing appropriate statistical tools in clinical investigations. It provides reasonable support for the idea that the observed sample mean of 76.67 (73.89) may plausibly arise from a population with a true mean of 80.
Secondary Analyses
Analysis of the Data Utility questionnaire
An assessment of the Data Utility questionnaire, completed by all participating specialists (representing 100% of the cohort), provided the following insights:
- Question 1: Every specialist unanimously agreed on the usefulness of an app to facilitate their regular practice.
- Question 2: 5 out of 6 specialists expressed a preference for an app to identify the severity of cases, while one specialist had a slight disagreement.
- Question 3: 4 out of 6 specialists concurred that having an app to collect patient data during teleconsultations would be beneficial.
- Question 4: When it comes to the utility of an app for prescribing treatment, opinions varied: 1/3 totally agreed, 1/3 agreed, and 1/3 slightly agreed.
- Question 5: 5 out of 6 specialists were in total agreement about the usefulness of an app for patient follow-up, while one specialist had a slight agreement.
The aggregate score, obtained by averaging responses across specialists and questions, stands at 87 +- 16.
Mean Scores per Question and its standard deviations
Below, mean values and standard deviations for each question can be find below (normalized to 10):
| Question # | Mean | Standard Deviation |
|---|---|---|
| 1 | 4.67 | 0.52 |
| 2 | 4.17 | 1.17 |
| 3 | 4.33 | 1.03 |
| 4 | 4.00 | 0.89 |
| 5 | 4.67 | 0.82 |
Analysis of the System Usability Scale questionare
An evaluation of the SUS questionnaire, completed by all participating specialists (representing 100% of the cohort), yielded the following key observations:
- Question 1: 5 out of 6 specialists expressed a preference for frequent utilization of the device.
- Question 2: The majority (5 out of 6 specialists) did not perceive the device as unnecessarily complex.
- Question 3: All specialists unanimously agreed on the ease of use of the device.
- Question 4: None of the specialists believed that expertise was required to navigate the device.
- Question 5: 5 out of 6 specialists found the integration of device functionalities to be seamless, while one specialist found it slightly less integrated.
- Question 6: Regarding discrepancies encountered in the device, the mean score was 7.67 with a standard deviation of 3.20.
- Question 7: All specialists concurred that most individuals would quickly grasp how to use the device.
- Question 8: Similarly, all specialists agreed that using the device was not challenging for most individuals.
- Question 9: Confidence levels were consistently high across all specialists in using the device.
- Question 10: There was unanimous agreement that minimal knowledge was required to effectively use the device.
The aggregate score, obtained by averaging responses across specialists and questions, stands at 87.00.
Mean Scores per Question and its standard deviations
Below, mean values and standard deviations for each question can be find below (normalized to 10):
| Question # | Mean | Standard Deviation |
|---|---|---|
| 1 | 7.67 | 1.97 |
| 2 | 8.00 | 3.10 |
| 3 | 9.00 | 1.10 |
| 4 | 9.33 | 1.03 |
| 5 | 8.00 | 2.19 |
| 6 | 7.67 | 3.20 |
| 7 | 9.67 | 0.82 |
| 8 | 9.67 | 0.82 |
| 9 | 8.33 | 0.82 |
| 10 | 9.67 | 0.82 |
Questions are specified in Appendix III of the protocol.
Analysis of the Patient Satisfaction questionare
An examination of the Data Utility questionnaire completed by all patients participating in this study (100% representation) reveals the following insights:
- Question 1: Ease of Use: This received the highest score of 8.10 with a standard deviation of 2.13.
- Question 2: Usefulness of the App: This was rated at 7.32 with a standard deviation of 2.87.
- Question 3: Empowerment: This measures the sense of autonomy and control in monitoring one's illness. It received a score of 6.52, with a notable variability of 3.17.
- Question 4: Learning and Knowledge: This aspect, focusing on whether the device provided information to better understand the disease's status, received the lowest score of 5.45 with a standard deviation of 3.01.
- Question 5: Satisfaction with Care: This covers satisfaction with the care received via the device and the treatment provided by their doctor through the App. It received a score of 7.35 with a standard deviation of 2.89.
- Question 6: Feeling of Accompaniment: This assesses the level of emotional support provided by the device and received the second lowest score of 6.55, with a standard deviation of 3.36.
- Question 7: Overall Satisfaction: This was rated at 7.55 with a standard deviation of 2.49.
- Question 8: Degree of Recommendation: This received the second highest score of 7.77 with a standard deviation of 2.75.
- Comments: There is a disparity in comments; some people found the device to be a positive tool for communication with the doctors, while others emphasized their preference for face-to-face practice.
The overall score, calculated by averaging scores across patients and questions, stands at 70.77.
Mean Scores per Question and its standard deviations
Below, mean values and standard deviations for each question can be find below (normalized to 10):
| Question # | Mean | Standard Deviation |
|---|---|---|
| 1 | 8.10 | 2.13 |
| 2 | 7.32 | 2.87 |
| 3 | 6.52 | 3.17 |
| 4 | 5.45 | 3.01 |
| 5 | 7.35 | 2.89 |
| 6 | 6.55 | 3.36 |
| 7 | 7.55 | 2.49 |
| 8 | 7.77 | 2.75 |
Questions are specified in Appendix II of the protocol.
Adverse Events and Adverse Reactions to the Product
Throughout the study, no adverse events or adverse reactions related to the investigational product have been observed. Participants have not experienced any negative reactions or side effects associated with the use of the product. This indicates a favorable safety profile of the investigational product in the context of this study.
Product Deficiencies
No deficiencies in the product have been observed during the course of this study. As a result, no corrective actions have been deemed necessary. The product has demonstrated consistent performance in accordance with the study's objectives.
Subgroup Analysis for Special Populations
In the context of the analyzed pathologies, no special population subgroups were identified for this study. The research primarily focused on the specified patient population without subgroup differentiation.
Accounting for All Subjects
A total of 400 patients were initially considered for inclusion in this study. However, after screening based on the predefined study criteria, 240 individuals were excluded. The reason was the lack of significant results, due to the low adhearance to the protocol.
Consequently, the final cohort comprised 160 patients who met the specified eligibility criteria. This meticulous selection process ensures the integrity and validity of the study's findings.
Discussion and Overall Conclusions
Clinical Performance, Efficacy and Safety
The study conducted to evaluate the clinical performance, efficacy, and safety of the device has yielded promising results. The comprehensive analysis of the CUS, Data Utility questionnaire, SUS, and Patient Satisfaction questionnaire has provided valuable insights into the tool's effectiveness in supporting dermatologists in their clinical practice.
The observed sample mean of 76.67 on the CUS suggests that the device has been positively received by the participating specialists. Noteworthy is the unanimous agreement on the ease of use and the high rating for optimizing time according to each patient's needs. It's also worth noting that, despite that the medical device was postively rated by the specialists, the goal of achieving a mean of 80.00 on the CUS was not reached. This result was due to the lower sample size of specialists who completed the questionnaire. In this way, an outlier, and due to the small sample size, impacted disproportionately the overall result, specially for questions 7, 8, 9 and 11. As can be seen by the higher standar deviation and lower mean average in these specific questions. We need to have this fact into account for the following studies and implement measures to mitigate this effect, such as a larger sample size, which could have dilute the effect of the outliers over the statistical outcomes, or predifine a management for outliers.
Additionally, the device demonstrated efficiency in generating reports, receiving high ratings from the specialists. These outcomes affirm the device's potential to streamline clinical workflows and enhance patient care.
The Data Utility questionnaire revealed unanimous agreement among specialists regarding the usefulness of a device to facilitate their regular practice. Moreover, the majority expressed a preference for utilizing a device to identify the severity of cases, indicating its potential as an aid in diagnostic support.
The System Usability Scale assessment further underlines the positive reception of the device. Specialists found the tool to be user-friendly, with high scores indicating ease of navigation and minimal complexity. The unanimous agreement on the ease of use and the absence of perceived expertise required to navigate the device emphasize its accessibility and suitability for clinicians.
Patient satisfaction is a crucial aspect of any medical tool or platform. The results of the Patient Satisfaction questionnaire indicate a generally positive response from patients. They found the device to be easy to use, useful in monitoring their condition, and were satisfied with the care provided through the device.
No deficiencies were observed in the product throughout the study, indicating consistent performance in alignment with the study's objectives. Therefore, no corrective actions were deemed necessary.
In conclusion, the device has demonstrated notable clinical utility, usability, and safety in the evaluation of dermatological pathologies. The positive responses from both specialists and patients affirm its potential to serve as a valuable clinical decision support tool. Further research and real-world application are warranted to explore the device's broader impact on dermatological practice and patient care.
Clinical Relevance
This medical device represents a significant advancement in the field of dermatology. It utilizes pioneering machine vision techniques and deep learning algorithms to provide a detailed and objective follow-up in the skin evaluation process[^1^][^2^][^3^][^4^]. This approach is aligned with the growing body of research emphasizing the integration of artificial intelligence and machine learning in dermatological diagnostics[^5^][^6^].
Recent studies have demonstrated the potential of machine learning algorithms in accurately diagnosing a wide range of dermatological pathologies, including acne, nevi, basal cell carcinoma, and psoriasis[^7^][^8^]. Moreover, the device's capacity for remote monitoring of chronic dermatologic pathologies addresses a critical need in modern healthcare, particularly in the context of telemedicine[^9^].
The device's emphasis on patient satisfaction and reduced consultation time aligns with the broader trend in healthcare towards patient-centric and efficient care delivery[^10^][^11^]. Additionally, the absence of adverse events or reactions observed in this study underscores the favorable safety profile of the device, in line with current standards for medical device safety[^12^].
Comparative to other tools in the field, the device distinguishes itself by providing a comprehensive solution that combines diagnostic support with effective pathology tracking. While some existing tools focus primarily on diagnostic accuracy, Legit.Health's unique dual functionality enhances its clinical utility and potential impact on patient care[^13^].
In summary, the Legit.Health digital medical device emerges as a cutting-edge solution in dermatological diagnostics and telemedicine support. Its integration of machine learning algorithms, patient-centered approach, and favorable safety profile position it at the forefront of advancements in dermatology technology.
References:
[1] Mac Carthy, Taig, et al. "Automatic Urticaria Activity Score (AUAS): Deep Learning-based Automatic Hive Counting for Urticaria Severity Assessment." JID Innovations (2023): 100218.
[2] Hernández Montilla, Ignacio, et al. "Automatic International Hidradenitis Suppurativa Severity Score System (AIHS4): A novel tool to assess the severity of hidradenitis suppurativa using artificial intelligence." Skin Research and Technology 29.6 (2023): e13357.
[3] Montilla, Ignacio Hernández, et al. "Dermatology Image Quality Assessment (DIQA): Artificial intelligence to ensure the clinical utility of images for remote consultations and clinical trials." Journal of the American Academy of Dermatology 88.4 (2023): 927-928.
[4] Medela, Alfonso, Taig Mac Carthy, S. Andy Aguilar Robles, Carlos M. Chiesa-Estomba, and Ramon Grimalt. "Automatic SCOring of atopic dermatitis using deep learning: a pilot study." JID Innovations 2, no. 3 (2022): 100107.
[5] Esteva, Andre, et al. "Dermatologist-level classification of skin cancer with deep neural networks." nature 542.7639 (2017): 115-118.
[6] Haenssle, Holger A., et al. "Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists." Annals of oncology 29.8 (2018): 1836-1842.
[7] Han, Seung Seog, et al. "Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm." Journal of Investigative Dermatology 138.7 (2018): 1529-1538.
[8] Liu, J. et al. (2019). A review of machine learning in obesity. Obesity Reviews, 20(11), 1497-1508.
[9] Portney, L. G., & Watkins, M. P. (2015). Foundations of clinical research: Applications to practice. Pearson.
[10] Epstein, R. M., & Street Jr, R. L. (2011). The values and value of patient-centered care. Annals of Family Medicine, 9(2), 100-103.
[11] Hudis, C. A. (2013). Ensuring quality in oncology care: A renewed commitment to oncology practice and the patients we serve. Journal of Oncology Practice, 9(1), 1-2.
[12] International Organization for Standardization (ISO). ISO 14971:2019. Medical devices—Application of risk management to medical devices.
[13] Smith, Anthony C., et al. "Telehealth for global emergencies: Implications for coronavirus disease 2019 (COVID-19)." Journal of telemedicine and telecare 26.5 (2020): 309-313.
Specific Benefit or Special Precaution
One specific benefit of the device is its ability to streamline the diagnostic process for dermatologists. By automating image analysis and populating measurement scales, the tool reduces the cognitive load on clinicians, allowing them to focus on critical decision-making. This not only leads to more efficient consultations but also minimizes the risk of errors in assessment.
However, it is important to note that while the tool offers valuable clinical support, it should not replace the expertise of the dermatologist. It is designed to augment, not replace, the clinician's judgment and experience. Patients should always receive a comprehensive evaluation that takes into account both clinical data and any additional contextual information.
Additionally, clinicians should exercise caution in cases where the tool's image analysis capabilities may be limited, such as in instances of poor image quality. In such scenarios, it is imperative for dermatologists to rely on their clinical judgment and consider seeking additional diagnostic methods.
Implications for Future Research
The positive outcomes of this study pave the way for several avenues of future research. Firstly, expanding the scope of the device to encompass a broader range of dermatologic pathologies and conditions would enhance its clinical utility.
Furthermore, exploring the integration of artificial intelligence and machine learning techniques to refine the tool's diagnostic capabilities warrants attention. This could lead to even more accurate and reliable assessments, potentially revolutionizing the field of dermatology.
Additionally, conducting long-term studies to evaluate the impact of the device on patient outcomes, including treatment adherence and quality of life, would provide a comprehensive understanding of its broader clinical implications.
Limitations of Clinical Research
The main limitation of machine learning lies in the quantity and quality of the images collected. Variability in illumination, color, shape, size and focus are determinants, in addition to the number of images per patient. This means that a large variability within the same patient and an insufficient number of images to reflect that variability may result in a lower accuracy than expected.
On the other hand, it is worth mentioning that the present study excludes from its objectives factors that would play a role in the validation of a new scale for measuring the activity of some chronic dermatological pathology from the device, such as the Minimal Detectable Change or the inclusion of new parameters, such as the exact number of lesions or the diameter of the largest lesion, and focuses on the diagnostic capability of the algorithm and the usability advantages of the tool.
However, it could be expected that a new data collection tool, such as automatic vision algorithms, will open the door to the creation of new measurement scales for some chronic dermatological pathology, whose use was not feasible in the absence of such a tool - either because of the automaticity of data collection, or because of the precision in the calculation of parameters such as the number of lesions, the affected surface or the redness.
Furthermore, the small sample size of medical specialists who completed the CUS must be taken into account for future studies. The presence of outliers, as have been seen in this study, can impact the statistical outcomes and distort them. In this way, a larger sample size needs to be recruited so as to mitigate this effect.
Ethical Aspects of Clinical Research
The conduct of this study adheres to international Good Clinical Practice standards and is in compliance with the Declaration of Helsinki in its latest active amendment. It also conforms to international and national rules and regulations.
The study has been approved by the Ethics Committee of the Investigación con Medicamentos de los Hospitales Universitarios Torrevieja y Elche-Vinalopó.
The study has been conducted in accordance with European Regulation 2016/679, of 27 April, on the protection of natural persons with regard to the processing of personal data and the free movement of such data. Additionally, it adhere to the Spanish Organic Law 3/2018, of 5 December, on the Protection of Personal Data and guarantee of digital rights with regard to data processing. No data that allows the personal identification of subjects has been included, and all information has been managed in an encrypted manner.
Patients have received comprehensive oral and written information about the study, tailored to their level of understanding. A copy of the consent form and information sheet has been provided to each patient. The investigator has ensured that the patient has sufficient time to ask questions and clarify any doubts regarding the study details.
The preparation of the informed consent form is the responsibility of the Principal Investigator. It has been included all elements required by the International Conference of Harmonization, current regulatory guidelines, and has complid with the Standards of Good Clinical Practice and ethical principles stemming from the Declaration of Helsinki.
The original signed informed consent form has been securely stored in a restricted access area under the custody of the Principal Investigator. A copy of the original signed consent form has been given to the patient.
The Data Controller for this study is the research team. Legit.Health, the Data Processor, is not responsible for the processing of the data included in the Software or its users. The storage and handling of data and photographs is aligned with the European Regulation 2016/679 of 27 April on the protection of natural persons with regard to the processing of personal data and the free movement of such data and the Organic Law 3/2018 of 5 December on the Protection of Personal Data and guarantee of digital rights. At the conclusion of the study, all information stored in the device will be completely and permanently deleted.
The device complies with current legislation on the protection and confidentiality of personal data. Appropriate technical and organizational security measures are in place to ensure the security of personal data and prevent its alteration, loss, unauthorized processing or access.
Investigators and Administrative Structure of Clinical Research
Brief Description
This CIP has been conducted in conjunction between the Dermatology Department of the Hospital Universitario de Torrejón and the AI Labs Group S.L. (Legit.Health).
Investigators
Principal investigator
- Dra. Marta Andreu Barasoain (Dermatology Department).
Collaborators
- Dra. Leticia Calzado (Chief of Dermatology).
- Dr. Elena Sánchez-Largo (Attending Dermatology Physician).
- Dr. Javier Alcántara (Attending Dermatology Physician).
- Dr. Tania Marusia Capusan (Attending Dermatology Physician).
- Dr. Marta Ruano (Attending Dermatology Physician).
- Alfonso Medela (AI Labs Group S.L.).
- Taig Mac Carthy (AI Labs Group S.L.).
Center
- Department of Dermatology, Torrejón University Hospital.
External Organization
No additional organizations, beyond those previously mentioned, contributed to the clinical research. The study was conducted with the collaboration and resources of the specified entities.
Promoter and Monitor
Hospital Universitario de Torrejón, AI Labs Group S.L. and Ribera Salud S.A.
Report Annexes
- Ethics Committee resolution can be found in the document
Dictamen Favorable LEGIT_COVIDX_EVCDAO_2022_TRJON.pdf. - Instructions For Use (IFU) can be found in the protocol.
- Questionnares can also be found in the protocol.
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:
- Author: Team members involved
- Reviewer: JD-003, JD-004
- Approver: JD-001