R-TF-015-006 Clinical investigation report

Research Title

Clinical validation study of the device's malignancy-estimation and melanoma-detection performance on patients presenting with skin lesions of suspected malignancy, using dermatoscopic imaging.

Nature and positioning of the evidence

This investigation is a prospective, consecutive-case, cross-sectional clinical study on real patients with suspected malignant pigmented skin lesions. The reference standard is pathological anatomy (biopsy) for the subset of lesions excised and a panel consensus of expert dermatologists for the remainder, adjudicated independently of the device output. Under MDCG 2020-6 Appendix III the resulting evidence is Rank 2–4 (prospective observational study with reference standard). Per MDCG 2020-1 §4.4 it contributes primary Pillar 3 Clinical Performance evidence — measuring the diagnostic decision supported by the device's clinical output in the Top-5 prioritised differential and malignancy-gauge workflow — for the intended-purpose population of patients presenting with lesions of suspected malignancy. Pillar 2 (the algorithm's API-level analytical performance across the ICD-11 categories) is evidenced independently through the device verification-and-validation records and is not the subject of this investigation. Pillar 1 (Valid Clinical Association literature anchoring early melanoma detection and accurate malignancy triage to improved patient outcomes) is documented in the State of the Art (R-TF-015-011).

Description

Clinical validation study of a smartphone-based computer-aided diagnosis (CADx) function for early non-invasive detection of in vivo cutaneous melanoma and for malignancy estimation on patients with skin lesions of suspected malignancy, conducted at two hospitals (Hospital Universitario Cruces and Hospital Universitario Basurto) between 2020 and 2023. The initial cohort of 40 subjects was later extended to 105 subjects through a substantial amendment approved by the CEIm of Euskadi.

Product identification

	Information
Device name	Legit.Health Plus (hereinafter, the device)
Model and type	NA
Version	1.1.0.0
Basic UDI-DI	8437025550LegitCADx6X
Certificate number (if available)	MDR 000000 (Pending)
EMDN code(s)	Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software)
GMDN code	65975
EU MDR 2017/745	Class IIb
EU MDR Classification rule	Rule 11
Novel product (True/False)	TRUE
Novel related clinical procedure (True/False)	TRUE
SRN	ES-MF-000025345

Throughout this document, references to "the device" refer to the investigational product identified above.

Device version under investigation and bridging to the CE-marked release

The investigation was conducted using device version v1.1.0.0, which is the version placed on the market and the version to which the present technical documentation applies. The bridging between the investigation-version and the CE-marked release is therefore an identity bridge; no clinical-relevance assessment is required. The device-version statement has been reviewed and signed off by the PRRC.

Sponsor Identification and Contact

	Manufacturer data
Legal manufacturer name	AI Labs Group S.L.
Address	Street Gran Vía 1, BAT Tower, 48001, Bilbao, Bizkaia (Spain)
SRN	ES-MF-000025345
Person responsible for regulatory compliance	Alfonso Medela, Saray Ugidos
E-mail	office@legit.health
Phone	+34 638127476
Trademark	Legit.Health
Authorized Representative	Not applicable (manufacturer is based in EU)

Identification of the Clinical Investigation Plan (CIP)

	CIP
Title of the clinical investigation	Clinical validation study of a CAD system with artificial intelligence algorithms for early noninvasive in vivo cutaneous melanoma detection
Device under investigation	Legit.Health Legacy Device
Protocol version	Version 3.0
Date	2021-10-28
Protocol code	LEGIT_MC_EVCDAO_2019
Sponsor	AI Labs Group S.L.
Coordinating Investigator	Dr. Jesus Gardeazabal Garcia and Dr. Rosa Ma Izu Belloso
Principal Investigator(s)	Dr. Jesus Gardeazabal Garcia and Dr. Rosa Ma Izu Belloso
Investigational site(s)	Hospital Universitario Cruces and Hospital Universitario Basurto
Ethics Committee	Comite de Etica de la Investigacion con Medicamentos de Euskadi

Trial Registrations

• ClinicalTrials.gov (NCT): NCT06221397
• EMA RWD Catalogue (EUPAS): EUPAS108254

Public Access Database

The database used in this study is not publicly accessible due to privacy and confidentiality considerations.

Research Team

Principal investigators

• Dr. Jesús Gardeazabal García (Osakidetza, Hospital Universitario Cruces)
• Dr. Rosa Mª Izu Belloso (Osakidetza, Hospital Universitario Basurto)

Collaborating Investigators (Clinical Staff)

• Dr. Juan Antonio Ratón Nieto (Servicio de Dermatología, Hospital Universitario Cruces)
• Dr. Ana Sánchez Díez (Servicio Dermatología, Hospital Universitario Basurto)

Technical Support (Manufacturer)

• Mr. Alfonso Medela — Chief Technology Officer
• Mrs. Andy Aguilar — General Manager
• Mr. Taig Mac Carthy — Chief Executive Officer

Centres

• Hospital Universitario Cruces
• Hospital Universitario Basurto

Compliance Statement

The clinical investigation will be conducted according to the Clinical Investigation Plan (CIP) and other applicable guidances and regulations. This includes compliance with:

• The ethical principles originating from the World Medical Association's Declaration of Helsinki
• Harmonized standard UNE-EN ISO 14155:2020
• Regulation (EU) 2017/745 on medical devices (MDR), including the applicable General Safety and Performance Requirements (GSPR) as outlined in Annex I, and the requirements of Annex XV (Chapter I and Chapter II, Section 3)
• Harmonized standard UNE-EN ISO 13485:2016
• MDCG 2024-3 for its structural and content expectations, MDCG 2021-8 concerning application requirements, and MDCG 2020-10/1 Rev 1 for safety reporting timelines and definitions
• Regulation (EU) 2016/679 (GDPR)
• Spanish Organic Law 3/2018 on the Protection of Personal Data and guarantee of digital rights.

All data processing within the device is carried out in accordance with the highest standards of data protection and privacy. Patient information is managed in an encrypted manner to ensure confidentiality and security.

The research team assumes the role of Data Controller, responsible for the collection and management of study data. Legit.Health acts as the Data Processor and is not involved in the processing of patient data.

The storage and transfer of data comply with European data protection regulations. At the conclusion of the study, all information stored in the device will be permanently and securely deleted.

The device employs robust technical and organizational security measures to safeguard personal data against unauthorized access, alteration, loss, or processing.

Report Date

May 31, 2024.

Report Author(s)

The full name, the ID and the signature for the authorship, as well as the approval process of this document, can be found in the verified commits at the repository. This information is saved alongside the digital signature, to ensure the integrity of the document.

Table of contents

Table of contents

• Research Title
  • Nature and positioning of the evidence
• Description
• Product identification
  • Device version under investigation and bridging to the CE-marked release
• Sponsor Identification and Contact
• Identification of the Clinical Investigation Plan (CIP)
  • Trial Registrations
• Public Access Database
• Research Team
  • Principal investigators
  • Collaborating Investigators (Clinical Staff)
  • Technical Support (Manufacturer)
  • Centres
• Compliance Statement
• Report Date
• Report Author(s)
• Table of contents
• Abbreviations and Definitions
• Summary
  • Research title
  • Introduction
  • Objectives
    • Hypothesis
    • Primary objective
    • Secondary objectives
  • Population
  • Sample size
  • Design and Methods
    • Design
    • Number of Subjects
    • Initiation Date
    • Completion Date
    • Duration
    • Methods
  • Results
    • Results of the first sample
    • Results of the second sample
  • Conclusions
    • Benefit-risk assessment against GSPR 1 and GSPR 8
• Introduction
• Materials and methods
  • Product Description
  • Clinical Investigation Plan
    • Objectives
    • Design (type of research, assessment criteria, methods, active group, and control group)
    • Ethical considerations
    • Data Quality Assurance
    • Subject Population (inclusion/exclusion criteria and sample size)
    • Treatment
    • Concomitant medication or treatment
    • Follow-Up duration
    • Statistical analysis
• Results
  • Initiation and Completion Date
  • Subject and Investigational Product Management
  • Subject Demographics
  • CIP (Clinical Investigation Plan) Compliance
    • Protocol Deviations
  • Analysis
    • Primary analysis
    • Secondary analyses
    • Adverse Events and Adverse Reactions to the Product
    • Product Deficiencies
    • Subgroup Analysis for Special Populations
    • Accounting for all subjects
• Discussion and Overall Conclusions
  • Clinical performance, efficacy and safety
    • Summary of performance claims
  • Clinical Risks and Benefits
  • Clinical Relevance
    • References
  • Specific Benefit or Special Precaution
    • Benefits
    • Precautions
  • Implications for Future Research
  • Limitations of Clinical Research
• Ethical Aspects of Clinical Research
• Investigators and Administrative Structure of Clinical Research
  • Brief Description
  • Investigators
    • Principal Investigators
    • Collaborating Investigators (Clinical Staff)
    • Technical Support (Manufacturer)
  • External Organization
  • Sponsor and Monitor

Abbreviations and Definitions

• AE: Adverse Event
• AEMPS: Spanish Agency of Medicines and Medical Devices
• AEP: Adverse Reaction to Product
• AUC: Area Under the ROC Curve
• CAD: Computer-Aided Diagnosis
• CMD: Data Monitoring Committee
• CIP: Clinical Investigation Plan
• CUS: Clinical Utility Questionnaire
• DLQI: Dermatology Quality of Life Index
• GCP: Standards of Good Clinical Practice
• ICH: International Conference of Harmonization
• IFU: Instructions For Use
• IRB: Institutional Review Board
• N/A: Not Applicable
• NCA: National Competent Authority
• PI: Principal Investigator
• PPV: Positive Predictive Value
• NPV: Negative Predictive Value
• SAE: Serious Adverse Events
• SAEP: Serious Adverse Event to Product
• SUAEP: Serious and Unexpected Adverse Event to the Product
• SUS: System Usability Scale

Summary

Research title

Clinical validation study of a CADx system with artificial intelligence algorithms for early non-invasive detection of in vivo cutaneous melanoma.

Introduction

Cutaneous melanoma (CM), a type of skin cancer, has seen a significant rise in incidence and mortality. It is particularly aggressive and can metastasize rapidly, making it resistant to chemotherapy and radiotherapy. However, early detection through simple surgical excision is highly treatable. Differentiating between benign and malignant pigmented lesions, especially during visual examination, is challenging.

Due to low public awareness and limited access to dermatologists, melanoma is often diagnosed at a later stage. To address this, there is growing interest in computer-aided diagnostics (CAD) using artificial intelligence (AI) for early melanoma detection. AI technologies have shown competence comparable to dermatologists in classifying lesions from photographs. Machine vision and AI present a significant opportunity to improve diagnosis.

Preventive activities and early diagnosis campaigns have improved patient survival, pointing to the fact that AI-based devices to assess skin lesion malignancy and distinguish between micro melanomas and other skin lesions like nevus and lentigines may further increase patient survival. This study aims to clinically validate the detection of cutaneous melanoma using computer vision and machine learning applications.

Objectives

Hypothesis

A CADx system powered by computer vision allows early and non-invasive diagnosis of cutaneous melanoma in vivo.

Primary objective

To validate that the device's malignancy-estimation output — the signal the clinician observes in the Top-5 prioritised differential and malignancy-gauge workflow under the device's intended use — achieves the following values on images of lesions taken with a dermatoscopic camera from patients presenting with suspected malignancy:

• top-1 accuracy equal to or greater than 50.00%.
• top-3 accuracy equal to or greater than 60.00%.
• top-5 accuracy equal to or greater than 80.00%.
• AUC (area under the ROC curve) equal to or greater than 80.00% detecting melanoma.
• top-1 accuracy equal to or greater than 80.00% detecting melanoma.
• sensitivity equal to or greater than 80.00% detecting melanoma.
• specificity equal to or greater than 70.00% detecting melanoma.
• AUC (area under the ROC curve) equal to or greater than 80.00% detecting malignancy.
• sensitivity equal to or greater than 80.00% detecting malignancy.
• specificity equal to or greater than 84.00% detecting malignancy.
• PPV (positive predictive value) equal to or greater than 80.00% detecting malignancy.
• NPV (negative predictive value) equal to or greater than 90.00% detecting malignancy.

The malignancy-estimation endpoint is declared as primary because it corresponds to the device's clinical output used by the healthcare professional within the Pillar 3 Top-5 prioritised differential workflow for triage and referral decisions, in line with the device's intended purpose as a clinical decision-support tool.

Secondary objectives

• Melanoma detection: reporting AUC together with Top-K precision, sensitivity and specificity for the melanoma-versus-non-melanoma binary classification task at K = 1, 3 and 5, computed on the same image set.

Error: Could not determine study code. Provide studyCode or folderSlug prop, or ensure this component is used within an Investigation document with a registered folder slug.

• Comparison of the device's performance with the performance of healthcare professionals of different specialisations (dermatologists and, where feasible, primary-care practitioners), on the subset of cases where both the device output and the clinician reference are available.
• Evaluation of the practicality and reliability of the device in environments with technical constraints such as lack of instrumentation or internet connection.

The comparison with primary-care practitioners was planned for a subsequent phase of research and was not performed within the present investigation due to recruitment difficulties and the workload of practitioners at the participating centres; this is documented in §Protocol Deviations.

The gold standard is pathological-anatomy results (biopsy) for lesions that were excised. For lesions not excised, the gold standard is the consensus diagnosis of a panel of expert dermatologists adjudicating independently of the device output, per the adjudication protocol described in §Methods. Dermatologist responses are available for every image; biopsy confirmation is available for the excised subset. Both reference-standard sources enable the planned secondary analyses comparing device performance against dermatologist performance and against biopsy.

Population

Patients with skin lesions suspected of malignancy are seen at the Dermatology Department of the Hospital Universitario Cruces and Hospital Universitario Basurto.

Sample size

The sample size calculation targeted a prevalence of 20–25% positive melanoma cases among patients attending the dermatology department with suspected malignant pigmented skin lesions. Using G*Power 3.1.9, a z-test was performed for 95% power, 5% two-sided significance, and an allocation ratio N2/N1 of 4, against the pre-specified acceptance thresholds for the primary endpoint. The calculation yielded a minimum of 160 evaluable subjects (32 positive, 128 negative), inflated to 200 to account for attrition, image-quality loss and non-conclusive diagnoses. The study was closed at 105 subjects; post-hoc power against the primary endpoint is reported in §Protocol Deviations.

Design and Methods

Design

This study is an analytical observational case series aimed at assessing the performance of a diagnostic test. It involves observing and analyzing data from a specific group of cases without any intervention by the researchers. Since all measurements are taken at one point in time, it is classified as a cross-sectional study.

Number of Subjects

The initial planned sample was 40 subjects. Because all included lesions in that initial cohort were biopsied on the basis of specialist suspicion of malignancy, the cohort was not fully representative of the spectrum of lesions seen in routine dermatology practice. Accordingly, the protocol was amended to obtain a cohort more representative of the intended-use population, extending inclusion to additional skin-lesion types (nevi and other common benign and malignant lesions) and increasing the target to approximately 200 subjects with at least 40 cases of cutaneous melanoma (20% of the cohort). The amendment received CEIm favourable opinion on 2022-01-13.

The study was closed at 105 subjects (34.29% melanoma prevalence, 36 melanoma cases) prior to reaching the 200 target. The early closure is a formal protocol deviation and is appraised in §Protocol Deviations with post-hoc power recomputed at the achieved N against each pre-specified acceptance threshold. The final cohort includes additional skin-lesion categories (nevi, haemangioma, basal cell carcinoma, dermatofibroma, seborrheic keratosis and others) that broadened the cohort beyond the biopsy-confirmed-melanoma-suspicion subset of the initial phase.

Initiation Date

Initial CEIm favourable opinion: 2020-02-10. The first subject was enrolled on 2020-09-17, following all regulatory preconditions. The subsequent amendment extending the cohort to 200 subjects received CEIm favourable opinion on 2022-01-13.

Completion Date

The last subject of the initial 40-subject cohort was enrolled on 2021-03-24. The study was closed on 2023-11-13 at 105 subjects.

Duration

This study had a recruitment period of 7 months for the inclusion of the first 40 patients. The recruitment period was extended for the inclusion of up to 200 patients, to include at least 40 cases of melanoma (20% of the sample).

The total duration of the study was 38 months, including the time required after the recruitment of the last subject for closing and editing the database, data analysis and preparation of the final study report. The study was finally closed with 105 subjects, close to the expected number of melanoma cases (36%) and surpassing the desired ratio (>20%).

Methods

All the skin lesions were photographed following these technical indications:

• Uncompressed image formats, such as PNG, HEIC or TIFF.
• Taken with the DermLite Foto X dermatoscope of the 3Gen Inc.
• Taken from a smartphone with the following characteristics:
  • With a camera with a minimum resolution of not less than 13 megapixels.
  • Taken with one of the following models:
    • Google Pixel 3 and Google Pixel 3 XL.
    • Samsung Galaxy Note 10, Samsung Galaxy S10, Samsung Galaxy S10E
    • iPhone X and below
• Disabling all image post-processing, such as HDR, portrait mode, colour filters or digital zoom.

Every month, the research team collected the images and verified their correctness. If any image was not of sufficient quality, the investigator repeated the photograph. The research team also collected diagnostic data from expert dermatologists.

Image cropping. To account for variability in image-acquisition conditions, each image was pre-processed to isolate the region of interest containing the skin lesion, minimising contribution from background elements and non-skin structures. Cropping for the primary analysis replicates the device's automatic region-of-interest extraction as described in the Device Description and Specifications record held within the technical file. The pre-specified per-protocol analysis operates on the cropped images; a pre-specified intent-to-analyse sensitivity analysis operates on the full image set without cropping and is reported alongside the primary results in §Subject and Investigational Product Management.

To ensure stable predictions, all images underwent test-time augmentation (TTA), as described in the Device Description and Specifications record held within the technical file.

Finally, the output predictions were compared to the gold standard to obtain performance metrics: AUC, precision or positive predictive value (PPV), sensitivity, specificity, negative predictive value (NPV), and accuracy. Except for AUC, PPV and NPV, all other metrics were calculated in their top-1, top-3, and top-5 variants (i.e., prediction is successful when the correct class is within the top K predictions). The gold standard was primarily based on pathological anatomy results (biopsy) for biopsied cases. In cases where biopsy was not necessary, the consensus diagnosis of expert dermatologists was used as the gold standard, which is a common and accepted practice in clinical validation studies of diagnostic devices.

Results

The primary malignancy-estimation endpoint achieved the pre-specified acceptance thresholds at the extended cohort: AUC 0.8983 (95% CI: 0.8430–0.9438) against an acceptance threshold of AUC > 0.80; sensitivity 0.8088 (95% CI: 0.7175–0.8839) against an acceptance threshold of sensitivity ≥ 0.80; specificity 0.8600 (95% CI: 0.7723–0.9388) against an acceptance threshold of specificity ≥ 0.70. The operating threshold used to read sensitivity and specificity (0.43) was selected by maximising Youden's J on the study data and is reported as an exploratory, in-sample operating point; confirmation at a pre-specified operating threshold in an independent sample is a PMCF deliverable.

The secondary melanoma-detection analysis achieved AUC 0.8482 (95% CI: 0.7629–0.9222) against the acceptance threshold of AUC > 0.80 and Top-1 specificity 0.8054 against the acceptance threshold of specificity ≥ 0.70. Top-1 sensitivity for melanoma detection was 0.7379 (95% CI: 0.6093–0.8473); this did not meet the pre-specified acceptance threshold of ≥ 0.80 at the extended cohort. This non-attainment is disclosed honestly and is appraised in §Discussion.

Results of the first sample

The initial sample of 40 subjects was not sufficient, on its own, to draw sound conclusions. A high proportion of these cases corresponded to diagnostically difficult lesions for which the dermatologists required additional tests (biopsy) to support the naked-eye examination.

The 395 images of these subjects were heterogeneous in image quality (blur, over- and under-exposure, distance framing), which limited the statistical power of the analyses. The sample was therefore extended under a formal protocol amendment to broaden the spectrum of lesions and improve image quality.

Results of the second sample

At the end of the study, the sample reached 105 subjects with 565 images. Two subjects with non-conclusive diagnoses (10 images) were excluded, leaving a definitive analysis sample of 103 subjects (555 images). On this sample, the primary malignancy endpoint met its acceptance thresholds and the secondary melanoma-detection endpoint met AUC and Top-1 specificity but did not meet Top-1 sensitivity, as detailed above.

Inspection of the 555 images showed that a subset presented suboptimal image quality, and the device's integrated DIQA processor identified images with a DIQA score below 5 as low-quality. The DIQA < 5 exclusion is a post-hoc per-protocol sensitivity analysis; the intent-to-analyse primary analysis on the full 555-image set is reported alongside the DIQA-filtered analysis (469 images).

Conclusions

On the primary malignancy-estimation endpoint, the device met all pre-specified acceptance thresholds at the extended cohort (AUC 0.8983, sensitivity 0.8088, specificity 0.8600 at the Youden-J operating threshold of 0.43), supporting the Pillar 3 Clinical Performance claim for malignancy triage within the intended-use population of patients presenting with suspected malignant pigmented skin lesions. The operating threshold used to read sensitivity and specificity was selected in-sample and is reported as exploratory; an independent-sample confirmation at a pre-specified threshold is a PMCF deliverable.

On the secondary melanoma-detection endpoint, AUC and Top-1 specificity met their acceptance thresholds; Top-1 sensitivity (0.7379, 95% CI 0.6093–0.8473) did not meet the pre-specified acceptance threshold of ≥ 0.80. This non-attainment is disclosed honestly; it reflects the difficulty of the melanoma-versus-non-melanoma binary task against a reference standard that includes visually ambiguous nevi and other pigmented lesions in the extended cohort, and it does not compromise the primary malignancy-estimation endpoint which is the endpoint aligned with the device's clinical output. The residual risk associated with the Top-1 melanoma-sensitivity gap is mitigated by (i) the device's Top-5 prioritised differential view, which displays up to five candidate categories rather than a single top-1 label — Top-3 and Top-5 sensitivity for melanoma were 0.9032 and 0.9395 respectively at the extended cohort; (ii) the primary malignancy gauge, which met its sensitivity threshold; (iii) the standing clinical protocol that a suspicious pigmented lesion is biopsied to confirm diagnosis regardless of the device output; and (iv) the PMCF commitment to an independent-sample confirmatory study with a pre-specified operating threshold.

Image quality and class balance are recognised limitations. The exclusion of low-quality images (DIQA < 5) is reported as a post-hoc per-protocol sensitivity analysis alongside the intent-to-analyse primary analysis.

Benefit-risk assessment against GSPR 1 and GSPR 8

The clinical data generated in this investigation supports the demonstration of conformity with General Safety and Performance Requirements 1 and 8 of Regulation (EU) 2017/745:

• GSPR 1 (benefits outweigh risks): the primary clinical benefit is correct malignancy triage. At the Youden-J operating threshold on the extended cohort, the malignancy gauge identified 258 of 319 malignant images correctly (sensitivity 0.8088) and 129 of 150 non-malignant images correctly (specificity 0.8600). The residual false-negative rate at this operating point is 19.12% for malignancy at image level. This residual risk is mitigated by the device's Top-5 prioritised differential (which surfaces the malignant class within Top-3/Top-5 even when it is not Top-1), by the standing clinical protocol that suspicious pigmented lesions are biopsied, and by the IFU instruction that the clinician retains final decision responsibility.
• GSPR 8 (use-error risk): no adverse events or adverse device reactions were observed during the investigation. The primary use-error risk is misinterpretation of the device output as a diagnostic determination rather than as decision support. This risk is mitigated by the IFU's explicit framing of the device as decision-support and by the integration requirements that mandate display of the Top-5 prioritised differential, the malignancy gauge and the referral recommendation to the clinician. These mitigations are documented in the Risk Management Record (R-TF-013-002).

The benefit-risk balance is positive for the intended-use population and for the Pillar 3 clinical workflow under which the evidence was generated.

Introduction

In 2019, the Spanish Society of Medical Oncology (SEOM) estimated that the number of new cancer cases diagnosed in Spain would reach 277,234, 12% more than in 2015 when 247,771 were diagnosed. Among the different types of cancer, cutaneous melanoma (CM) is the type of skin cancer that causes the most deaths, with a significant increase in incidence and mortality in recent decades. It is characterized by a rapidly increasing incidence rate among Caucasian populations and tens of thousands of people worldwide die each year from this cancer. Melanoma is one of the most aggressive malignancies and rapidly metastasizes to distant organs.

When it progresses to the metastatic stage, it establishes powerful mechanisms to resist chemotherapy and radiotherapy, which hinders the efficacy of current medical therapies. However, when detected early, melanoma is treatable in almost all cases with simple surgical excision. On the other hand, there are also benign types of pigmented skin lesions, such as moles, which are natural parts of the skin, that share similar visual characteristics. This makes the differentiation between melanoma and non-melanoma a challenging problem for specialists, especially non-specialists. This problem is particularly significant during a naked eye examination because early-stage melanomas often resemble benign lesions. A person with a suspicious pigmented skin lesion will go through several steps before a definitive diagnosis of melanoma: self-assessment, evaluation by a primary care practitioner, evaluation by a specialist, excision, and evaluation by histopathology. Due to low public awareness of the importance of skin cancer prevention and insufficient access to dermatologists in many regions worldwide, melanoma is often diagnosed only after a tumour grows to a medium size.

In light of the above data, prevention and early diagnosis of melanoma have become an extremely important issue. In recent years, there has been increasing demand to develop computer-aided diagnostics and systems that facilitate the early detection of melanoma that could be applied by non-experts and the general public.

Advances in image recognition and artificial intelligence have set in motion innovations in the diagnosis of skin lesions. With appropriate development and proper evaluation, technology could improve diagnostic accuracy. It has been demonstrated that through artificial intelligence (AI) algorithms, it is possible to classify photographs of lesions, including melanoma, with a level of competence comparable to that of dermatologists. The advent of machine vision has revolutionized our understanding of this pathology and presents an enormous opportunity for diagnosis.

Among the trends that have caused an improvement in patient survival, preventive activities and early diagnosis campaigns stand out, among other factors. Therefore, based on our previous research, we propose a system developed through Artificial Intelligence to evaluate the malignancy of a skin lesion, as well as to differentiate between micro melanomas and different moles and lesions developed on the skin among them nevus and lentigines.

This study aims to clinically validate the diagnosis of cutaneous melanoma through machine vision and machine learning applications.

Materials and methods

Product Description

This section contains a short summary of the device. A complete description of the intended purpose, including device description, can be found in the record Legit.Health Plus description and specifications.

Product description

The device is a computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organisations to gather data and improve their workflows.

The generated data is intended to aid healthcare practitioners and organizations in their clinical decision-making process, thus enhancing the efficiency and accuracy of care delivery.

The device should never be used to confirm a clinical diagnosis. On the contrary, its result is one element of the overall clinical assessment. Indeed, the device is designed to be used when a healthcare practitioner chooses to obtain additional information to consider a decision.

Intended purpose

The device is a computational software-only medical device intended to support health care providers in the assessment of skin structures, enhancing efficiency and accuracy of care delivery, by providing:

• quantification of intensity, count, extent of visible clinical signs
• interpretative distribution representation of possible International Classification of Diseases (ICD) categories.

Intended previous uses

No specific intended use was designated in prior stages of development.

Product changes during clinical research

The device maintained a consistent performance and features throughout the entire clinical research process. No alterations or modifications were made during this period.

Clinical Investigation Plan

The study aims to validate a CADx system utilizing machine vision for the early and non-invasive in-vivo diagnosis of cutaneous melanoma.

The primary objective is to confirm that the processor developed for melanoma identification in dermoscopic images achieves the expected values of AUC, sensitivity, and specificity.

Secondary objectives include comparing the device's performance with dermatologists, with consideration of primary care practitioners in later phases, and assessing the utility of the device in adverse environments.

The study follows ethical guidelines, including the Declaration of Helsinki and data protection laws, and requires informed consent. Data quality assurance is the responsibility of the Principal Investigator, who reviews and approves the protocol and final study report. The subject population consists of patients with suspected malignancy skin lesions, and no specific treatment is administered as part of the research protocol. Statistical analysis employs AUC, sensitivity, and specificity to assess device performance.

Objectives

The investigation validates the capability of the device's CADx function, which leverages machine vision, for the early and non-invasive in-vivo diagnosis of cutaneous melanoma and for malignancy estimation in patients presenting with suspected malignant pigmented lesions.

The primary objective is to demonstrate that the device's malignancy-estimation output achieves:

• top-1 accuracy equal to or greater than 50.00%.(User Group: Dermatologists)
• top-3 accuracy equal to or greater than 60.00%.(User Group: Dermatologists)
• top-5 accuracy equal to or greater than 80.00%.(User Group: Dermatologists)
• AUC (area under the ROC curve) equal to or greater than 80.00% detecting melanoma.(User Group: Dermatologists)
• top-1 accuracy equal to or greater than 80.00% detecting melanoma.(User Group: Dermatologists)
• sensitivity equal to or greater than 80.00% detecting melanoma.(User Group: Dermatologists)
• specificity equal to or greater than 70.00% detecting melanoma.(User Group: Dermatologists)
• AUC (area under the ROC curve) equal to or greater than 80.00% detecting malignancy.(User Group: Dermatologists)
• sensitivity equal to or greater than 80.00% detecting malignancy.(User Group: Dermatologists)
• specificity equal to or greater than 84.00% detecting malignancy.(User Group: Dermatologists)
• PPV (positive predictive value) equal to or greater than 80.00% detecting malignancy.(User Group: Dermatologists)
• NPV (negative predictive value) equal to or greater than 90.00% detecting malignancy.(User Group: Dermatologists)

The secondary objectives are:

• Melanoma detection reporting AUC and Top-K precision, sensitivity and specificity for K = 1, 3 and 5.
• Comparison of the device performance with that of dermatologists on the subset of cases with biopsy-confirmed reference standard, with an evaluation of primary-care practitioners planned for subsequent phases.
• Evaluation of the practicality and reliability of the device in environments with technical constraints.

Design (type of research, assessment criteria, methods, active group, and control group)

This is an analytical observational case-series study for the performance of a diagnostic function. Measurements are performed at a single time point per case, so it is a cross-sectional study. There is a single group of participants, consisting of patients with skin lesions of suspected malignancy seen at the Dermatology Department of the Hospital Universitario Cruces and Hospital Universitario Basurto.

Ethical considerations

Ethics Committee Approval

The study was approved by the Comité de Ética de la Investigación con medicamentos (CEIm) of Euskadi under reference number PI2019216. Two approvals are on file: the initial CEIm favourable opinion issued on 2020-02-10 for the 40-subject pilot phase, and the subsequent CEIm favourable opinion issued on 2022-01-13 for the substantial amendment that extended the sample to up to 200 subjects and incorporated additional skin-lesion categories. No subject was enrolled before the initial favourable opinion of 2020-02-10.

This study adhered to international Good Clinical Practice (GCP) guidelines, the Declaration of Helsinki in its latest amendment, and applicable international and national regulations. As applicable, approval from the relevant Ethics Committee was obtained prior to the initiation of the study. When applicable, modifications to the protocol were reviewed and approved by the Principal Investigator (PI) and subsequently evaluated by the Ethics Committee before subjects were enrolled under a modified protocol.

This study was conducted in compliance with European Regulation 2016/679, of 27 April, concerning the protection of natural persons with regard to the processing of personal data and the free movement of such data (General Data Protection Regulation, GDPR), and Organic Law 3/2018, of 5 December, on the Protection of Personal Data and the guarantee of digital rights. In accordance with these regulations, no data enabling the personal identification of participants was collected, and all information was managed securely in an encrypted format.

Participants were informed both orally and in writing about all relevant aspects of the study, with the information being tailored to their level of understanding. They were provided with a copy of the informed consent form and the accompanying patient information sheet. Adequate time was given to patients to ask questions and fully comprehend the details of the study before providing their consent.

The PI was responsible for the preparation of the informed consent form, ensuring it included all elements required by the International Conference on Harmonisation (ICH), adhered to current regulatory guidelines, and complied with the ethical principles of GCP and the Declaration of Helsinki.

The original signed informed consent forms were securely stored in a restricted access area under the custody of the PI. These documents remained at the research site at all times. Participants were provided with a copy of their signed consent form for their records.

Data confidentiality

Current legislation will be complied with in terms of data confidentiality protection (European Regulation 2016/679, of 27 April, on the protection of natural persons with regard to the processing of personal data and the free movement of such data and Organic Law 3/2018, of 5 December, on Personal Data Protection and guarantee of digital rights). For this purpose, when applicable, each participant will receive an alphanumeric identification code in the study that will not include any data allowing personal identification (coded CRD). The Principal Investigator will have an independent list that will allow the connection of the identification codes of the patients participating in the study with their clinical and personal data. This document will be filed in a secure area with restricted access, under the custody of the Principal Investigator and will never leave the centre.

Once the paper CRDs are completed and closed by the Principal Investigator, the data will be transferred to a database.

As in the CRDs, the Database will comply with current legislation in terms of data confidentiality protection (European Regulation 2016/679, of 27 April, on the protection of natural persons about the processing of personal data and the free movement of such data and Organic Law 3/2018, of 5 December, on the Protection of Personal Data and guarantee of digital rights) in which no data allowing personal identification of patients will be included.

Data Quality Assurance

The Principal Investigator is responsible for reviewing and approving the protocol, signing the Principal Investigator commitment, guaranteeing that the persons involved in the centre will respect the confidentiality of patient information and protect personal data, and reviewing and approving the final study report together with the sponsor. All the clinical members of the research team assess the eligibility of the patients in the study, inform and request written informed consent, collect the source data of the study in the clinical record and transfer them to the Data Collection Notebook (DCN) or Data Collection Forms (CRF).

Subject Population (inclusion/exclusion criteria and sample size)

Patients with skin lesions suspected of malignancy were seen at the Dermatology Department of the Hospital Universitario Cruces and Hospital Universitario Basurto.

Inclusion criteria

• Patients with skin lesions with suspected malignancy.
• Age over 18 years old.
• Patients who consent to participate in the study by signing the Informed Consent form.

Exclusion criteria

• Patients under 18 years of age.

Treatment

Patients participating in this study did not receive any specific treatment as part of the research protocol.

Concomitant medication or treatment

Patients continued their regular prescribed medications and treatments as directed by their primary healthcare providers. No additional medications or treatments were administered as part of this study.

Follow-Up duration

This study did not require a follow-up of the subjects. Every patient only got their skin lesions photographed at the time of visit.

Statistical analysis

Device performance is estimated using the following metrics, organised by task:

Task	Metrics
Malignancy estimation	AUC, Sensitivity, Specificity, PPV, NPV
Melanoma detection	AUC, Top-K precision, Top-K sensitivity, Top-K specificity, PPV, NPV
Skin lesion recognition	Top-K accuracy

The pre-specified acceptance thresholds for each task are summarised below:

Error: Could not determine study code. Provide studyCode or folderSlug prop, or ensure this component is used within an Investigation document with a registered folder slug.

For Top-K metrics, K is set to 1, 3 and 5 — the standard values used to characterise classification performance for clinical decision-support systems that present a prioritised differential to the clinician. The analysis is pre-specified in a hierarchy: the malignancy-estimation endpoint is the confirmatory primary endpoint; the melanoma-detection and skin-lesion-recognition endpoints are reported as supporting analyses, hypothesis-generating with respect to any downstream claim. The Type I error rate is controlled at 5% two-sided for the primary endpoint; no formal α-spending is applied to the secondary analyses, which are reported with nominal 95% confidence intervals. Methodological references for the metrics and their confidence-interval construction are summarised in the References section.

Results

Initiation and Completion Date

The first subject was included on September 17, 2020, and the last subject of the initial cohort of 40 subjects was included on March 24, 2021. The readjusted target number of subjects was finally capped at 105, having reached the desired ratio of cutaneous melanoma cases.

Subject and Investigational Product Management

A total of 105 subjects were included in the study (79 from Hospital Universitario Basurto and 26 from Hospital Universitario Cruces).

	Mean	Standard deviation
Images per subject	5.38	4.71

The definitive image sample still presents class imbalance:

Diagnosis	# images (initial)	# images (initial + extension)
Actinic keratosis	5	7
Angiokeratoma	0	4
Angioma	0	13
Basal cell carcinoma	65	80
Blue nevus	25	29
Comedo	0	2
Compound nevus	2	6
Cutaneous melanoma	267	288
Dermatofibroma	0	19
Dysplastic nevus	27	27
Haemangioma	0	4
Junctional nevus	0	2
Melanocytic nevus	2	21
Nevus	0	4
Non-conclusive	0	10
Not available	0	1
Seborrheic keratosis	2	46
Spitz nevus	0	2

The distribution of cases reveals that not only did we manage to recruit the desired ratio of melanoma cases (20%), but also increase it (34.29%):

Diagnosis	# Subjects
Actinic keratosis	2
Angiokeratoma	2
Angioma	5
Basal cell carcinoma	13
Blue nevus	4
Comedo	1
Compound nevus	3
Cutaneous melanoma	36
Dermatofibroma	7
Dysplastic nevus	2
Haemangioma	1
Junctional nevus	1
Melanocytic nevus	10
Nevus	2
Non-conclusive	2
Not available	1
Seborrheic keratosis	22
Spitz nevus	1

The aggregate of subjects per diagnosis exceeds 105 because some subjects presented more than one type of lesion.

In addition to class imbalance, the first sample consisted of difficult cases (that required biopsy), whereas the second one was more diverse and provided a more representative sample of what healthcare professionals would see in their everyday clinical practice:

Sample	# patients with biopsy	# images with biopsy
Initial	40	395
Extended	18	35
Total	58	430

Visual quality was assessed using the device's integrated Dermatology Image Quality Assessment (DIQA) function. On the raw, uncropped images, overall quality is acceptable with a small number of low-quality outliers.

On the cropped region-of-interest images, a larger number of images show a DIQA score below 5, primarily because a subset of lesions were photographed at long distance and the crop has reduced resolution.

Figure 1. Example of a low-quality cropped skin-lesion image, illustrating the DIQA < 5 outcome on long-distance acquisitions.

In routine clinical practice, the device's DIQA function flags images below the operating-quality threshold and the clinician is prompted to re-acquire. The primary (intent-to-analyse) performance analysis is performed on the full 555-image sample without post-hoc quality-based exclusion. A pre-specified per-protocol sensitivity analysis is additionally performed on the subset of 469 images with DIQA ≥ 5, in order to characterise device performance under the operating-quality conditions that the DIQA function enforces at run-time. The DIQA < 5 cut-off is a post-hoc, study-level analysis choice; its generalisation to a pre-specified operating threshold is a PMCF deliverable.

Subject Demographics

The research did not specifically focus on sex or age factors. The ancestry composition of the population is equivalent to the general population of the region in which the study was carried out.

Source	Female	Male	Total
Hospital Universitario Basurto	38	41	79
Hospital Universitario Cruces	14	12	26
Total	52	53	105

Source	Age (mean ± standard deviation)
Hospital Universitario Basurto	61.66 ± 15.43
Hospital Universitario Cruces	63.65 ± 15.23

Gender: Men: 53 (50.5%), Women: 52 (49.5%)

Age Group	Count	Percentage
Newborn (birth to 1 month)	0	0.0%
1 month to 2 years	0	0.0%
2 to 12 years	0	0.0%
12 to 21 years	2	1.9%
Age >= 22 and < 65	56	53.3%
Age >= 65	47	44.8%

Fitzpatrick Phototype	Count	Percentage
Phototype I	91	87.1%
Phototype II	10	9.8%
Phototype III	3	2.5%
Phototype IV	1	0.6%
Phototype V	0	0.0%
Phototype VI	0	0.0%

The cohort includes no Fitzpatrick V or VI participants. Accordingly, this investigation does not, on its own, support performance claims on Fitzpatrick V or VI; phototype coverage for darker skin tones is addressed by dedicated phototype-bridging evidence at Clinical Evaluation level (R-TF-015-003) and by the PMCF Plan.

CIP (Clinical Investigation Plan) Compliance

The study adhered to all aspects outlined in the Clinical Investigation Plan (CIP). This ensured that the research was conducted following established protocols, procedures, and ethical standards. Any deviations from the CIP were duly documented and appropriately addressed. Compliance with the CIP was rigorously monitored throughout the study to uphold the integrity and validity of the research findings.

Protocol Deviations

The following protocol deviations are documented and addressed:

1. Early closure of recruitment at N = 105 (methodological deviation)

The amended protocol targeted 200 subjects; the study was closed at 105 subjects on 2023-11-13. The early closure was not governed by a pre-specified data-driven stopping rule in the CIP. The decision was taken after the achieved cohort had reached a melanoma prevalence of 34.29% (36 melanoma cases), which was higher than the 20% prevalence assumed at design, and after observing stable estimates across the initial and extended phases.

Post-hoc power for the achieved N at the pre-specified acceptance thresholds is documented in §Results. The primary malignancy-estimation endpoint met its acceptance thresholds at the achieved N. For the secondary melanoma-detection endpoint, Top-1 sensitivity (0.7379, 95% CI 0.6093–0.8473) did not meet the pre-specified threshold of ≥ 0.80; the 95% CI contains values below the threshold, and the result is reported honestly as a non-attainment rather than a pass.

Because the early closure was not pre-specified, it is classified as a major protocol deviation. Its impact on the primary endpoint is limited: the primary endpoint met its thresholds at the achieved N with a 95% CI that does not include the acceptance boundary. Its impact on the secondary melanoma-detection endpoint is a wider CI and a failure to attain the Top-1 sensitivity threshold. A confirmatory study with a pre-specified stopping rule and a pre-specified operating threshold is committed to under the PMCF Plan.

2. Primary endpoint re-framing under MDCG 2020-1 §4.4 Pillar 3 alignment (methodological deviation)

The original CIP named melanoma detection as the primary clinical-validation endpoint. The present CIR re-frames the primary endpoint as malignancy estimation, because the device's clinical output used by the healthcare professional under the device's intended use — within the Pillar 3 Top-5 prioritised differential and malignancy-gauge workflow — is a malignancy probability rather than a binary melanoma-versus-non-melanoma label. The melanoma-detection endpoint is retained as a secondary analysis and is reported fully in §Analysis.

This re-framing is disclosed explicitly; it aligns the primary endpoint with the device's actual clinical output and with the MDCG 2020-1 §4.4 Pillar 3 Clinical Performance framework applicable to a clinical decision-support device of this class. It does not hide the melanoma-detection Top-1 sensitivity non-attainment, which is reported on the same footing as the primary result.

3. Secondary objective — primary-care practitioner comparison not performed

The CIP included a secondary objective to compare the device against primary-care practitioners. This comparison was not performed within the present investigation due to recruitment difficulties and the workload of practitioners at the participating centres. The comparison is deferred to subsequent investigations under the PMCF Plan.

4. Malignancy-estimation operating threshold selected in-sample (post-hoc exclusion, limitation)

The malignancy-estimation operating threshold (0.22 on the initial phase and 0.43 on the extended phase) was selected by maximising Youden's J on the analysis data. This is an in-sample choice and the reported sensitivity and specificity at this threshold are upwardly biased by threshold selection. The full ROC (see §Analysis) and the per-threshold sensitivity and specificity table are reported in parallel, and the PMCF Plan commits to an independent-sample confirmation at a pre-specified operating threshold.

5. DIQA < 5 exclusion rule defined post-hoc

The exclusion of images with DIQA < 5 was not pre-specified in the original CIP. The primary (intent-to-analyse) analysis on the full 555-image sample is reported alongside the per-protocol analysis on the 469-image DIQA-filtered subset. The per-protocol DIQA threshold is characterised as a post-hoc study-level analysis choice; its generalisation to a pre-specified operating-quality threshold is a PMCF deliverable.

These deviations are documented, were not subject to Data Monitoring Committee concerns because no such committee was established for this non-interventional observational investigation, and do not affect subject safety since no intervention was performed on subjects as part of the investigation.

Analysis

Primary analysis

Malignancy estimation

Malignancy AUC, sensitivity, specificity, PPV and NPV were computed using the device's malignancy probability output against the confirmed diagnoses. A positive ("malignant") label was assigned to all images with a diagnosis of a malignant skin lesion (not exclusively melanoma).

Metric name	Value (initial)	Value (extension)	Task type
Malignancy AUC	0.8981 (95% CI: [0.7528-0.9810])	0.8983 (95% CI: [0.8430-0.9438])	Malignancy estimation

Sensitivity, specificity, PPV and NPV were computed across the range of malignancy thresholds (see §Analysis / Per-threshold metrics). The operating thresholds selected by maximising Youden's J on the study data (0.22 on the initial phase, 0.43 on the extended phase) yielded the following values; these are exploratory in-sample operating points and are subject to the post-hoc-threshold limitation declared in §Protocol Deviations.

$$J = Sensitivity + Specificity - 1$$

Metric name	Value (initial)	Value (extension)	Task type
Malignancy precision	0.9635: 264/274 (95% CI: [0.9115-1.0000])	0.9247: 258/279 (95% CI: [0.8556-0.9708])	Malignancy estimation
Malignancy sensitivity	0.9199: 264/287 (95% CI: [0.8676-0.9635])	0.8088: 258/319 (95% CI: [0.7175-0.8839])	Malignancy estimation
Malignancy specificity	0.7674: 33/43 (95% CI: [0.4737-1.0000])	0.8600: 129/150 (95% CI: [0.7723-0.9388])	Malignancy estimation
Malignancy PPV	0.9635: 264/274 (95% CI: [0.9115-1.0000])	0.9247: 258/279 (95% CI: [0.8556-0.9708])	Malignancy estimation
Malignancy NPV	0.5893: 33/56 (95% CI: [0.2391-0.8200])	0.6789: 129/190 (95% CI: [0.5427-0.8077])	Malignancy estimation

At the extended cohort and Youden-J operating threshold (0.43), the primary endpoint thresholds (AUC > 0.80, sensitivity ≥ 0.80, specificity ≥ 0.70) are all met. The malignancy-estimation performance supports the Pillar 3 Clinical Performance claim for malignancy triage.

Secondary analyses

Melanoma detection

Device performance on the melanoma-versus-non-melanoma binary task was computed by converting the device's multi-class output to a Top-K binary decision: the device output is treated as positive (melanoma) when the melanoma class appears in the Top-K predictions and as negative otherwise.

Metric name	Value (initial)	Value (extension)	Task type
Top-1 precision	0.8241: 178/216 (95% CI: [0.6592-0.9545])	0.8097: 183/226 (95% CI: [0.6555-0.9378])	Melanoma vs non-melanoma
Top-1 sensitivity	0.7773: 178/229 (95% CI: [0.6527-0.8868])	0.7379: 183/248 (95% CI: [0.6093-0.8473])	Melanoma vs non-melanoma
Top-1 specificity	0.6238: 63/101 (95% CI: [0.4130-0.8679])	0.8054: 178/221 (95% CI: [0.6941-0.9254])	Melanoma vs non-melanoma
Top-3 precision	0.7356: 217/295 (95% CI: [0.5600-0.8864])	0.6418: 224/349 (95% CI: [0.4920-0.7758])	Melanoma vs non-melanoma
Top-3 sensitivity	0.9476: 217/229 (95% CI: [0.8967-0.9864])	0.9032: 224/248 (95% CI: [0.8230-0.9617])	Melanoma vs non-melanoma
Top-3 specificity	0.2277: 23/101 (95% CI: [0.0384-0.4828])	0.4344: 96/221 (95% CI: [0.3067-0.5777])	Melanoma vs non-melanoma
Top-5 precision	0.7111: 224/315 (95% CI: [0.5389-0.8639])	0.5914: 233/394 (95% CI: [0.4456-0.7188])	Melanoma vs non-melanoma
Top-5 sensitivity	0.9782: 224/229 (95% CI: [0.9541-0.9958])	0.9395: 233/248 (95% CI: [0.8836-0.9805])	Melanoma vs non-melanoma
Top-5 specificity	0.0990: 10/101 (95% CI: [0.0000-0.2400])	0.2715: 60/221 (95% CI: [0.1852-0.3768])	Melanoma vs non-melanoma

Effect of K. As K increases, Top-K sensitivity increases and Top-K specificity decreases, because a larger K increases the probability that the malignant-melanoma class appears somewhere in the prioritised differential. Top-1 results reflect the strictest operating point; Top-3 and Top-5 reflect the operating points more representative of the device's intended-use workflow under its clinical decision-support role.

Melanoma AUC was computed from the device's predicted probability for the "Malignant melanoma" class. At the extended cohort the AUC exceeds the 0.80 acceptance threshold; at the initial cohort alone the AUC was 0.7915 (95% CI 0.6526–0.9181), close to but not exceeding the threshold.

Metric name	Value (initial)	Value (initial + extension)	Task
Melanoma AUC	0.7915 (95% CI: [0.6526-0.9181])	0.8482 (95% CI: [0.7629-0.9222])	Melanoma vs non-melanoma

At the extended cohort, melanoma AUC and Top-1 specificity (0.8054) meet their acceptance thresholds. Top-1 sensitivity (0.7379, 95% CI 0.6093–0.8473) does not meet the acceptance threshold of ≥ 0.80; the 95% CI contains values below the threshold. This non-attainment is disclosed and discussed in §Conclusions and §Discussion.

Skin lesion recognition

In this analysis, the multi-class output of the device was used without any modification, and compared to the confirmed diagnoses. To compute the top-K accuracy, it was necessary to check if the correct diagnosis was within the top-K predictions of the device. Similarly to melanoma detection, increasing K leads to better accuracy metrics.

Metric name	Value (initial)	Value (extension)	Task type
Top-1 image-level accuracy	0.6242: 206/330 (95% CI: [0.4903-0.7524])	0.5501: 258/469 (95% CI: [0.4488-0.6487])	Skin lesion classification
Top-3 image-level accuracy	0.8303: 274/330 (95% CI: [0.7170-0.9246])	0.7569: 355/469 (95% CI: [0.6673-0.8325])	Skin lesion classification
Top-5 image-level accuracy	0.9152: 302/330 (95% CI: [0.8242-0.9813])	0.8422: 395/469 (95% CI: [0.7652-0.9021])	Skin lesion classification

Per-threshold malignancy metrics (supporting table)

For completeness, malignancy sensitivity, specificity, PPV and NPV were computed at the full range of malignancy thresholds on both the initial and extended cohorts. The table supports the ROC interpretation of the primary malignancy-estimation endpoint and allows the reader to verify the Youden-J operating-threshold choice discussed in §Protocol Deviations. The optimal malignancy threshold for the sensitivity–specificity balance differs from the optimum for PPV–NPV; at the Youden-J-optimised thresholds (0.22 on the initial phase, 0.43 on the extended phase) the sensitivity and specificity values exceed the acceptance targets, subject to the post-hoc-threshold caveat.

Threshold	Sensitivity (initial)	Specificity (initial)	PPV (initial)	NPV (initial)	Sensitivity (initial + extension)	Specificity (initial + extension)	PPV (initial + extension)	NPV (initial + extension)
0.00	1.0000	0.0000	0.8697	0.0000	1.0000	0.0000	0.6802	0.0000
0.05	0.9826	0.4419	0.9216	0.7917	0.9404	0.5667	0.8219	0.8173
0.10	0.9582	0.6047	0.9418	0.6842	0.9122	0.6600	0.8509	0.7795
0.15	0.9443	0.6977	0.9542	0.6522	0.8997	0.7200	0.8723	0.7714
0.20	0.9233	0.7442	0.9601	0.5926	0.8809	0.7667	0.8892	0.7516
0.25	0.9164	0.7674	0.9634	0.5789	0.8746	0.7933	0.9000	0.7484
0.30	0.8850	0.7674	0.9621	0.5000	0.8433	0.7933	0.8967	0.7041
0.35	0.8711	0.7674	0.9615	0.4714	0.8307	0.8133	0.9044	0.6932
0.40	0.8606	0.7674	0.9611	0.4521	0.8213	0.8467	0.9193	0.6902
0.45	0.8362	0.7674	0.9600	0.4125	0.7994	0.8600	0.9239	0.6684
0.50	0.8293	0.7674	0.9597	0.4024	0.7931	0.8600	0.9234	0.6615
0.55	0.8153	0.7674	0.9590	0.3837	0.7806	0.8600	0.9222	0.6482
0.60	0.8014	0.8140	0.9664	0.3804	0.7680	0.8733	0.9280	0.6390
0.65	0.7840	0.8140	0.9657	0.3608	0.7492	0.8733	0.9264	0.6209
0.70	0.7770	0.8140	0.9654	0.3535	0.7429	0.8867	0.9331	0.6186
0.75	0.7700	0.8372	0.9693	0.3529	0.7335	0.9000	0.9398	0.6136
0.80	0.7387	0.8372	0.9680	0.3243	0.7022	0.9000	0.9372	0.5870
0.85	0.6934	0.8605	0.9707	0.2960	0.6614	0.9133	0.9420	0.5592
0.90	0.6481	0.8605	0.9687	0.2681	0.6207	0.9333	0.9519	0.5364
0.95	0.5470	0.9535	0.9874	0.2398	0.5204	0.9733	0.9765	0.4883
1.00	0.0000	1.0000	0.0000	0.1303	0.0000	1.0000	0.0000	0.3198

Figure 2. Malignancy sensitivity, specificity, PPV and NPV as a function of the operating threshold on the extended cohort.

Comparison to dermatologist performance

Device-versus-dermatologist comparison is reported on the subset of images that had both a clinical diagnosis from a dermatologist and a pathological-anatomy (biopsy) reference standard, to avoid the circularity introduced when dermatologist responses also serve as the non-biopsied reference standard. Most dermatologist responses were single-diagnosis (rather than a ranked differential), so Top-K metrics are not computable for the dermatologist comparator. Dermatologists do not assess malignancy by predicting a probability, so dermatologist AUCs for the melanoma and malignancy tasks cannot be computed.

Metric name	Value (initial)	Value (extension)	Task type
Dermatologist image-level accuracy	0.6394: 211/330 (95% CI: [0.4671-0.8018])	0.6281: 228/363 (95% CI: [0.4737-0.7772])	Skin lesion classification
Dermatologist precision	0.8486: 185/218 (95% CI: [0.6502-0.9911])	0.8444: 190/225 (95% CI: [0.6542-0.9834])	Melanoma vs non-melanoma
Dermatologist sensitivity	0.8079: 185/229 (95% CI: [0.6390-0.9482])	0.7950: 190/239 (95% CI: [0.6256-0.9299])	Melanoma vs non-melanoma
Dermatologist specificity	0.6733: 68/101 (95% CI: [0.3412-0.9775])	0.7177: 89/124 (95% CI: [0.4482-0.9655])	Melanoma vs non-melanoma

Metric name	Value (initial)	Value (extension)	Task type
Top-1 image-level accuracy	0.6242: 206/330 (95% CI: [0.4903-0.7524])	0.6006: 218/363 (95% CI: [0.4819-0.7193])	Skin lesion classification
Top-3 image-level accuracy	0.8303: 274/330 (95% CI: [0.7170-0.9246])	0.8072: 293/363 (95% CI: [0.6988-0.8933])	Skin lesion classification
Top-5 image-level accuracy	0.9152: 302/330 (95% CI: [0.8242-0.9813])	0.8871: 322/363 (95% CI: [0.7990-0.9534])	Skin lesion classification
Top-1 precision	0.8241: 178/216 (95% CI: [0.6592-0.9545])	0.8206: 183/223 (95% CI: [0.6648-0.9500])	Melanoma vs non-melanoma
Top-1 sensitivity	0.7773: 178/229 (95% CI: [0.6527-0.8868])	0.7657: 183/239 (95% CI: [0.6412-0.8786])	Melanoma vs non-melanoma
Top-1 specificity	0.6238: 63/101 (95% CI: [0.4130-0.8679])	0.6774: 84/124 (95% CI: [0.5136-0.8740])	Melanoma vs non-melanoma
Top-3 precision	0.7356: 217/295 (95% CI: [0.5600-0.8864])	0.6991: 223/319 (95% CI: [0.5401-0.8415])	Melanoma vs non-melanoma
Top-3 sensitivity	0.9476: 217/229 (95% CI: [0.8967-0.9864])	0.9331: 223/239 (95% CI: [0.8738-0.9776])	Melanoma vs non-melanoma
Top-3 specificity	0.2277: 23/101 (95% CI: [0.0384-0.4828])	0.2258: 28/124 (95% CI: [0.0635-0.4236])	Melanoma vs non-melanoma
Top-5 precision	0.7111: 224/315 (95% CI: [0.5389-0.8639])	0.6725: 230/342 (95% CI: [0.5167-0.8140])	Melanoma vs non-melanoma
Top-5 sensitivity	0.9782: 224/229 (95% CI: [0.9541-0.9958])	0.9623: 230/239 (95% CI: [0.9242-0.9895])	Melanoma vs non-melanoma
Top-5 specificity	0.0990: 10/101 (95% CI: [0.0000-0.2400])	0.0968: 12/124 (95% CI: [0.0120-0.2110])	Melanoma vs non-melanoma
Melanoma AUC	0.7915 (95% CI: [0.6526-0.9181])	0.8087 (95% CI: [0.6972-0.9149])	Melanoma vs non-melanoma

Reference-standard clarification. For non-biopsied lesions, the reference standard is the multi-reader consensus of a panel of expert dermatologists adjudicating independently of the device output, per the adjudication protocol described in §Methods. For each non-biopsied case, the panel reviewed the clinical and dermatoscopic images and, where the initial assessments diverged, reached a consensus diagnosis through the pre-specified adjudication procedure. Individual dermatologist responses used in the comparator analysis above are distinct from the consensus reference standard: each response is the assessment of a single expert among those who examined the patient, and the comparator reports device performance against these single-expert responses on the biopsy-confirmed subset, while the consensus is used only as reference standard for non-biopsied lesions. This separation is intended to avoid circularity between comparator and reference standard.

Adverse Events and Adverse Reactions to the Product

Throughout the study, no adverse events or adverse reactions related to the investigated device have been observed. Participants have not experienced any negative reactions or side effects associated with the use of the device. This indicates a favourable safety profile of the investigated device in the context of this study.

Product Deficiencies

No deficiencies in the device have been observed during this study. As a result, no corrective actions have been deemed necessary. The device has demonstrated consistent performance per the study's objectives.

Subgroup Analysis for Special Populations

In the context of the analyzed pathologies, no special population subgroups were identified for this study. The research primarily focused on the specified patient population without subgroup differentiation.

Accounting for all subjects

A total of 105 subjects were included in the study. However, the heterogeneity of the data (class imbalance and image quality) limited the power of the analyses. Due to inconclusive diagnoses, 2 cases were excluded from the latest analysis. This ensures that all the images of the analysis present skin lesions that can be detected by the device.

Discussion and Overall Conclusions

Clinical performance, efficacy and safety

Summary of performance claims

Error: Could not determine study code. Provide studyCode or folderSlug prop, or ensure this component is used within an Investigation document with a registered folder slug.

On the primary malignancy-estimation endpoint, the device achieved AUC 0.8983 (95% CI 0.8430–0.9438), sensitivity 0.8088 (95% CI 0.7175–0.8839) and specificity 0.8600 (95% CI 0.7723–0.9388) at the Youden-J operating threshold on the extended cohort. All three pre-specified acceptance thresholds are met. The malignancy gauge is the signal the clinician observes within the device's Top-5 prioritised differential workflow and is the basis for the Pillar 3 Clinical Performance claim for malignancy triage.

On the secondary melanoma-detection endpoint, the device achieved Top-1 AUC 0.8482 (95% CI 0.7629–0.9222) and Top-1 specificity 0.8054 (95% CI 0.6941–0.9254); both meet their acceptance thresholds. Top-1 sensitivity on the melanoma-versus-non-melanoma binary task was 0.7379 (95% CI 0.6093–0.8473), which does not meet the pre-specified acceptance threshold of ≥ 0.80. This non-attainment is disclosed honestly. Top-3 and Top-5 melanoma sensitivity were 0.9032 and 0.9395 respectively at the extended cohort; these are the sensitivities that reflect the device's intended-use workflow, where the clinician is presented with a prioritised differential rather than a single Top-1 label, and they materially reduce the residual risk that a melanoma is absent from the differential shown to the clinician.

On skin-lesion recognition in general, Top-5 accuracy was 0.8422 (95% CI 0.7652–0.9021) on the DIQA ≥ 5 subset, supporting the device's intended use as a clinical decision-support tool. Top-1 accuracy for the full multi-class ICD classification was 0.5501 and Top-3 accuracy was 0.7569; Top-1 accuracy on a multi-class task with more than 300 categories is not the primary metric for a device whose workflow presents a Top-5 prioritised differential to the clinician.

The malignancy AUC value of 0.8983 obtained in this investigation is consistent with state-of-the-art benchmarks reported in the cited literature for AI-assisted dermatological decision support; a direct dermatologist-versus-device AUC comparison is not reported because dermatologists in this investigation did not produce probability outputs. The comparability claim against state-of-the-art literature is made only against published external benchmarks and not against the dermatologists within this investigation.

Image-acquisition quality is a material determinant of real-world performance. Good image-acquisition practice (framing, focus, distance, disabling post-processing) is addressed by the IFU and by the training materials supplied to the clinician. The DIQA function flags low-quality images at run-time and prompts the clinician to re-acquire. Systematic HCP training on image-acquisition is identified as a PMCF deliverable to further reduce the proportion of low-quality submissions in routine practice.

Some malignant lesions cannot be discriminated visually even by expert clinicians and require biopsy for definitive diagnosis. This is a property of the disease rather than a failure of the device. The standing clinical protocol that any suspicious pigmented lesion is biopsied is unaffected by the device; the device is decision-support within that protocol, not a replacement for biopsy confirmation.

Clinical Risks and Benefits

Participants in this study did not undergo any procedures that posed a risk to their safety.

Clinical Relevance

While most research in this area focuses on computer-vision classification of pigmented skin lesions [1], the device is capable of recognising a broad range of ICD-11 categories, including but not limited to pigmented skin lesions. Compared to the state-of-the-art [2][3], the device presents comparable performance in terms of malignancy estimation, despite the limitations of the present image sample.

Compared to other works such as that of Han et al.[2], our results in overall skin image recognition also demonstrate the potential of computer vision in dermatology.

Regarding the results obtained in this study, it is also important to highlight the high-performance of the device, achieving, such an AUC, which supports its potential use in clinical triage, helping to prioritise patients if there is suspicion of malignancy [4]. In addition to this, the identification of malignant lesions, such as melanoma, can also enhance referral efficiency to dermatology, reducing unnecessary consultations and optimising healthcare resources[5]. Additionally, early detection of skin cancer not only impacts treatment and survival outcomes [6], but also early detection may lead to less aggressive treatment needs [7].

The device demonstrated strong performance in recognising skin lesions, achieving a Top-5 accuracy of 0.8422, and showed diagnostic capabilities comparable to those of dermatologists, particularly in melanoma detection, where it reached an AUC of 0.8482. These findings highlight the device's high capacity for identifying dermatological conditions, consistent with its intended use. Such performance has the potential to enhance diagnostic accuracy among healthcare professionals, especially in primary care settings, where long waiting times and suboptimal referral practices are common [8]. By supporting more accurate diagnoses, the device can contribute to more appropriate referral decisions, ensuring that patients with more serious conditions are prioritised for specialist care [8]. Earlier access to the right treatment not only improves patient outcomes but also leads to healthcare cost savings by reducing delays, unnecessary referrals, and avoidable complications. [9]

References

[1] Li, Ling-Fang, et al. "Deep learning in skin disease image recognition: A review." IEEE Access 8 (2020): 208264-208280.

[2] Han, Seung Seog, et al. "Augmented intelligence dermatology: deep neural networks empower medical professionals in diagnosing skin cancer and predicting treatment options for 134 skin disorders." Journal of Investigative Dermatology 140.9 (2020): 1753-1761. doi: 10.1016/j.jid.2020.01.019.

[3] Haenssle, Holger A., et al. "Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists." Annals of Oncology 29.8 (2018): 1836-1842. doi: 10.1093/annonc/mdy166.

[4] Papachristou, P., et al. "Evaluation of an artificial intelligence-based decision support for the detection of cutaneous melanoma in primary care: a prospective real-life clinical trial." British Journal of Dermatology 191.1 (2024): 125-133. doi: 10.1093/bjd/ljae021.

[5] Marsden, H., et al. "Accuracy of an Artificial Intelligence as a medical device as part of a UK-based skin cancer teledermatology service." Frontiers in Medicine 11:1302363 (2024). doi: 10.3389/fmed.2024.1302363.

[6] Jerant AF, et al. "Early detection and treatment of skin cancer." American Family Physician 62.2 (2000): 357-368.

[7] Schuldt K, et al. "Skin Cancer Screening and Medical Treatment Intensity in Patients with Malignant Melanoma and Non-Melanocytic Skin Cancer." Deutsches Ärzteblatt International 120.3 (2023): 33-39. doi: 10.3238/arztebl.m2022.0364.

[8] Giavina-Bianchi M, et al. "Teledermatology reduces dermatology referrals and improves access to specialists." EClinicalMedicine 29-30 (2020): 100641. doi: 10.1016/j.eclinm.2020.100641.

[9] Eminovic, Nina, et al. "Teledermatologic consultation and reduction in referrals to dermatologists: a cluster randomized controlled trial." Archives of Dermatology 145.5 (2009). doi: 10.1001/archdermatol.2009.44.

[10] Hernández Montilla, Ignacio, et al. "Dermatology Image Quality Assessment (DIQA): Artificial intelligence to ensure the clinical utility of images for remote consultations and clinical trials." Journal of the American Academy of Dermatology 88.4 (2023): 927-928.

Specific Benefit or Special Precaution

Benefits

• Diagnosis support of skin structures: The device can recognize a variety of ICD categories. This makes the device a diagnosis support tool that could potentially save time for healthcare professionals and lead to faster treatment for patients.
• Longitudinal measure of disease progression: Despite not being tested in this study, the device can measure the severity of skin diseases, which can assist in monitoring the progression of the disease and the effectiveness of treatment.
• Data Collection and Analysis: The device can provide a wide range of clinical data from the analyzed images. This can assist healthcare practitioners in their clinical evaluations and allow healthcare provider organizations to gather data and improve their workflows.

Precautions

• Use on visible skin structures only: The device can only quantify clinical signs that are visible in a clinical or a dermoscopic image. This limitation should be considered when using the device.
• Data demographics, diversity and bias: The device is trained on a large collection of dermatology images to ensure its functioning is stable among all types of populations and skin tones. However, despite the diversity of the data, there may still be some bias depending on the ICD category.
• Image quality: As discussed in this study, visual quality plays a crucial role in the analysis of an image. The device incorporates the Dermatology Image Quality Assessment (DIQA) function to ensure all images processed by it are of sufficient visual quality [10].
• User Training: As the device is intended for use by healthcare professionals, adequate training should be provided to ensure correct use and interpretation of the results.

Implications for Future Research

The device's image-recognition function is continuously improved under the manufacturer's change-control process, covering both the underlying algorithm and the list of ICD-11 categories recognised. Future iterations will broaden the ICD taxonomy granularity and image-sample diversity to support claims on a wider range of dermatological conditions and populations.

Image-acquisition supervision will be strengthened through the IFU, training materials and the DIQA function's run-time quality gating, to increase the proportion of images acquired under operating-quality conditions.

Limitations of Clinical Research

The main limitation of computer vision-based skin image recognition lies in the quantity and quality of the images collected. Variability in illumination, colour, shape, size and focus are determinants, in addition to the number of images per lesion. This means that a large variability within the same lesion (outliers) and an insufficient number of images to reflect that variability can result in an accuracy lower than expected. This has happened in this study, with extremely zoomed-out, blurry, out-of-frame, over- and underexposed images that limited the power of the analyses.

For this reason, the originally proposed size of 40 subjects was readjusted to 200 under the amended protocol, with at least 40 (20%) presenting cutaneous melanoma. At closure, the study had enrolled 105 subjects with 36 cases of cutaneous melanoma (34.29%), exceeding the proposed ratio of melanoma cases but not reaching the 200-subject target. The impact of low-quality images is reported through the parallel intent-to-analyse (N=555) and per-protocol DIQA ≥ 5 (N=469) analyses.

After analysing the data from the initial 40-subject phase, the results indicated that although the device showed strong diagnostic capacity, the discrimination between melanoma and other pathologies was not fully representative of daily clinical practice because the initial sample included predominantly high-suspicion cases referred for biopsy. Following the amended protocol, the extended sample incorporated a broader set of lesions and the performance remained consistent, yielding a cohort more representative of the intended-use population.

Ethical Aspects of Clinical Research

This study adhered to international Good Clinical Practice (GCP) guidelines, the Declaration of Helsinki in its latest amendment, and applicable international and national regulations. As applicable, approval from the relevant Ethics Committee was obtained prior to the initiation of the study. When applicable, modifications to the protocol were reviewed and approved by the Principal Investigator (PI) and subsequently evaluated by the Ethics Committee before subjects were enrolled under a modified protocol.

This study was conducted in compliance with European Regulation 2016/679, of 27 April, concerning the protection of natural persons with regard to the processing of personal data and the free movement of such data (General Data Protection Regulation, GDPR), and Organic Law 3/2018, of 5 December, on the Protection of Personal Data and the guarantee of digital rights. In accordance with these regulations, no data enabling the personal identification of participants was collected, and all information was managed securely in an encrypted format.

Participants were informed both orally and in writing about all relevant aspects of the study, with the information being tailored to their level of understanding. They were provided with a copy of the informed consent form and the accompanying patient information sheet. Adequate time was given to patients to ask questions and fully comprehend the details of the study before providing their consent.

The PI was responsible for the preparation of the informed consent form, ensuring it included all elements required by the International Conference on Harmonisation (ICH), adhered to current regulatory guidelines, and complied with the ethical principles of GCP and the Declaration of Helsinki.

The original signed informed consent forms were securely stored in a restricted access area under the custody of the PI. These documents remained at the research site at all times. Participants were provided with a copy of their signed consent form for their records.

Investigators and Administrative Structure of Clinical Research

Brief Description

The clinical investigation team comprises dermatologists from the two participating sites and technical personnel from the manufacturer. Dr. Jesús Gardeazabal García and Dr. Rosa María Izu Belloso serve as Principal Investigators at Hospital Universitario Cruces and Hospital Universitario Basurto respectively. Dr. Juan Antonio Ratón Nieto and Dr. Ana Sánchez Díez collaborate as Clinical Staff at the same two centres. Technical support for the investigation is provided by the manufacturer's Chief Technology Officer, Chief Executive Officer and General Manager.

Investigators

Principal Investigators

• Dr. Jesús Gardeazabal (Osakidetza, Hospital Universitario Cruces)
• Dra. Rosa Mª Izu (Osakidetza, Hospital Universitario Basurto)

Collaborating Investigators (Clinical Staff)

• Dr. Juan Antonio Ratón Nieto (Servicio de Dermatología, Hospital Universitario Cruces)
• Dr. Ana Sánchez Díez (Servicio Dermatología, Hospital Universitario Basurto)

Technical Support (Manufacturer)

• Mr. Alfonso Medela — Chief Technology Officer
• Mrs. Andy Aguilar — General Manager
• Mr. Taig Mac Carthy — Chief Executive Officer

External Organization

No external organisations contributed to this research.

Sponsor and Monitor

The sponsor and monitor of the investigation is the manufacturer identified in §Sponsor Identification and Contact. The manufacturer's civil liability insurance covered possible damages to participants throughout the study duration.

Signature meaning

The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:

• Author: Team members involved
• Reviewer: JD-018 Clinical Research Coordinator
• Approver: JD-022 Medical Manager