T-200-001 Summary of Clinical Validation of Data Acquisition Digital Health Technology

• Governed by procedure GP-200 Remote Data Acquisition in Clinical Investigations
• Comes from template T-200-001 Summary of Clinical Validation of Data Acquisition Digital Health Technology

Identification of the Technology

This summary describes the clinical validation of the Legit.Health Data Acquisition Technology (hereinafter, DAT) in accordance with Guidance FDA-2021-D-1128 for Digital Health Technologies for Remote Data Acquisition in Clinical Investigations to assess how the DAT is fit-for-purpose for use in the clinical investigation.

Design and related technological characteristics

The Legit.Health DAT is a software tool that was designed and developed following IEC 62304:2006 Medical device software — Software life cycle processes. The life cycle requirements and the set of processes, activities, and tasks described in this standard establish a common framework for medical device software life cycle processes.

The DAT is a computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. The DAT comprises several object detection models that are trained for a specific task. For each model, a basic dataset is constructed by taking the images of the desired ICD category from the main image recognition dataset.

The relevant performance attributes are:

Metric	Value
Weight	33 kilobytes
Average response time	1400 miliseconds
Maximum requests per second	no limit
Service availability time slot	The service is available at all times
Service availability rate during its working slot (in % per month)	100%
Maximum application recovery time in the event of a failure (RTO/AIMD)	6 hours
Maximum data loss in the event of a fault (none, current transaction, day, week, etc.) (RPO/PDMA)	None
Maximum response time to a transaction	10 seconds
Backup device (software, hardware)	Software (AWS S3)
Backup frequency	12 hours
Backup modality	Incremental
Recomended dimensions of images sent	10,000px2

The following diagram splits each high-level software item into software units that we have identified as indivisible:

The following list shows the documents related to the design and development procedure together with their purpose and intended audience:

• Legit.Health Plus description and specifications
  • Purpose: to document the device's information and specifications, including the applicable standards;
  • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
• R-TF-008-001 GSPR
  • Purpose: to evaluate and document the safety and performance requirements applicable to the device and to document how the applicable requirements are implemented;
  • Intended audience: regulatory and quality team, product development team, clinical team, Notified Body and internal/external auditors.
• Design History File, comprised of:
  • Requirements
    • Purpose: to document user requirements, software requirement specification, design requirement and regulatory requirements;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • Activities
    • Purpose: to document the design verification;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • Test plans
    • Purpose: to document the design verification plans to ensure the device is capable of meeting the requirements established for its intended purpose;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • Test runs
    • Purpose: to document the design verification results to ensure the device is capable of meeting the requirements established for its intended purpose;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • Version release
    • Purpose: to document the design transfer to production;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • Design stage review
    • Purpose: to document the design review at each stage of the design and development process;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • SOUP
    • Purpose: to document the SOUP used in the software development, their requirements and any anomalies;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
• R-TF-012-005 Design change control
  • Purpose: to document the list of device's version releases and the changes implemented in each release;
  • Intended audience: regulatory and quality team, product development team, customer success team, Notified Body and internal/external auditors.
• R-TF-012-006 Life cycle plan and report
  • Purpose: to define the techniques, tools, resources and activities related to the development of the device to guarantee this development is performed following UNE-EN 62304:2007/A1:2016 Medical device software. Software life-cycle processes standard;
  • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
• R-TF-012-007 Formative evaluation plan, R-TF-012-014 Summative evaluation plan
  • Purpose: to document plans for software usability testing according to the requirements set out in UNE-EN 62366-1:2015/A1:2020 Application of usability engineering to medical devices;
  • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
• R-TF-012-008 Formative evaluation report, R-TF-012-015 Summative evaluation report
  • Purpose: to document the results of the software usability testing activities
  • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
• R-TF-012-009 Validation and testing of machine learning models
  • Purpose: to define the metrics and methodologies to test the performance of the different machine learning models implemented in the device;
  • Intended audience: regulatory and quality team, product development team (especially medical data scientists, JD-009), Notified Body and internal/external auditors.
• R-TF-012-012 Customers product version control
  • Purpose: to monitor and document the device's version used by customers;
  • Intended audience: regulatory and quality team, product development team, customer success team, Notified Body and internal/external auditors.
• R-TF-013-002 Risk management record
  • Purpose: to document the risk management process performed according to the requirements set out in UNE-EN ISO 14971:2020 Medical devices - Application of risk management to medical devices;
  • Intended audience: regulatory and quality team, product development team, clinical team, Notified Body and internal/external auditors.
• Legit.Health Plus IFU
  • Purpose: to provide the users with all the necessary information according to the requirements set out in MDR 2017/745, Annex I for the safe use of the device;
  • Intended audience: regulatory and quality team, product development team, customer success team, clinical team, sales team, intended users, Notified Body and internal/external auditors.
• R-TF-001-008 Legit.Health Plus label
  • Purpose: to provide the users with the device's information according to the requirements set out in MDR 2017/745, Annex I;
  • Intended audience: regulatory and quality team, product development team, intended users, Notified Body and internal/external auditors.

Relationship between the DAT and the medical device

The Legit.Health DAT is a software tool that is used in clinical research for drug development. The development of the DAT was done in parallel with the development of the Legit.Health Plus device, which is a certified medical device manufactured by the same company. Likewise, the validation of the DAT was done in conjunction with the validation of the Legit.Health Plus device, following ISO 13485:2016 and IEC 62304:2006. This adds a layer of validation to the DAT, as it was validated in conjunction with the medical device, and is subject to the same quality control and regulatory requirements.

Medical device details

	Information
Device name	Legit.Health Plus (hereinafter, the device)
Model and type	NA
Version	1.0.0.0
Basic UDI-DI	8437025550LegitCADx6X
Certificate number (if available)	MDR 792790
EMDN code(s)	Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software)
GMDN code	65975
Class	Class IIb
Classification rule	Rule 11
Novel product (True/False)	FALSE
Novel related clinical procedure (True/False)	FALSE
SRN	ES-MF-000025345

Manufacturer of the Technology

	Manufacturer data
Legal manufacturer name	AI Labs Group S.L.
Address	Street Gran Vía 1, BAT Tower, 48001, Bilbao, Bizkaia (Spain)
SRN	ES-MF-000025345
Person responsible for regulatory compliance	Alfonso Medela, María Diez, Giulia Foglia
E-mail	office@legit.health
Phone	+34 638127476
Trademark	Legit.Health

Related Clinical Validation

The clinical validation of the Legit.Health DAT has been published in peer-reviewed journals. The following articles describe the validation of the DAT for different skin conditions:

• Alfonso Medela, Taig Mac Carthy, S. Andy Aguilar Robles, Carlos M. Chiesa-Estomba, Ramon Grimalt, Automatic SCOring of Atopic Dermatitis Using Deep Learning: A Pilot Study, JID Innovations, Volume 2, Issue 3, 2022, 100107, ISSN 2667-0267, https://doi.org/10.1016/j.xjidi.2022.100107.
• Hernández Montilla, I., Medela, A., Mac Carthy, T., Aguilar, A., Gómez Tejerina, P., Vilas Sueiro, A., González Pérez, A. M., Vergara de la Campa, L., Luna Bastante, L., García Castro, R., & Alfageme Roldán, F. (2023). Automatic International Hidradenitis Suppurativa Severity Score System (AIHS4): A novel tool to assess the severity of hidradenitis suppurativa using artificial intelligence. Skin Research and Technology, 29(6). https://doi.org/10.1111/srt.13357
• Taig Mac Carthy, Ignacio Hernández Montilla, Andy Aguilar, Rubén García Castro, Ana María González Pérez, Alejandro Vilas Sueiro, Laura Vergara de la Campa, Fernando Alfageme, Alfonso Medela, Automatic Urticaria Activity Score: Deep Learning–Based Automatic Hive Counting for Urticaria Severity Assessment, JID Innovations, Volume 4, Issue 1, 2024, 100218, ISSN 2667-0267, https://doi.org/10.1016/j.xjidi.2023.100218.
• Hernández Montilla I, Mac Carthy T, Aguilar A, Medela A. Dermatology Image Quality Assessment (DIQA): Artificial intelligence to ensure the clinical utility of images for remote consultations and clinical trials. J Am Acad Dermatol. 2023;88(4):927-928. https://doi.org/10.1016/j.jaad.2022.11.002

Data output provided

The data is provided to the sponsor according to the Data Transfer Agreement (DTA) which specifies the format, the content, the frequency and the people involved in the data transfer process. In general terms, as is pertains to this scoring system, the keys contained in the data output are:

Key	Description	Type	Example
isValid	Whether or not the image has enough quality	Boolean	True, false
qualityScore	Numeric value that represents the quality of the image	Interger	e.g : 75, 80
severityScore	Numeric value that represents the severity of the condition according to the scoring system	Interger	e.g : 75, 80
visitId	Visit Name	String
date	Date performed DD-MMM-YYY	String
time	Time performed HH:MM	String
imageUid	Unique universal identification of the image	String	e.g. 90925097-820b-403d-a75d-4cd989903df1

Overview

The goal of the validation exercise was to establish the accuracy, repeatability, and reproducibility of the Legit.Health DAT as a tool to measure the clinical event or characteristic of interest: i.e. extent of involvement of psoriasis according to the SALT scoring system. This validation required an experimental methodology that compared the measured accuracy of the device against a gold standard, as well the performance of the clinicians using the DAT to measure repeatability and reproducibility.

The validation was conducted using Legit.Health, a validated and 21 CFR Part 11 compliant application designed for the purpose of capturing images and measuring the severity of skin conditions.

The gold standard involved n expert clinicians that evaluated the images and provided the score for the intensity of the clinical signs that comprise the scoring system, according to their medical opinion. The clinicians did not use the DAT during their assessment.

• Number of clinicians: n
• Number of cases (images): n

To create the ground-truth, each case was evaluated by all clinicians. This is true for all clinical signs.

Clinical signs

The clinical signs evaluated as part of the scoring system were:

Name	Description	Score range
Name	Description	$[0, 4]$
Name	Description	$[0, 4]$

These clinical signs comprise the scoring system i.e. PASI and are used to evaluate disease progression.

Accuracy, repeatability, and reproducibility

The performance, effectiveness, and safety of the DAT was evaluated through the following parameters and performance metrics:

Accuracy

Accuracy was measured through Performance metric name (i.e. AUC). This metric is key to validating the Legit.Health DAT because reason why the metric is important. The performance metric was calculated by explanation on how the performance metric was calculated:

performance metric
$00.0000$

Results

The detailed results of the accuracy metric are as follows:

Full table of results

Repeatability (intra-rater reliability)

To test repeatability of the DAT, we recorded the results of three clinicians using the DAT independently. A total of 30 images were analysed per technician. The technicians where aided by the DAT and asked to score the severity of the disease, with the help of the output of the DAT.

To measure if the result of each clinician was consistent across multiple instances, we compared the clinician-assigned score with the ground truth score.

Results

The detailed results of the repeatability study are as follows:

Clinician 1

The scores provided by clinician 1 while using the Legit.Health DAT were compared to the ground truth scores:

	Clinician-assigned score	Ground truth score	Difference
Case 1	43	43	0
Case 2	12	40	3
Case 3	32	43	0
...	...	...	...
Case 30	45	45	0

Mean Differente for Clinician 1:
$3.96\%$

Clinician 2

The scores provided by clinician 2 while using the Legit.Health DAT were compared to the ground truth scores:

	Clinician-assigned score	Ground truth score	Difference
Case 1	43	43	0
Case 2	12	40	3
Case 3	32	43	0
...	...	...	...
Case 30	45	45	0

Mean Differente for Clinician 2:
$3.96\%$

Clinician 3

The scores provided by clinician 3 while using the Legit.Health DAT were compared to the ground truth scores:

	Clinician-assigned score	Ground truth score	Difference
Case 1	43	43	0
Case 2	12	40	3
Case 3	32	43	0
...	...	...	...
Case 30	45	45	0

Mean Differente for Clinician 3:
$3.96\%$

Reproducibility (inter-rater reliability)

To test repeatability of the DAT, we recorded the results of three clinicians using the Legit.Health DAT system. A total of 30 images were analysed per clinicians. The %CV (coefficient of variation) was calculated for each sample and averaged across all 30 samples.

Results

The detailed results of the reproducibility study are as follows:

Technician	Case 1	Case 2	Case 2	...	Case 30	%CV
User 1	43	12	32	...	45
User 2	40	15	33	...	45
User 3	43	14	33	...	45

Overall mean %CV:
$3.96\%$

Conclusion

The DAT is fit-for-purpose for use in the clinical investigation.

The measures of i.e. extent of involvement of psoriasis according to the SALT scoring system by the Legit.Health DAT are considered accurate, reproducible, and validated for use. The DAT produces accurate and reproducible data, is considered to be a validated method, and is suitable for use.

The assessments of accuracy, repeatability, and reproducibility using the methods described in the validation plan demonstrate that the DAT produces measurements within acceptable limits of variation.

Confidentiality statement

This document has ben provided as a professional courtesy to Name of the recipient person of Name of the recipient company. All the information contained in this document is propietary is intended for the exclusive use of the recipient, limiting the use of the information to the professional relationship between the recipient and Legit.Health.

Approval

The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. The team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:

• Author: JD-009 Ignacio Hernández
• Reviewer: JD-003 Alfonso Medela
• Approver: JD-001 Andy Aguilar

Signed by:

Signer Name: Andy Aguilar (JD-001 General Manager)

Signing Reason: I approve this document

Signing Time: 23/05/2023 | 9:27:36 PM PDT

7CE73AB2FC2E4D9DA6568C1BE1BA6354