R-200-001-SALT Summary of Clinical Validation of Data Acquisition Digital Health Technology

• Governed by procedure GP-200 Remote Data Acquisition in Clinical Investigations
• Comes from template T-200-001 Summary of Clinical Validation of Data Acquisition Digital Health Technology

Identification of the Technology

This summary describes the clinical validation of the Legit.Health Data Acquisition Technology (hereinafter, DAT) in accordance with Guidance FDA-2021-D-1128 for Digital Health Technologies for Remote Data Acquisition in Clinical Investigations to assess how the DAT is fit-for-purpose for use in the clinical investigation.

Design and related technological characteristics

The Legit.Health DAT is a software tool that was designed and developed following IEC 62304:2006 Medical device software — Software life cycle processes. The life cycle requirements and the set of processes, activities, and tasks described in this standard establish a common framework for medical device software life cycle processes.

The DAT is a computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. The DAT comprises several object detection models that are trained for a specific task. For each model, a basic dataset is constructed by taking the images of the desired ICD category from the main image recognition dataset.

The relevant performance attributes are:

Metric	Value
Weight	33 kilobytes
Average response time	1400 miliseconds
Maximum requests per second	no limit
Service availability time slot	The service is available at all times
Service availability rate during its working slot (in % per month)	100%
Maximum application recovery time in the event of a failure (RTO/AIMD)	6 hours
Maximum data loss in the event of a fault (none, current transaction, day, week, etc.) (RPO/PDMA)	None
Maximum response time to a transaction	10 seconds
Backup device (software, hardware)	Software (AWS S3)
Backup frequency	12 hours
Backup modality	Incremental
Recomended dimensions of images sent	10,000px2

The following diagram splits each high-level software item into software units that we have identified as indivisible:

The following list shows the documents related to the design and development procedure together with their purpose and intended audience:

• Legit.Health Plus description and specifications
  • Purpose: to document the device's information and specifications, including the applicable standards;
  • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
• R-TF-008-001 GSPR
  • Purpose: to evaluate and document the safety and performance requirements applicable to the device and to document how the applicable requirements are implemented;
  • Intended audience: regulatory and quality team, product development team, clinical team, Notified Body and internal/external auditors.
• Design History File, comprised of:
  • Requirements
    • Purpose: to document user requirements, software requirement specification, design requirement and regulatory requirements;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • Activities
    • Purpose: to document the design verification;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • Test plans
    • Purpose: to document the design verification plans to ensure the device is capable of meeting the requirements established for its intended purpose;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • Test runs
    • Purpose: to document the design verification results to ensure the device is capable of meeting the requirements established for its intended purpose;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • Version release
    • Purpose: to document the design transfer to production;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • Design stage review
    • Purpose: to document the design review at each stage of the design and development process;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
  • SOUP
    • Purpose: to document the SOUP used in the software development, their requirements and any anomalies;
    • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
• R-TF-012-005 Design change control
  • Purpose: to document the list of device's version releases and the changes implemented in each release;
  • Intended audience: regulatory and quality team, product development team, customer success team, Notified Body and internal/external auditors.
• R-TF-012-006 Life cycle plan and report
  • Purpose: to define the techniques, tools, resources and activities related to the development of the device to guarantee this development is performed following UNE-EN 62304:2007/A1:2016 Medical device software. Software life-cycle processes standard;
  • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
• R-TF-012-007 Formative evaluation plan, R-TF-012-014 Summative evaluation plan
  • Purpose: to document plans for software usability testing according to the requirements set out in UNE-EN 62366-1:2015/A1:2020 Application of usability engineering to medical devices;
  • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
• R-TF-012-008 Formative evaluation report, R-TF-012-015 Summative evaluation report
  • Purpose: to document the results of the software usability testing activities
  • Intended audience: regulatory and quality team, product development team, Notified Body and internal/external auditors.
• R-TF-012-009 Validation and testing of machine learning models
  • Purpose: to define the metrics and methodologies to test the performance of the different machine learning models implemented in the device;
  • Intended audience: regulatory and quality team, product development team (especially medical data scientists, JD-009), Notified Body and internal/external auditors.
• R-TF-012-012 Customers product version control
  • Purpose: to monitor and document the device's version used by customers;
  • Intended audience: regulatory and quality team, product development team, customer success team, Notified Body and internal/external auditors.
• R-TF-013-002 Risk management record
  • Purpose: to document the risk management process performed according to the requirements set out in UNE-EN ISO 14971:2020 Medical devices - Application of risk management to medical devices;
  • Intended audience: regulatory and quality team, product development team, clinical team, Notified Body and internal/external auditors.
• Legit.Health Plus IFU
  • Purpose: to provide the users with all the necessary information according to the requirements set out in MDR 2017/745, Annex I for the safe use of the device;
  • Intended audience: regulatory and quality team, product development team, customer success team, clinical team, sales team, intended users, Notified Body and internal/external auditors.
• R-TF-001-008 Legit.Health Plus label
  • Purpose: to provide the users with the device's information according to the requirements set out in MDR 2017/745, Annex I;
  • Intended audience: regulatory and quality team, product development team, intended users, Notified Body and internal/external auditors.

21 CFR Part 11

The DAT complies with provisions from Electronic Records 21 CFR Part 11, Electronic Records; Electronic Signatures.

Such provisions include the following requirements:

• The signature must be validated.
• The signing process must be secure.
• The signer should not be able to repudiate the signature.
• The signature must include the signer's full name.
• The signature must include the date and time of signing.
• The signature must include a brief summary of the reason for signing.

The DAT includes a full audit trail that records all the actions performed by the user, including the date and time of the action, the user who performed the action, and the reason for the action.

Privacy and data protection

The DAT complies with the Regulation (EU) 2016/679 (General Data Protection Regulation) as well as the UK GDPR (Data Protection Act 2018), which are the European Union's and the UK's data protection and privacy regulations.

Compliance with the regulations is carried out and documented in our procedures GP-050 Data Protection, GP-051 Security violations and GP-052 Data Protection Impact Assessment.

Upon request, the manufacturer can provide a Data Protection Impact Assessment (DPIA) detailing the processing activities and the measures taken to ensure compliance.

Relationship between the DAT and the medical device

The Legit.Health DAT is a software tool that is used in clinical research for drug development. The development of the DAT was done in parallel with the development of the Legit.Health Plus device, which is a certified medical device manufactured by the same company. Likewise, the validation of the DAT was done in conjunction with the validation of the Legit.Health Plus device, following ISO 13485:2016 and IEC 62304:2006. This adds a layer of validation to the DAT, as it was validated in conjunction with the medical device, and is subject to the same quality control and regulatory requirements.

Medical device details

	Information
Device name	Legit.Health Plus (hereinafter, the device)
Model and type	NA
Version	1.0.0.0
Basic UDI-DI	8437025550LegitCADx6X
Certificate number (if available)	MDR 792790
EMDN code(s)	Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software)
GMDN code	65975
Class	Class IIb
Classification rule	Rule 11
Novel product (True/False)	FALSE
Novel related clinical procedure (True/False)	FALSE
SRN	ES-MF-000025345

Manufacturer of the Technology

	Manufacturer data
Legal manufacturer name	AI Labs Group S.L.
Address	Street Gran Vía 1, BAT Tower, 48001, Bilbao, Bizkaia (Spain)
SRN	ES-MF-000025345
Person responsible for regulatory compliance	Alfonso Medela, María Diez, Giulia Foglia
E-mail	office@legit.health
Phone	+34 638127476
Trademark	Legit.Health

Related Clinical Validation

The clinical validation of the Legit.Health DAT has been published in peer-reviewed journals. The following articles describe the validation of the DAT for different skin conditions:

• Alfonso Medela, Taig Mac Carthy, S. Andy Aguilar Robles, Carlos M. Chiesa-Estomba, Ramon Grimalt, Automatic SCOring of Atopic Dermatitis Using Deep Learning: A Pilot Study, JID Innovations, Volume 2, Issue 3, 2022, 100107, ISSN 2667-0267, https://doi.org/10.1016/j.xjidi.2022.100107.
• Hernández Montilla, I., Medela, A., Mac Carthy, T., Aguilar, A., Gómez Tejerina, P., Vilas Sueiro, A., González Pérez, A. M., Vergara de la Campa, L., Luna Bastante, L., García Castro, R., & Alfageme Roldán, F. (2023). Automatic International Hidradenitis Suppurativa Severity Score System (AIHS4): A novel tool to assess the severity of hidradenitis suppurativa using artificial intelligence. Skin Research and Technology, 29(6). https://doi.org/10.1111/srt.13357
• Taig Mac Carthy, Ignacio Hernández Montilla, Andy Aguilar, Rubén García Castro, Ana María González Pérez, Alejandro Vilas Sueiro, Laura Vergara de la Campa, Fernando Alfageme, Alfonso Medela, Automatic Urticaria Activity Score: Deep Learning–Based Automatic Hive Counting for Urticaria Severity Assessment, JID Innovations, Volume 4, Issue 1, 2024, 100218, ISSN 2667-0267, https://doi.org/10.1016/j.xjidi.2023.100218.
• Hernández Montilla I, Mac Carthy T, Aguilar A, Medela A. Dermatology Image Quality Assessment (DIQA): Artificial intelligence to ensure the clinical utility of images for remote consultations and clinical trials. J Am Acad Dermatol. 2023;88(4):927-928. https://doi.org/10.1016/j.jaad.2022.11.002

Data output provided

The data is provided to the sponsor according to the Data Transfer Agreement (DTA) which specifies the format, the content, the frequency and the people involved in the data transfer process. In general terms, as is pertains to this scoring system, the keys contained in the data output are:

Key	Description	Type	Example
isValid	Whether or not the image has enough quality	Boolean	True, false
qualityScore	Numeric value that represents the quality of the image	Interger	e.g : 75, 80
severityScore	Numeric value that represents the severity of the condition according to the scoring system	Interger	e.g : 75, 80
visitId	Visit Name	String
date	Date performed DD-MMM-YYY	String
time	Time performed HH:MM	String
imageUid	Unique universal identification of the image	String	e.g. 90925097-820b-403d-a75d-4cd989903df1

Overview

The goal of the validation exercise was to establish the accuracy, repeatability, and reproducibility of the Legit.Health DAT as a tool to measure the clinical event or characteristic of interest: the extent of hair loss. This validation required an experimental methodology that compared the measured accuracy of the device against a gold standard. Such gold standard involved a total of 6 trained professionals that evaluated the images and provided the extent of hair loss in each case. The clinicians did not use the DAT during their assessment.

• Number of professionals: 6
• Number of images: 1,826

Clinical signs

The clinical sign evaluated was hair loss ($H$), which is computed as the ratio between the extent of baldness ($B$) and total extent of scalp in the image ($S$):

$$H = \frac{|B|}{|S| \cup |B|}$$

This value was computed for every image using segmentation masks of baldness and scalp annotated by the professionals.

Image dataset

Data collection

For this validation, we used a total of 1,826 images, collected in three batches:

• First batch: 390 images of patients diagnosed with alopecia taken from online dermatology atlases. It included close-up views of alopecia patients, as well as pictures of the entire patient's head.
• Second batch: 436 images from online sources, including image search engines, free stock images, and open image datasets of hair detection and segmentation. The goal of this second batch was to introduce more variability in the dataset, incorporating healthy subjects and more diverse hairstyles, as well as more challenging images of heterogeneous quality.
• Third batch: This was the largest batch, consisting of 1,000 images of male and female subjects with different grades of alopecia. This dataset was collected specifically for this purpose in collaboration with a data collection and annotation service provider. Each subject took 5 views of the head: back, front, left, right, and top. The goal of this batch was to provide the algorithm with real data that would resemble the real use case of the SALT scoring system.

Data annotation

Once the images were collected, they were labeled by a team of workers trained specifically for this task. Annotators (A1, A2, ..., A6) were instructed to draw segmentation masks for:

• Scalp contour
• Areas with hair loss

To reduce the impact of observer variability, each image was annotated by three professionals. The final segmentation masks were then obtained by averaging the annotations of the three professionals.

Model training

Each batch was split into a training and validation set. The training set was used to fit a deep learning-based image segmentation model for the task of scalp and alopecia segmentation, and the validation set was used to measure performance on unseen image data.

Accuracy, repeatability, and reproducibility

The performance, effectiveness, and safety of the DAT was evaluated through the Intersection over Union (IoU). IoU is the most common metric in image segmentation, as it compares the area of overlap between the predicted mask ($m_{p}$) and the ground truth mask of an image ($m_{gt}$). IoU can also be used to measure inter-rater variability by comparing the masks of one annotator to another's. IoU is calculated as the intersection (overlapping area between masks) divided by the union (total area of both masks):

$$IoU = \frac{|m_{p} \cap m_{gt} |}{|m_{p} \cup m_{gt}|}$$

Results

Before evaluating model performance, we computed the IoU between annotators for alopecia segmentation. This provided a baseline to assess the model's ability to detect hair loss in an image.

First image batch

	A1	A2	A3	Average
A1	1.00	0.76	0.74	0.75
A2	0.76	1.00	0.76	0.76
A3	0.74	0.76	1.00	0.75

Second image batch

	A1	A2	A3	Average
A1	1.00	0.63	0.66	0.64
A2	0.63	1.00	0.69	0.66
A3	0.66	0.69	1.00	0.67

Third image batch

	A4	A5	A6	Average
A4	1.00	0.77	0.75	0.76
A5	0.77	1.00	0.85	0.81
A6	0.75	0.85	1.00	0.80

Model performance

Task	IoU
Hair loss segmentation	0.6454
Scalp segmentation	0.7202
Head segmentation	0.8754

Conclusion

The DAT is fit-for-purpose for use in the clinical investigation.

The measures of hair loss according to the SALT scoring system by the Legit.Health DAT are considered accurate, reproducible, and validated for use. The IoU metrics for the segmentation were higher than 0.5, which is considered as a good performance value for segmentation within acceptable limits of variation.

Limitations

The SALT scoring system quantifies hair loss based on the percentage of hair loss across four distinct regions of the head.

This method directly correlates with the calculation of alopecia severity.

In this study, we demonstrated the high performance of the DAT for this specific task, benchmarking it against a gold standard in which three annotators manually drew head and alopecia masks on images.

However, this benchmark does not fully represent the real-world performance of specialists, which is typically lower.

This discrepancy arises because, in practice, specialists do not manually delineate areas of alopecia for a computer to calculate a percentage. Instead, they visually assess the subject and estimate the percentage based on their expertise.

Calculating the percentage of hair loss relative to the head is inherently complex. According to Cognitive Load Theory (CLT), working memory is limited when processing novel or intricate tasks.

This particular task requires analyzing the total hair area, accurately identifying the lost hair area, and performing proportional calculations.

Such demands impose a high intrinsic cognitive load due to the complexity of geometric and proportional reasoning, compounded by extraneous load if the tools or methods used are inefficient. Balancing these cognitive demands while maintaining accuracy can easily overwhelm working memory, leading to errors.

Given these challenges, we anticipate that visually derived SALT scores by specialists are likely to deviate more significantly from the DAT's results, underscoring the need for reliable tools like DAT in clinical practice.

Confidentiality statement

This document has ben provided as a professional courtesy to Mr. Jason Kato of Sagimet Biosciences Inc. All the information contained in this document is propietary is intended for the exclusive use of the recipient, limiting the use of the information to the professional relationship between the recipient and Legit.Health.

Approval

The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. The team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:

• Author: JD-009 Ignacio Hernández
• Reviewer: JD-003 Alfonso Medela
• Approver: JD-001 Andy Aguilar

Signed by:

Signer Name: Andy Aguilar (JD-001 General Manager)

Signing Reason: I approve this document

Signing Time: 23/05/2023 | 9:27:36 PM PDT

7CE73AB2FC2E4D9DA6568C1BE1BA6354