R-TF-007-004 Post-Market Surveillance (PMS) Report

This report summarises the post-market surveillance activities conducted according to the R-TF-007-001 Post-market surveillance (PMS) plan for the legacy device called Legit.Health (hereinafter, the Legacy Device) which is a class I medical device according to MDD 93/42/EEC. The legacy device is equivalent to the device called Legit.Health Plus that is classified as class II medical device according to MDR 2017/745.

Surveillance period

The period during which we conducted post-market activities to collect data about safety and performance of our device is from January 2021 to December 2023.

Abbreviations

• PMS: Post-market surveillance
• PMCF: Post-market clinical follow up
• FSCA: Field Safety Corrective Action
• CAPA: Corrective Action Preventive Action
• CER: Clinical Evaluation Report

Post-market surveillance activities

Sales data

The Legacy Device was first available by the end of 2020, when we obtained the Spanish manufacturer license. Since then we have signed contracts with 21 customers, ranging from remote to on-site use of the device, and from government-run care providers to for-profit care providers. During this period, more than 4500 reports have been created by more than 500 practitioners who have used the product to help more than 1000 patients.

Size and other characteristics of the population using the device

In this section, we aim to provide a comprehensive evaluation of the population exposure to the device, delineating its utilization across various patient demographics and settings.

Data privacy

It is imperative to note that our commitment to data privacy and safety is paramount; hence, we intentionally refrain from collecting demographic information that is not essential for providing our services, as the General Data Protection Regulation requires in the following provision:

• Article 5.1.c: Data Minimization
• Article 25: Privacy by design

These stipulations are mandatory for any organization processing the personal data of individuals within the EU.

Patient exposure and device usage

Over the course of the device's availability in the market, we have observed a significant engagement, with more than 4500 reports generated, aiding over 1000 patients. This utilization encompasses more than 500 practitioners, demonstrating the device's acceptance and integration into clinical practice.

The device has found application in diverse settings, ranging from remote consultations to on-site medical evaluations, and is employed by a spectrum of care providers including government-run institutions and private for-profit entities.

Our records indicate that since obtaining the Spanish manufacturer license at the end of 2020, we have successfully established partnerships with 21 customers. This varied customer base has contributed to the extensive usage and adaptability of the device across different patient populations and healthcare settings.

Patient populations and usage patterns

Given our stringent privacy protocols, we do not have access to demographic data of the patients using the device. This approach aligns with our commitment to ensuring privacy and safeguarding patient information, a feature designed to uphold safety and data protection.

Despite this limitation in demographic data availability, our extensive usage records provide valuable insights into the device's adoption and application trends. The most notable of these trends is the two clinical situations in which the device was used, being:

1. First consultations
2. Follow-up consultations, especially for chronic patients.

During follow-up consultation, the most commonly represented ICD classes are psoriasis, acne, atopic dermatitis, followed by urticaria and hidradenitis suppurativa. Regarding first consultations, the distribution of ICD classes is very heterogeneous.

We meticulously monitor the usage patterns to identify any deviations from expected utilization and to ensure equitable access across different patient groups.

Equitable usage and representation

Up to this point, we have not identified any issues concerning over-represented or under-represented patient groups. Our vigilant monitoring and continuous evaluation processes are in place to promptly detect any disparities or anomalies in device usage across various patient populations.

We remain dedicated to ensuring that the device serves its intended purpose effectively and is accessible to all eligible patients, irrespective of their background or healthcare setting. Our commitment to safety, privacy, and efficacy drives our efforts to constantly improve and adapt our practices, ensuring the device continues to be a reliable tool in clinical settings.

Serious incidents and FSCA

We are proud to confirm that during the review period, no serious incidents associated with the use of the device has been reported. This reflects our unwavering commitment to delivering safe and reliable medical devices, and our proactive stance in identifying and mitigating potential risks.

Throughout the period under review, no Field Safety Corrective Actions were required for the device.

To ensure the suitability of our internal procedure SP-004-001 Product withdrawal, we conducted a simulation of product removal from the market according to the process described in the procedure. Results of the simulation were satisfactory: the process well describes the steps to remove the product from the market, that in our specific case it consists of software access removal. The results are stored in the following record: R-004-001 Withdrawal record 2023.

Non-serious incidents and side effects

After receiving the customer complaints, we analysed them to evaluate whether they are incident according to the definition provided in MDR 2017/745.

We registered 6 customer complaints during the period under evaluation and 4 out of 6 have been classified as incident because the complaints indicated that there was a malfunction or a deterioration in the performance of the device.

Same approach has been followed for internal and audit non-conformities related to our device: we evaluated them to classify them as incident or non-incident.

The summary of non-serious incidents registered during the analysed period is shown in the following table:

Non-serious incident	Initiation date	Non-serious incident description	Source
Using diagnosis_support with only one image	2023-12-21	The client reported an issue with the device's endpoint diagnosis_support when a single image is sent to the device. This caused json deserialization (on client's side) to fail	Customer complaint
API Service down (no response within 60 seconds)	2023-09-13	The client reported that when they send an image to our JSON API, they do not get a response within 60s timeout window	Customer complaint
Algorithm performance results did not match with the skin conditions that were being photographed	2023-05-02	The client reported that they have been using our API to review and analyse 50 photos and they spotted that the results from the API did not match with the skin conditions that were being photographed	Customer complaint
CN Vitup: Confusing AI results	2023-03-01	The client communicated that they got a few incidents related to AI results returning totally different pathology lists for the same skin condition	Customer complaint
The images of pigmented skin lesions of the diagnosis dataset (LegitHealth-DX) are biased	2023-06-01	We found that when we send to the API a picture of a nevus, depending on the zoom performed on the image, the result changed from a high percentage of being nevus to a high percentage of being melanoma or another skin lesion	Internal non-conformity
The algorithm results after analysis of benign pigmentation images are suspiciously high	2023-04-28	During a kick-off session with a client, physicians took images with different camera devices of two benign pigmented lesions and we found that the malignancy score was high in both cases	Internal non-conformity
When uploading a photo of atopic dermatitis, the device is 99,98% sure that it's psoriasis	2023-01-10	When we uploaded an image of atopic dermatitis, the device was 99,98% sure that it was psoriasis	Internal non-conformity

More details about the customer complaints and non-conformities originating the above incidents and their evaluation is provided in the section Customer complaints and Corrective and preventive actions of this report.

We did not receive any communications about undesirable side-effects related to the use of our device.

Trend analysis

In accordance with the PMS plan, we conducted an analysis of non-serious incidents reported from January 2021 to December 2023. During this period, no non-serious incidents were reported in 2021 and 2022, while 7 non-serious incidents were reported in 2023 (more details about the non-serious incidents are provided in the table below).

Non-serious incident	Initiation date	Non-serious incident description	Source
Using diagnosis_support with only one image	2023-12-21	The client reported an issue with the device's endpoint diagnosis_support when a single image is sent to the device. This caused json deserialization (on client's side) to fail	Customer complaint
API Service down (no response within 60 seconds)	2023-09-13	The client reported that when they send an image to our JSON API, they do not get a response within 60s timeout window	Customer complaint
Algorithm performance results did not match with the skin conditions that were being photographed	2023-05-02	The client reported that they have been using our API to review and analyse 50 photos and they spotted that the results from the API did not match with the skin conditions that were being photographed	Customer complaint
CN Vitup: Confusing AI results	2023-03-01	The client communicated that they got a few incidents related to AI results returning totally different pathology lists for the same skin condition	Customer complaint
The images of pigmented skin lesions of the diagnosis dataset (LegitHealth-DX) are biased	2023-06-01	We found that when we send to the API a picture of a nevus, depending on the zoom performed on the image, the result changed from a high percentage of being nevus to a high percentage of being melanoma or another skin lesion	Internal non-conformity
The algorithm results after analysis of benign pigmentation images are suspiciously high	2023-04-28	During a kick-off session with a client, physicians took images with different camera devices of two benign pigmented lesions and we found that the malignancy score was high in both cases	Internal non-conformity
When uploading a photo of atopic dermatitis, the device is 99,98% sure that it's psoriasis	2023-01-10	When we uploaded an image of atopic dermatitis, the device was 99,98% sure that it was psoriasis	Internal non-conformity

During the period analysed, no non-serious incidents were reported in 2021 or 2022, while seven non-serious incidents were reported in 2023.

The increase in reported non-serious incidents in 2023 compared to the previous years is noteworthy. However, due to the small number of incidents overall and the small number of devices available in the market in 2021 and 2022, caution is necessary when interpreting these data. From a statistical perspective, the dataset's limited size introduces the risk of random variability, which could mask any underlying trends.

In situations where event frequencies are low, as in our case, it is challenging to distinguish between random fluctuations and meaningful changes. Statistical tools typically used for trend analysis would have low power due to the small sample size, leading to a higher chance of incorrect conclusions.

While the increase in incidents warrants monitoring, it is premature to assert any definitive trend or systemic issue based on the current data. As more data is gathered in future PMS activities, we will be better equipped to apply more robust statistical methods to identify potential patterns in non-serious incident reports. For now, we consider the rise in incidents in 2023 to be part of normal variability in the dataset.

Corrective and preventive actions

The corrective/preventive actions started because of a non-conformity (internal or audit non-conformity) or customer complaints during the analysed period are provided in the table below, together with information about the status of actions, effectiveness verification, notification to the Competent Authorities.

Corrective or preventive?	Initiation date	Product or process?	Source	Summary	CAPA Reference number	Description	Root cause	Correction	CAPA plan	CAPA implementation due date	Effectiveness plan	Effectiveness verification	CAPA status	FSCA required	Notification to CA required
Corrective action	2023-12-21	Product	Customer	The format for the 'diagnosis_support' endpoint differs when sending a single image compared to sending multiple images	R006001-82	One of our clients explain us the non-conformity as follows (complaint record ID 2201467799): I'm implementing and evaluating the use of the "diagnosis_support" endpoint for multiple images. First of a design question: How do you intend your endpoints to be used? Should /diagnosis_support be used with 1-many images or should /predict be used for single image and /diagnosis_support be used for 2-many? If the endpoint should be used only for 2 or more images, then I think you should validate against usage with only one image and send a validation error. If it is allowed to send only one image, then I think I have found a bug. explainabilityMedia.modality for the combined analysis result is specified as string[]. Eg: "modality": ["Clinical", "Clinical", "Clinical"], When a single image is sent it becomes just string. Eg: "modality": "Clinical", This causes our json deserialization to fail as I expect it to be an array with a single item. Eg: "modality": ["Clinical"],	At that moment, the API at the /diagnosis_support endpoint was giving the explainabilityMedia.modality in an array for multiple images and as a value for a single image.	Reply to the customer specifying that the second question is a bug and the development team is working on its resolution.	Unify the format for single and multiple images	2024-02-23	Monitor customer complaints to identify any similar issues for 6 months after the implementation of corrective actions. Target: no similar customer complaints in the defined period.	Effectiveness under evaluation	Effectiveness under evaluation	No	No
Corrective action	2023-09-13	Product	Customer	API service does not respond within 60 seconds	R006001-81	One of our customers related that they had had issues with our service since the 7th of September. When they sent an image to our API, they did not get a response within 60s timeout window. They also reported that not a single patient had had a successful analysis since then.	Initially, two options seemed probable: # That the customer was sending the API call with a mistake. # That this is an actual problem of the API, related to the fact that we added the endpoint ai.legit.health, instead of having only the endpoint ia.legit.health. After more investigation, the second option became more likely, for two reasons: A. The timing matches, since we added the endpoint ai.legit.health around the time of the complaint B. The instances that we are aware of people using the new endpoint (ai.legit.health) do not report any problems. However, we should keep in mind that: C. This should not have generated any issues, because we did not change anything in the pre-existing endpoint (ia.legit.health) we simply added a new one. Due to all of this, the investigation does require to opening a CAPA. After more investigation, it turns out that the issue happened because we restarted the servers for maintenance reasons, but the restart of the endpoint ia.legit.health could not load the domain check model and, despite being active, did not return the response.	Customer who reported the issue was put in contact with the development team to solve the issue		2024-03-29	Tests designed to verify the planned actions give successfull results	Yes	Closed	No	No
Corrective action	2023-06-01	Product	Internal non-conformity	The images of pigmented skin lesions of the diagnosis dataset (LegitHealth-DX) are biased	R006001-80	Taig found that when he sent to the API a picture of a nevus, depending on the zoom performed on the image, the result changed from a high percentage of being nevus to a high percentage of being melanoma or another skin lesion, as it can be observed in the attached images.	The images of the LegitHealth-DX dataset present a very high heterogeneity. There are close-up images of lesions and also images where the lesion is far away and there is so much context (as in the last image of the hand). This may make the model learn the wrong patterns to classify lesions in some cases, as some of the classes may have only one type of image (e.g. close-up shots). This issue is currently being solved by manually annotating all the images of the LegitHealth-DX dataset. This annotation involves cropping the areas of the image that correspond to the lesion. Later in training, the model will be fed with both the full image AND the crops, hopefully making it capable of learning to focus on the correct elements of the image and ignore other contextual noisy patterns (e.g. cloth, hands, dirt, background…).	None; the development team started the investigation of the issue to define corrective actions	The Medical Data Science team’s plan is to: - Compile a new batch of images of several skin lesions (such as Nevus) to perform a new algorithm training. - Manually annotate the entire LegitHealth-DX image dataset to extract crops of every image and obtain close-up views of all the conditions.	2024-03-29	Monitor performance of the device for 3 months after the implementation of corrective actions	N/A	CAPA implementation	No	No
Corrective action	2023-05-24	Product	Customer	Algorithm performance results did not match with the skin conditions that were being photographed	R006001-56	Our customer Consultant Connect informed us that they have been using our API to review and analyse 50 photos and they spotted that the results from the API did not match with the skin conditions that were being photographed. Only 8 out of the 50 photos had a correct diagnosis. They are obviously concerned that, for whatever reason, the AI is not producing the right answers.	The first step was to ask Rabi to share with us more information about how they performed the test and the images they have used for it. After a preliminary analysis of them, we found that their quality was not adequate to perform the analysis: As an example, in the image of the leg shown here we found two important issues that may have caused the output to be wrong. !leg.png|width=180,height=180! # Firstly, it seems to be a photo of a photo. I looks like that someone took a picture, printed the picture, and then took a digital picture of the printed picture. That is not the type of situation in which the AI has been trained, and that could have been a factor in confusing the AI. # Secondly, the issue is that the image is not centred on the lesions. The AI needs to receive images in which the region of interest is the main feature of the image. However, in that image, there is more chair, mode floor, more jacket and more door than there are lesions. The right way of reporting the condition would be cropping the image to centre it on the lesion, but the quality is not ideal with the image. !leg_1.png|width=264,height=262! !leg_2.png|width=128,height=124! As another example, the following image is neither centred on the lesions and the lesion is surrounded by artefacts that can affect the algorithm performance. !mole.png|width=179,height=180! Again, cropping the image to centre it on the lesion, the image sent to the API will be the one shown below, which quality is not proper contributing to the output being wrong. !mole_2.png|width=124,height=98! To sum up, we have demonstrated that the performance of the algorithm is not compromised and the problem reported was caused by the inappropriate quality of the images used for the analysis, that do not comply with the minimum requirements established to obtain satisfactory results.	Our technical manager (Alfonso Medela) contacted the customer to explain how to interpret teh results of the device, in case there was any misunderstandings	Include more information on how to take a picture to optimize the device's performance on the IFU	2023-07-21	Monitor customer claims to identify any similar issue for 3 months after the implementation of corrective actions	Yes	Closed	No	No
Corrective action	2023-05-03	Product	Internal non-conformity	The DIQA results after analysis of images with low quality are higher than expected	R006001-79	During a kick-off session at the Ribera Molina Hospital, physicians took an image (see attached) of a pigmented lesion of Taig and we found that the DIQA assigned was unexpectedly high (76). Additionally, we have received the same feedback from the Torrejón Hospital saying that the image quality is not good enough.	The Medical Data Science team suggests a combination of different aspects: _ The DIQA threshold is set to 50. Given the limitations of the current model, this threshold may need to be increased. _ The app sends the image well, the only thing is that in the preview it shows a very ugly image with reduced quality. * Many times the overall image is pretty decent and then the crop over the area of interest is bad. However, the main reason for this undesirable performance is that the current version of DIQA model was mostly trained on non-medical images (only ~900 images were actually dermatological). Future versions of this model will use many more medical images in training.	None; the development team started the investigation of the issue to define corrective actions	The Medical Data Science team is going to prepare a new dataset for DIQA. This new dataset will include a larger sample of dermatology images with both real and artificial distortions that will be assessed by a board of approximately 40 observers (as in the current version of the DIQA dataset). This will provide the DIQA model with more examples of low-quality images and should be able to assess any input image more precisely.	2024-04-19	Monitor customer communications, or internal findings, related to DIQA performance for 3 months after the implementation of the corrective actions	Effectiveness under evaluation	Effectiveness under evaluation	No	No
Corrective action	2023-04-28	Product	Internal non-conformity	The algorithm results after analysis of benign pigmentation images are suspiciously high	R006001-78	During a kick-off session at the Ribera Molina Hospital, physicians took images with different camera devices (see attached) of two benign pigmented lesions of Taig and we found that the malignancy score was high in both cases: * Image 1 results: ** DIQA score 76 ** Is Malignant Suspicion 45.5 ** Conclusions *** Malignant melanoma: 39.33 _ Nevus: 24.10 _ Actinic keratosis: 9.06 * Image 2 results: ** DIQA score 76 ** Is Malignant Suspicion 91.4 ** Conclusions *** Malignant melanoma: 48.60 _ Basal cell carcinoma: 42.05 _ Vascular lesion: 1.40 After that, Taig took another image (image 3) with his mobile and got a high malignancy score again: _ DIQA score 81 _ Is Malignant Suspicion 75.55 ** Conclusions ** Malignant melanoma: 74.40 ** Nevus: 10.93	The reason for this behaviour is that the model is overfitting to some classes due to the current class imbalance in our dataset (LegitHealth-DX). We need to add more data and train the model again. Also manually annotating the dataset and extracting crops of the lesions in every image may improve model performance as it would learn not to focus on the noisy elements of an image and pay attention to the actual lesion. See [https://legithealth.atlassian.net/browse/R006001-57|https://legithealth.atlassian.net/browse/R006001-57|smart-link] for more details on how image cropping is being carried out.	None; the development team started the investigation of the issue to define corrective actions	Add new images to create a more balanced dataset and re-train the model	2024-08-30	Monitor customer communications, or internal findings, related to device performance for 3 months after the implementation of the corrective actions	N/A	CAPA implementation	No	No
Corrective action	2023-03-28	Process	Internal non-conformity	ICD-coded diseases requirement is included at the DHF	R006001-53	During the risk management review I found that the ICD-coded disease requirement was not properly documented within the DHF, but included on a Trello card at the Product development workspace.	The Technical Responsible did not know that this requirement should be documented at the DHF.	Inform Technical Manager about the issue	Document ICD code diseases in a requirement of the DHF	2023-09-28	Monitor the application of the design development process for 3 months after the implementation of the corrective action to identify any deviation from the process	Yes	Closed	No	No
Corrective action	2023-02-08	Process	Audit non-conformity	Annual review cycle in 2021 was missed for many procedures	R006001-49	During the audit performed by FDAQRC on behalf of ICON they found a minor observation that stated that there were multiple instances of non-compliance to company policies and procedures: * Annual review cycle in 2021 was missed for many procedures. Some examples are Quality Planning (GP-002), Internal Audit (GP-003), Human Resources and Training (GP-005), Non-conformity, Corrective and Preventative Actions (GP-006), Purchases and Suppliers Evaluation (GP-010), Design, Redesign and Development (GP-012), Risk Management (GP-013), Infrastructure and Facilities (GP-018).	We have only reviewed the procedures that have suffered modifications during the last period due to a lack of resources.	Revision of procedures	Update frequency of revision of all procedures of the QMS to 3 years (if there are no changes to the process to be implemented)	2023-02-08	Monitor the procedures' review for the year 2023 after the implementation of the corrective actions	Yes	Closed	No	No
Corrective action	2023-02-08	Process	Audit non-conformity	CVs management procedure must be clarified and properly performed	R006001-46	During the audit performed by FDAQRC on behalf of ICON they found a major observation that stated that some procedures were inadequate: * CVs are to be signed and dated annually – This requirement was verbally shared; however, it was not captured in procedural documents. Of the three (3) CVs reviewed, one (1) CV was signed and dated.	We did not know that all the CVs needed to be signed and dated annually.	All employees update and sign the CV	Include the CV update activity in GP-005 Human resources and training	2023-02-08	Monitor whether CV are updated and signed for year 2024	Yes	Closed	No	No
Corrective action	2023-01-27	Process	Internal non-conformity	Some TF records for the 2021 period are missing	R006001-44	There are no evidence of TF documentation review and update for the 2021 and 2022 periods, neither for the following activities performance: _ Risk management _ Clinical evaluation * PMS activities	Last records found were created at the end of 2020, along with the manufacturer license application and obtention. We planned then to transition to the new MDR and to prepare all the documents and records according to this new regulation during the following year, but the Notified Bodies' bottleneck made it impossible to achieve during this period.	A new index file containing the TF documents and annexes, and their correspondence with the new QMS developed to comply with the MDR is prepared to ensure the equivalence between the documents created for the TF prepared under the MDD and the new updated documents prepared for the MDR is properly traced	Activities that required periodic performance will be included at the R-002-005 Quality calendar	2023-10-27	Monitor the periodic activities execution for 6 months after the implementation of corrective actions to ensure the activities are performed according to the set timelines	Yes	Closed	No	No
Corrective action	2023-01-10	Product	Internal non-conformity	When uploading a photo of atopic dermatitis, the app is 99,98% sure that it's psoriasis	R006001-76	Taig found that when he logged in the app as a doctor using his e-mail and the Chrome navigator, and he uploaded a photo of atopic dermatitis, the app was 99,98% sure that it is psoriasis.	The Medical Data Science team looked for the image in the LegitHealth-DX dataset and it confirmed that it had been assigned the wrong pathology (psoriasis). They also detected other images from the same source that may also lead to the same undesired behaviour. Moreover, a closer analysis of the list of conditions of the DX dataset suggests that some conditions that currently coexist in the class list (i.e. the list of conditions that the model can predict) may be causing the model to learn the wrong patterns when classifying the images. This means that not only they will have to remove inconsistently labelled images but also work on the definition of a coherent taxonomy of conditions.	None; the development team started the investigation of the issue to define corrective actions	- Remove the conflictive images from the dataset to prevent similar issues. - Generate a new version of the diagnosis dataset (LegitHealth-DX) with more curated data.	2024-03-29	Monitor the device performance for 3 months after the implementation of corrective actions	Effectiveness under evaluation	Effectiveness under evaluation	No	No
Corrective action	2022-12-13	Process	Audit non-conformity	Missing written withdrawal procedure	R006001-42	During the last internal audit performed on November 2022 it was observed that the withdrawal procedure is in place, but it is not described on any procedure.	The production and design and development team though that if was sufficient with the GP-004 Vigilance System procedure	Creation of product withdrawal procedure	- Verify the process described in the product withdrawal procedure by performing a simulation - Set a frequency for the product withdrawal simulation and add this activity to the quality calendar	2023-08-18	Monitor the execution of the product withdrawal simulation	Yes	Closed	No	No
Corrective action	2022-12-13	Process	Audit non-conformity	Tickets tools usage is not explained in any document	R006001-41	During the internal audit performed last November 2022 it was found that all general communications received from clients will be registered in the NC template, but it is not explained the usage of the tickets tool in place to note down the communications that are not NC.	The platform was recently in use and it was being tested before updating the procedure.	None	Update GP-014 Feedback and complaints with missing information	2022-12-13	Monitor any changes in the feedback and complaints process to verify whether they are properly documented	Yes	Closed	No	No
Corrective action	2022-12-12	Process	Audit non-conformity	Inconsistent number of nonconformities found on different quality records	R006001-40	During the last internal audit performed on November 2022 it was found that the number of nonconformities reflected at the Annual Management Review does not match with the Indicators file.	The incidence was caused as the quality procedures were being perfomed by various members of the staff, and also the external consultant, and the number of Non-conformities were not controlled by the same person.	Fix the inconsistency in number of non-conformities in all quality records where this number is reported	Hire a Quality Manager to centralize all the quality activities and related records	2023-02-28	Review the non-conformities for 2023 in the different quality records to verify consistency	Yes	Closed	No	No
Corrective action	2022-12-12	Process	Audit non-conformity	QMSR has not been performed as stated at the Quality procedures	R006001-39	During the internal audit performed last November 2022 it was observed that although it is mentioned at the quality procedures that the information and conclusions from the QMS documentation backup validation must be compiled at the QMSR report, it has not been performed.	It was forgotten to compile the summary of the backup validation tests in the QMSR	Perform and document QMS documentation backup validation	Add the QMS documentation backup validation activity in the quality calendar	2023-02-28	Monitor execution of activities included in the quality calendar for 6 months after implementation of corrective actions	Yes	Closed	No	No
Corrective action	2022-12-07	Process	Audit non-conformity	Quality indicators for the 2022 period are missing	R006001-37	During the last internal audit performed in November 2022 (NC-09) no evidence of the establishment and monitoring of quality indicators for the 2022 period was found.	The Quality Manager forgot to create the record for the 2022 period	Creation of a template for quality indicators and associated record	Include the revision of quality indicators in the quality calendar	2023-01-31	Monitor execution of quality indicators review activity for 6 months after the implementation of corrective actions	Yes	Closed	No	No
Corrective action	2022-12-07	Product	Audit non-conformity	Software version is not included at the product labelling	R006001-36	During the last internal audit performed in November 2022 (NC-08) it was detected that the software version was not included at the product labelling.	The design & development team were not aware of the need to include the software version on product labelling.	Inform JD-007 about this requirement	Inclusion of the software version in the labeling of the product and in the IFU	2022-12-31	For any new version of the product, or for new product, the software version is correctly stated in teh product and in the IFU	Yes	Closed	No	No
Corrective action	2022-12-07	Process	Audit non-conformity	Design reviews are not performed by a person without direct responsibility for the design stage	R006001-35	During the last internal audit performed in November 2022 (NC-07) it was noticed that there is no evidence of design reviews conducted by a person who does not have direct responsibility for the design stage.	The design and development team does not know the necessity of performing the DHF records by a person without direct responsibility for the design stage.	Assignation of a responsible person for approval of design reviews	Modify the internal procedure GP-012 Design, redesign and development according to the new responsibility introduced for design reviews and review the design templates to include this role	2023-04-27	Monitor that DHF records are reviewed and approved by the responsible person defined in GP-012 for 6 months after implementation of corrective actions	Yes	Closed	No	No
Corrective action	2022-12-07	Process	Audit non-conformity	Product requirements do not include approval evidence	R006001-34	During the last internal audit performed in November 2022 (NC-06) it was noticed that, although the deliverables, tests and product releases were approved signatures, the product requirements do not include approval evidence.	The design and development team does not know the necessity of registering approval evidences to the product requirements.		Update GP-012 and template for requirement to include missing information	2023-04-27	Monitor that new requirements generated as part of the design and development process contain all the required information (in other words, new requirements are generated starting from the new requirement template) for 6 months after the implementation of corrective actions	Yes	Closed	No	No
Corrective action	2022-12-07	Process	Audit non-conformity	Personnel education certificate are missing	R006001-33	During the last internal audit performed in November 2022 (NC-05) it was noticed that personnel education certificates are not compiled within the employees files as evidence of their education.	We did not know that we must compile all the personnel education certificates.	None	- Requirement of storing all employees' educational certificates is communicated to the entire team - Update of tehe onboarding process to add a task related to the upload of educational certificates in the proper folder in Factorial (HR tool)	2023-02-23	Monitor the availability of educational certificates for new employees joining the company (for 6 months after the implementation of corrective actions)	Yes	Closed	No	No
Corrective action	2022-12-07	Process	Audit non-conformity	PRRC and QM Job descriptions do not cover their responsibilities to report to the GM and to promote awareness of the regulatory and other requirements	R006001-32	During the internal audit performed last November 2022 (NC-04) it was observed that the PRRC and QM job descriptions do not specify the required reporting to the General Manager, nor their responsibility to promote the awareness of regulatory and other requirements through the organisation.	During the JD preparation and revision it was not noticed that the reporting line to the General Manager was missing. In addition, we did not know that it was required to include in the JDs their responsibility to promote the awareness of regulatory and other requirements through the organisation.	None	Revision of affected JD according to regulatory requirements	2023-01-10	Monitor creation/revision of JD to verify whether all the applicable requirements are included (for 6 months after implementation of corrective actions)	Yes	Closed	No	No
Corrective action	2022-12-07	Process	Audit non-conformity	Backups verification yearly evidence is missing	R006001-31	During the internal audit performed last November 2022 (NC-03) it was found that although the procedures specified that the backups verification must be performed yearly for the QMS and DHF, there is no evidence for the verification performed a year after the last one done.	It was forgotten to perform the period backups verification	None	- Plan and perform backups activities - Add backup activities to quality calendar	2023-01-18	Monitor execution of activities included in the quality calendar for 6 months after implementation of corrective actions	Yes	Closed	No	No
Corrective action	2022-12-07	Process	Audit non-conformity	Quality objectives for the 2022 period are missing	R006001-30	During the internal audit performed last November (NC-02) it was observed that there was no evidence of the establishment and monitoring of quality objectives for the 2022 period.	Quality and General Manager did not know they need to create an objectives' record yearly.	None	- Define quality objectives - Document quality objectives and review them according to the established frequency - Add quality objectives review in the quality calendar	2022-12-16	Monitor execution of activities included in the quality calendar for 6 months after implementation of corrective actions	Yes	Closed	No	No
Corrective action	2022-12-05	Process	Audit non-conformity	There is no evidence of external documentation identification and control	R006001-29	During the internal audit performed last November (NC-01) it was noticed that the ISO 13485 requirement 4.2.4f was not being fulfilled since no evidence was found of external documentation identification and control.	We were not aware of the importance of having the external documentation identified and internally controlled.	None	- Creation of an external documentation folder within the QMS - Creation of a coding system for external documentation - Update GP-001 Control of documents to include requirements for external documentation - Assignation of code to external documents and storage in the proper folder	2023-02-28	Monitor the update of external documentation to verify whether new or revised external documents are properly controlled	Yes	Closed	No	No
Corrective action	2022-10-10	Process	Audit non-conformity	Lack of access management and traceability policy	R006001-27	Although the Company uses role-based permissions to control access rights, there is no procedure outlining the access management process, including a lack of traceability and policies for role-based permissions and individual access rights (e.g., periodic review of audit logs/trails). The Infrastructure List and Control Plan (T-018-001) referenced in SOP GP-018 Infrastructure and Facilities, lists only two items, the Quality Manager's laptop, and AWS server. The annual log audit for AWS does not have an entry under “previous” and the “next” review was due 17-Dec-2020. In addition, the laptop monthly checklist verification was dated 10-Nov-2020 under “previous.” It is not clear whether this document has not been maintained up-to-date, or the required verifications are not being performed as required by the plan.	The document was outdated and did not reflect the current access management and traceability procedures.	The Infrastructure List and Control Plan (T-018-001) referenced in GP-018 Infrastructure and Facilities to be updated	Creation of a specific procedure for access management and traceability policy	2023-01-17	Monitor the activities related to GP-018 for 6 months after the implementation of corrective actions	Yes	Closed	No	No
Corrective action	2022-09-28	Process	Audit non-conformity	The lack of an internal audit to date is in deviation with the responsibilities set out in the SOP GP-003 Internal Audit	R006001-16	The lack of evaluating the implementation of the QMS through an internal audit is a concern as Top Management does not have a representative understanding of the Company's level of compliance with internal procedures, and applicable regulations and standards. Conducting an internal audit is of particular importance if the Company plans to undergo an ISO 13485 pre-certification inspection. The lack of an internal audit to date is in deviation with the responsibilities set out in the SOP GP-003 Internal Audit, where the manager is required to annually approve the internal audit plan and to determine the internal auditors assigned.	We were not planning to be certified at that time and it was not considered necessary.	None	- Plan, execute internal audit for 2022 and store associated records - Add internal audit in the quality calendar	2022-12-12	Monitor planning and execution of internal audit for year 2023	Yes	Closed	No	No
Corrective action	2022-09-26	Process	Audit non-conformity	Basic security controls and patch management procedures are missing	R006001-11	There are no procedures on the basic security controls and patch management requirements to ensure the protection of company equipment and data from cybersecurity vulnerabilities such as malware and phishing attacks (e.g., unauthorized access to the network, timely software patches, etc.).	We considered that it was covered in the existing SOPs.	None	Creation of a dedicated procedure for security controls and associated templates	2023-02-17	Monitor the activities established in the new procedure for security controls to verify whether they are executed (for 6 months after implementation of corrective actions)	Yes	Closed	No	No
Corrective action	2022-09-26	Process	Audit non-conformity	Documentation available for the software was not sufficient to demonstrate adequate security and that intended use requirements have been met	R006001-8	Standard ISO 13485:2016 clauses 4.1.6 and 7.5.6 requires validating computer software for its intended use according to an established protocol when used as part of production or the quality system. The validation documentation available for these systems (e.g., Confluence, Google Workspace, JIRA) were not sufficient to demonstrate adequate security and intended use requirements have been met.	The documentation for the software was considered as sufficient.	None	Update validation template and records	2023-01-10	Monitor validation activities related to tools used to perform quality activities for 6 months after the implementation of corrective actions	Yes	Closed	No	No
Corrective action	2022-09-26	Process	Audit non-conformity	Records of changes, their review, and any necessary actions were not available for review	R006001-7	Standard ISO 13485:2016, clause 7.3.9 requires that records of changes, their review, and any necessary actions shall be maintained. Procedure GP-012 Design, re-design and development addresses change control in paragraph 9.10, but no record was available for review.	We considered that it was sufficient with the detailed procedures in GP-012.	None	- Update procedure GP-012 to include process for product change - Create template to document product changes - Create procedure GP-023 Design change management to document the process for product changes	2023-04-28	Monitor product changes to verify they are documented according to the requirements set out in the internal procedure (for 6 months after the implementation of corrective actions)	Yes	Closed	No	No
Corrective action	2022-09-26	Process	Audit non-conformity	Additional training or certifications in privacy and GDPR should be completed by the employees	R006001-77	Given the Quality Manager's role in privacy, additional training, or certifications in privacy and GDPR should be completed. Although an internal training in data privacy was provided by the COO who indicated he is an externally certified in ISO 27001, given ISO 27001 is a framework for managing IT security and sets out the specification for an information security management system (ISMS), this would not be sufficient for personnel to lead as one of the individuals responsible for data protection and GDPR compliance. Source: Critical finding 5 from Alira Health audit.	Training in data privacy was considered as sufficient.	None	- Include in the procedure GP-005 Human resources and training the requirement for GDPR training for all employees during onboarding process - Create template to document execution of GDPR training	2023-12-29	Monitor execution of GDPR training as part of the onboarding process for new employees joining the company (for 3 months after the implementation of corrective actions)	Yes	Closed	No	No

During the analysed period we initiated 30 CAPA:

• all of the CAPA are classified as corrective actions
• 22 CAPA are related to non-conformity in our processes, 8 CAPA are related to non-conformity in our device
• 3 non-conformities related to our device are classified as non-serious incident and they generated the following CAPA reference numbers: R006001-76, R006001-78, R006001-80
• 3 CAPA were triggered by customer complaints, 6 CAPA were triggered by internal non-conformities, 21 CAPA were triggered by audit non-conformities (both client and internal audits)
• 2 CAPA are under final implementation, 3 CAPA are under evaluation to verify their effectiveness, 25 CAPA are closed with effectiveness verified
• none of the CAPA includes the initiation of FSCA.

Customer feedback

We follow the process described in GP-014 Feedback and complaints to handle all the information coming from customers.

We divide the information coming from customers as follows:

• proactive customer's feedback: this information is collected by means of a customer's survey that we send to customers on a yearly basis to collect data related to our device and to user's satisfaction
• communications received from customers that are not classified as customer complaints: this information is documented and handled in Hubspot and follow up by the designated person
• customer complaints: communications received from customers that match the definition of customer complaints.

The analysis of customer complaints is documented in the specific section of this report named Customer complaints, while this section is focused on the analysis and results of the customers feedback that are not classified as customer complaints.

Proactive customer's feedback

Survey results in 2022

We gathered feedback exclusively from professionals within secondary healthcare.

• Performance: In 2022, users in the secondary healthcare specialty rated the application's performance highly, with an average score of 9, indicating that it met or exceeded their expectations for its intended use.
• Ease of use: Respondents found the application easy to use, with an average score of 8.
• Usefulness of information: Users rated the information provided by the application favorably, with an average score of 7, indicating that it was considered valuable for clinical decision-making.
• Remote consultations: About 5 users reported that the application had reduced the time of in-person consultations, suggesting that it had a positive impact on the efficiency of patient care.
• Time optimization: Users agreed on the device's capacity for time optimization, indicating that they found it helpful in managing their time in alignment with patient needs.
• Patient triage: Users found the tool useful for managing patients with different degrees of urgency or priority, with an average rating of 7, highlighting its role in patient prioritization.
• Speed: In terms of speed, the application received an average rating of 8, suggesting that users found it efficient in generating algorithm results.
• Diagnostic support: The tool was highly regarded for its diagnostic support, with an average score of 9, indicating that it was efficient in supporting clinical diagnoses.
• Patient state information: Users found that the application contributed to obtaining more information about the patient's state, which emphasizes its role in enhancing patient monitoring.
• Objectivity: Users reported an average score of 8, indicating that the application increased the objectivity in patient follow-up, potentially reducing subjectivity in clinical assessments.
• General satisfaction: Users were generally satisfied with the application, with an average rating of 9, indicating a high level of overall user satisfaction.
• Recommendation: Users were highly likely to recommend the service to other professionals, with an average rating of 9, suggesting strong confidence in the application's value.

Survey results in 2023

We received responses encompassing health care professionals across both primary and secondary healthcare specialties.

• Performance: Users from both specialties provided positive ratings for the application's performance. The positive trend from 2022 continued in 2023, indicating consistent performance satisfaction.
• Ease of use: Users in 2023 found the application equally easy to use as in 2022, maintaining a high level of usability.
• Usefulness of information: Users in 2023 found the information provided by the application to be valuable, suggesting that the application continued to deliver relevant clinical data.
• Remote consultations: Users in 2023 reported that the application continued to reduce the time of in-person consultations, emphasizing its ongoing role in enhancing efficiency.
• Time optimization: Users from both specialties in 2023 felt that the application helped optimize their time, indicating that it was equally valuable for both groups of professionals.
• Patient triage: Users in 2023 found the tool useful for patient triage, confirming its ongoing role in patient prioritization.
• Speed: The application received positive feedback for speed in 2023, maintaining an efficient performance.
• Diagnostic support: Users in 2023 found the tool efficient as diagnostic support, reinforcing its role in clinical decision-making.
• Patient state information: The application continued to contribute to obtaining more information about the patient's state in 2023, suggesting ongoing enhancements in patient monitoring capabilities.
• Objectivity: In 2023, the application continued to increase objectivity in patient follow-up, highlighting its role in reducing subjectivity.
• General satisfaction: Users were consistently satisfied with the application in 2023, with satisfaction levels matching those from 2022.
• Recommendation: Users in 2023 were equally likely to recommend the service to other professionals, reinforcing the application's value and trustworthiness.

In summary, the survey results in 2023 show a consistent positive trend in user satisfaction and perceived benefits of the device, with users in both primary and secondary health care specialties reporting similar positive experiences. The application appears to have addressed some of the concerns raised in 2022, such as image quality and diagnostic capabilities, resulting in improved ratings and user recommendations. This suggests that ongoing development and enhancements have been successful in meeting user needs and expectations.

Communications received from customer

We have received a total of 22 communications from customers that were not classified as customer complaints: the majority, as further detailed below, were inquiries related to the platform provided for device access, specifically doubts encountered during the login process. Common challenges included misplaced passwords and a general lack of understanding regarding the login procedure.

Moving forward, we are committed to enhancing our customer support resources and user guides to address these issues proactively. This will include clear instructions for password retrieval and detailed steps for logging in, ensuring a smoother, more user-friendly experience for all our customers.

Still, it is important to mention that these challenges are completely solved with the design of the new device, which is accessed via API.

Among the communications received from customers, we identify another category that is technical assistance service communications. If the communication is considered a technical assistance service, it is noted down as a ticket and assigned to the technical responsible to solve it, according to the GP-017 Technical Assistance Service procedure. This type of communications are first screened to understand the type of technical assistance: request for support, request for new features, and bug. In the last case (bug), the communication is considered a customer claim and it will be treated as such.

In the period analysed we received 19 technical assistance tickest: 18 classified as Request for support, that were addressed by the technical team by communicating the steps the customers need to follow to address their inquiries, and 1 classified as Request for new features.

The remaining 3 communications received during the period under evaluation were inquiries from customers not device-related.

Customer access to latest version of the device

One additional activity performed to collect customer feedback is to periodically check the version of our device that customers are using.

The most recent data are available in the R-012-012 Customers product version control record and they are also provided below:

Customer name	Version
DKV	2.1
Bella Aurora	2.1
Santagostino	2.1
VITUP	2.1
Visiba Care	2.1
Consultant Connect	2.1
DermTest	2.1
Pierre Fabre	2.1
Boehringer Ingelheim	2.1
Dermalens	2.1
Hospital Universitario Doctor Peset	2.1
Hospital Universitario Puerta de Hierro	2.1
Torrejón Salud, S.A.	2.1
DKV SERVICIOS SA	2.1
Centro residencial Sanitas Linares	2.1
Hospital Universitario Cruces	2.1
Terapias Médicas Domiciliarias, S.L.U.	2.1
Centro residencial Sanitas Mas Camarena	2.1
Centro residencial Sanitas Alameda de Osuna	2.1
H. U. San Roque en Las Palmas de Gran Canaria	2.1
Centro residencial Sanitas Jardines de Sabatini	2.1
Centro residencial Sanitas Mevefares	2.1
Centro residencial Sanitas Miramon	2.1
Centro residencial Sanitas Loramendi	2.1
Centro residencial Sanitas Las Rozas	2.1
Centro residencial Sanitas Henares	2.1
Centro residencial Sanitas Tarragona	2.1
Centro residencial Sanitas Valladolid	2.1
Doral Medical Research (DMR)	2.1
Centro residencial Sanitas Cornella	2.1
Centro residencial Sanitas Colmenar Viejo	2.1
MEDICAL SCREEN TECHNOLOGY S.A DE C.V.	2.1
Centro residencial Sanitas Carabanchel	2.1
Centro residencial Sanitas Altanova	2.1
Venture Spirit	2.1
Cigna Life Insurance Company of Europe	2.1
Hospital de Vinalopó	2.1
Hospital Universitari Arnau de Vilanova	2.1
iHumanity San Javier SL	2.1
Consultant Connect Clinic	2.1
Hospital Universitario La Moraleja	2.1
IDEI Instituto de Dermatología Integral	2.1
IGDERMA	2.1
Julia Welzel	2.1
MÁS ALLÁ DE LA PIEL, S.L.	2.1
Servicios centrales Sanitas Mayores	2.1
Chirurgische Praxis am Dornbusch	2.1
C.S.R. INVERSIONES SANITARIAS SUR, S.A.	2.1
DKV Cuenta de prueba	2.1
DIAGNOSIS DERMATOLOGICA, S.L.P.	2.1
Asociación de Enfermos de Hidrosadenitis (ASENDHI)	2.1
AREA DERMATOLOGICA IDERMA SL	2.1
Clinica Raul Ferrer	2.1
ACCIÓN PSORIASIS	2.1
DERMOMEDIC CENTER S.L.	2.1
Ann Marie Hake Lilly US	2.1
UCB Demo Account	2.1
Beiesdorf Demo Hospital	2.1
Serénitas, S.L	2.1
Joan Cristóbal	2.1
Fundación de Gestión Sanitaria del HOSSPAU	2.1
Farmacia MIR	2.1
Farmacia Jordi Boncompte	2.1
El nido	2.1
CLINICA DERMATOLOGICA INTERNACIONAL SLU	2.1
Clínica Ismael De la Torre	2.1
Clínica Marisol Ana Pelaez	2.1
Clínica Irene Redondo	2.1
Clínica Marta Mª Serrano Díaz	2.1
Clínica Marta Ruiz	2.1
Clínica Mónica Muñoz	2.1
Clínica Montserrat Ruiz-Olivares	2.1
Clínica Nuria Mendez	2.1
Clínica Pablo Guerreiro	2.1
Clínica Pablo Herrera	2.1
Clínica Patricia Lucía Lora	2.1
Clínica Patricia Moreno	2.1
Clínica Pietro Colombo	2.1
Clínica Pompeya Gonzalez	2.1
Clínica Guiomar Núñez-Morgades	2.1
Clínica Raúl Rubio	2.1
Clínica Ricardo Hernandez	2.1
Clínica Roberto Sanchez Fraile	2.1
Clínica Romualdo Nsue Nzamio	2.1
Clínica Samuel Martinez	2.1
Clínica Sara Garcia	2.1
Clínica Silvia Di Bonaventura	2.1
Clínica Tania Torres	2.1
Clinica Yelitza Carolina	2.1
Clínica Enrique Ciriano	2.1
Clínica Elena Neila	2.1
Dermanostic	2.1
DermoArgent	2.1
Clínica María González	2.1
Clínica Elena Martínez	2.1
Clínica Manuel Sanchez	2.1
Clínica de Carmen Mar Rodriguez	2.1
Clínica Daniel Torres	2.1
Clínica Mª Rosa González	2.1
Farmacia Colldeforn	2.1
Farmacia Guillen	2.1
Clínica Mª del Carmen Baron	2.1
Clínica Mª Ángeles Ruiz	2.1
Farmacia Senante	2.1
Clínica Mª Alicia Urraca	2.1
Clínica Daniel Jesús Fuentes	2.1
Clínica Cristina Pacheco	2.1
Clínica Cristina Garcia	2.1
Clínica César Arévalo	2.1
Clínica Carolina Fernández	2.1
Clínica Carmen Valencia	2.1
Clínica Carmen Sejas	2.1
Clínica Beatriz Eugenia Alvarado	2.1
Clínica Aurora María Araújo	2.1
Clínica Anna Rachael	2.1
Clínica Ángel Pablo Cachero	2.1
Clínica Luis Alberto Gonzalez	2.1
Clínica Ana Maria Montes	2.1
Clínica Alejandro León	2.1
Clínica Alejandro Ibañez	2.1
Clínica Alba Ramos	2.1
Oriol Yelamos	2.1
Clínica Laura Cardeña	2.1
Clínica Agustín Sánchez	2.1
Clínica Adriana Marín	2.1
Clínica Juan Carlos Gordillo	2.1
Clínica Javier Díaz	2.1
Clínica Javier Alonso	2.1
Clínica Maria Palacios	2.1

Customer complaints

We follow the process described in GP-014 Feedback and complaints to handle any communications coming from our customers. As first step, we analyse whether the communication can be classified as a customer complaint.

If this is the case, the communication is labeled as such and an investigation is started to identify the root cause and to define actions to address the customer complaints.

During the period under evaluation we received 6 customer complaints that are summarised below together with actions taken to solve the issue and further details:

Customer complaints	Initiation date	Customer complaints description	Action	Action status	CAPA required	CAPA ID	CAPA plan	CAPA status	Incident	Justification for classification	Notification to CA required
Using diagnosis_support with only one image	2023-12-21	The client reported an issue with the device's endpoint diagnosis_support when a single image is sent to the device. This caused json deserialization (on client's side) to fail	Start investigation and open CAPA	Closed	Yes	R006001-82	Unify the format for single and multiple images	Effectiveness under evaluation	Yes	This complaint indicates that there is a malfunction in the software, therefore it shall be classified as an incdent.	No
Badly formatted json in documentation examples	2023-12-18	The client informed us that the imagingStudySeries described in your documentation is badly formatted json because two of the conclusion objects are missing leading bracket	Update of IFU because it contained a mistake	Closed	No	Not applicable	Not applicable	Not applicable	No	This complaint highlights an issue with the formatting of the JSON in the documentation. While this is a technical documentation issue that could cause inconvenience or confusion for users, it does not relate to the performance or safety of the software in terms of diagnostic accuracy or clinical risk. Since this problem does not affect the device's ability to process images or provide correct diagnostic outcomes, it does not meet the criteria for an incident.	No
API Service down (no response within 60 seconds)	2023-09-13	The client reported that when they send an image to our JSON API, they do not get a response within 60s timeout window	- Update the ia.legit.health endpoint to address the issue promptly - Start investigation and open CAPA	Closed	Yes	R006001-81	Structure Review: Any alterations to the file path structure, including the introduction of new models, will undergo comprehensive evaluation and testing for all active instances. Regular Testing Protocol: A scheduled testing regimen will be established for all active instances. These tests will run daily, ensuring the following criteria are met: - The API is functioning correctly. - HTTP POST requests perform as expected when initiated from external servers. - The responses consistently include the required information for all potential inputs	Closed	Yes	This complaint indicates a potential deterioration of performance of the software.	No
Algorithm performance results did not match with the skin conditions that were being photographed	2023-05-02	The client reported that they have been using our API to review and analyse 50 photos and they spotted that the results from the API did not match with the skin conditions that were being photographed.	Start investigation and open CAPA	Closed	Yes	R006001-56	Since we have demonstrated that the performance of the algorithm is not compromised and the problem reported was caused by the inappropriate quality of the images used for the analysis, the corrective action is to include more detailed specifications on how to take and upload the images in the device's documentation to ensure the best algorithm performance	Closed	Yes	This complaint indicates a potential deterioration of performance of the software.	No
CN Vitup: Confusing AI results	2023-03-01	The client communicated that they got a few incidents related to AI results returning totally different pathology lists for the same skin condition.	Update of the API: better algorithms, multiple image input for reduced variability and enhanced result	Closed	No	Not applicable	Not applicable	Not applicable	Yes	This complaint indicates a potential malfunction or deterioration in the performance of the AI models. The AI is producing inconsistent results (different pathology lists) for the same skin condition, which is a clear deviation from expected performance.	No
Clínica Asunción: AI results are "nonsense"	2022-06-19	The client communicated that he thinks that the dermatologist is better than the algorithm overall, specially for the diagnosis of benign lesions. He also shared an example of a misdiagnoses red hand.	We analyzed the AI result that the client communicated and we found out that the client was expecting the AI to get the correct answer in the top1. Despite the AI is optimised to have the highest sensitivity and specificity, the top1 answer is not always the correct diagnosis. Therefore, we better explain the output of the device to the client.	Closed	No	Not applicable	Not applicable	Not applicable	No	This complaint does not indicate a malfunction, deterioration in performance, or use-error of the software as defined in MDR 2017/745. The customer is comparing the software's performance to that of a dermatologist and expressing a preference for human diagnosis, particularly in the case of benign lesions. However, no specific malfunction or deterioration in the software's performance has been reported. Plus, after talking with the customer, it became clear that the customer didn't fully understand the output of our device.	No

Of the 6 customer complaints received during the analysed period, 3 of them led to the opening of a CAPA and none of them required the initiation of a FSCA.

4 out of 6 customer complaints were classified as incident for the reasons explained in the table above.

None of the customer complaints were classified as serious incident, according to the definition provided in MDR 2017/745, therefore there was no need of notifying the Competent Authorities.

Clinical literature

This activity compiles and evaluates state-of-the-art publications on image diagnostic and severity measure methods. The overall objective of this strategy is to identify, select and collect the relevant literature to determine if the device is safe for its intended use and if there is any emergent risk that we must consider.

The literature review is performed according to the method and process established in the clinical evaluation plan (R-TF-015-001 Clinical evaluation plan) and the results are collected in the clinical evaluation report (R-TF-015-003 Clinical Evaluation Report) and in the post-market clinical follow up report (R-TF-0007-005).

Data related to similar devices

A search for safety incidents and alerts was conducted for the similar products to Legit.Health(SkinVision and MoleScope).

The information on vigilance was obtained from the following vigilance databases:

1. FDA (Food and Drug Administration): Provides Total Product Lifecycle data, including enforcement reports, warning letters, MAUDE database reports, CDRH inspections database, FDA recall database, and TPLC database.
2. Swissmedic: Swiss agency for the authorization and supervision of therapeutic products, offering a recall list of medical devices within the scope of market surveillance.
3. MHRA (Medicines and Healthcare Products Regulatory Agency): An executive agency of the Department of Health in Great Britain, responsible for ensuring the effectiveness and safety of medicines and medical devices.
4. AEMPS Vigilancia de productos sanitarios: A state agency in Spain attached to the Ministry of Health, responsible for guaranteeing the quality, safety, efficacy, and accurate information of medicines and health products.

The adverse events for similar devices were investigated evaluating PMS complaints in various international regulatory databases.

The results are summarized in the following table.

Source of information	Link	Search Team	Alerts	Relevant
FDA website Enforcement Report Searchable database	http://www.fda.gov/Safety/Recalls/EnforcementReports/default.htm	MoleScope, SkinVision	0	N/A
FDA website Warning letters Searchable database	http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/default.htm#recent	MoleScope, SkinVision	0	N/A
FDA website MAUDE – manufacturer and User Facility Device Experience Searchable database	http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfMAUDE/search.CFM	MoleScope, SkinVision	0	N/A
FDA website Medical Device Recalls	http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfRes/textsearch.cfm	MoleScope, SkinVision	0	N/A
FDA website TPLC – Total Product Life Cycle database	http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfTPLC/tplc.cfm	MoleScope, SkinVision	0	N/A
Swissmedic Swiss Competent Authority	https://www.swissmedic.ch/swissmedic/en/home/medical-devices/fsca.html	MoleScope, SkinVision	0	N/A
AEPMS Vigilancia de productos sanitarios	https://alertasps.aemps.es/alertasps/alertas	MoleScope, SkinVision	0	N/A
MHRA Adverse events reporting	https://www.gov.uk/drug-device-alerts	MoleScope, SkinVision	0	N/A

No relevant safety incidents, alerts, or adverse events were found across all databases for MoleScope and SkinVision within the search period. This indicates that similar products have not raised notable safety concerns in international regulatory databases, which supports the anticipated vigilance profile for Legit.Health.

Regulatory requirements

During the period analysed, the following new requirements have been published and considered:

• The Spanish Agency for Medicines and Medical Devices (AEMPS)
  • Real Decreto 192/2023 por el que se regulan los productos sanitarios_7416. It adapts and transposes the European regulatory requirements MDR 2017/745. It does not contain any additional requirements affecting our device that have not been taken into account following the implementation of the MDR.
• The International Medical Device Regulators Forum (IMDRF)
  • IMDRF/AIMD WG/N67 Machine Learning-enabled Medical Devices: Key Terms and Definitions. This guide was consulted to ensure we were aligned with it.
  • IMDRF 2020 Principles and Practices for Medical Device Cybersecurity. We have improved and documented our cybersecurity procedures according to this procedure and to the risk analysis performed.
• MedTech Europe
  • We have not found relevant guidelines that include additional information to our device development processes.
• Harmonized Standards for Medical devices (European Commission)
  • Commission Implementing Decision (EU) 2022/757 of 11 May 2022 amending Implementing Decision (EU) 2021/1182 as regards harmonised standards for quality management systems, sterilisation and application of risk management to medical devices. In this document they specify the new harmonized versions of ISO 13485:2016/A11:2021 and ISO 14971:2019/A11:2021 that we have already considered.
  • Implementing Decision (EU) 2021/1182 as regards harmonised standards for biological evaluation of medical devices, sterilisation of health care products, aseptic processing of health care products, quality management systems, symbols to be used with information to be supplied by the manufacturer, processing of health care products and home light therapy equipment. In this document they specify the new harmonized version of the ISO 15223-1:2021 that we have already considered.
• Public Health Policies (European Commission)
  • We have not found any relevant information that applies to us.
• The US Food and Drug Administration (FDA)
  • Cybersecurity in Medical Devices: Quality System Considerations and Content of Premarket Submissions (September 2023). The first review of the document showed us guidelines completely aligned with our current procedures.
  • Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence/Machine Learning (AI/ML)-Enabled Device Software Functions (April 2023). This guidance establish the recommendations to implement a Predetermined Change Control Plan (PCCP): The documentation describing what modifications will be made to the ML-DSF (Machine Learning-Enabled Device Software Function) and how the modifications will be assessed. We already have this plan in place in our GP-023 Design change management.
• The Notified Body Operations Group (NBOG): As the applicability of Best Practice Guides (BPGs) covering requirements set out in the new medical devices Regulations are contingent upon endorsement by the Medical Device Coordination Group (MDCG), we have reviewed the new guides published by this group and we will modify the procedure to update that the review is done in the MDCG and not in the NBOG. The new guideline published since our last external document review (R-001-005 List of external documents_2023_003) that directly apply to our procedures is:
  • Manual on borderline and classification under Regulation (EU) 2017/745 and Regulation (EU) 2017/746, but it did not include any aspect that affect our device classification.

Cybersecurity and state of the art

During the period analyzed the following websites have been visited to follow their recommendations:

• European Union Agency for Cybersecurity (ENISA): we are aligned with all the recommendations they provide in relation to cybersecurity and medical devices. As it recommends healthcare organisations to prioritise the procurement of devices and services certified under cybersecurity schemes/standards, we have integrated ISO 27001 concepts into our QMS.
• US Cybersecurity and Infrastructure Security Agency (CISA): as with the ENISA, we are aligned with all the recommendations provided by CISA. Additionally, we have not found any cybersecurity incidence related to our medical device nor medical devices similar to ours.

PMCF activities

The activities planned for the post-market phase to collect clinical evidence about the device are listed and described in the R-TF-007-002 Post-Market clinical follow-up (PMCF) Plan.

Results and conclusions of the planned activities are described in the R-TF-007-005 Post-Market Clinical Follow-up (PMCF) evaluation report.

Conclusions

Overall conclusions

Based on the data collected and analysed according to the activities defined for the post-market phase, we can conclude that our device demonstrates a strong safety and performance record. The absence of serious incidents and FSCAs is indicative of the device's reliability and the effectiveness of our quality management and surveillance systems.

The data on device utilization, particularly in clinical settings for first and follow-up consultations, provides valuable insights into the device's applicability and potential impact on patient care. The equitable usage and representation section confirms our commitment to ensuring the device's accessibility and effectiveness across diverse patient populations.

Necessary actions have been taken to address customer complaints, incidents, non-conformities and CAPA generated from them.

The 7 non-serious incident registered during 2023 have been addressed in a timely manner and corrective actions have been implemented to avoid recurrence of same issues.

Most of the CAPA (28 out of 30) have been implemented and successfully closed because we could verified their effectiveness (25 out of 30, 3 CAPA are under evaluation to verify the effectiveness), indicating our effort to address issues and take actions without undue delay (always prioritizing issues to be addressed with a risk-based approach).

We are committed to monitor, evaluate, and improve our device to ensure patient safety, device efficacy, and quality excellence.

Risk analysis considerations

As part of our continuous post-market surveillance activities, all customer complaints and vigilance data received have been thoroughly evaluated to assess their potential impact on the existing risk profile of the device. Specifically, we analyze whether the reported issues affect previously identified risks or introduce new risks that had not been anticipated during the initial risk assessment.

For each complaint and vigilance case, an investigation is conducted to determine the root cause and its relevance to the current risk management file. If the findings from this analysis suggest a need for updates to the risk analysis (e.g., changes in risk severity, likelihood, or new risk introduction), the risk management documentation is accordingly revised to reflect these insights, as also explained in the procedures GP-004 Vigilance system and GP-014 Feedback and complaints.

By integrating this feedback loop, we ensure that our risk management process remains dynamic and aligned with real-world device performance, ultimately enhancing the safety and effectiveness of the product.

In the course of this post-market surveillance period, all relevant data were systematically reviewed and analysed. As part of this process, the impact of the received data on the device's risk profile was carefully evaluated.

After investigation, it was determined that the data collected during this period did not introduce any new risks nor did it significantly alter the severity or probability of the existing risks. Therefore, no changes were deemed necessary to the current risk analysis documentation.

This evaluation confirms that the device continues to operate within its expected safety and performance parameters, and no modifications to the risk management file are required at this time.

Signature meaning

The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:

• Author: Team members involved
• Reviewer: JD-003, JD-004
• Approver: JD-001