R-TF-025-004 Summative evaluation protocol

Table of contents

• List of Tables
• List of Figures
• Terminology and Definitions
• Applicable Standards and Guidance
• Internal References
• Introduction
• Description of Intended Device Users, Uses, Use Environments, and Training
  • Intended Device Users
  • Intended Uses
  • Intended Use Environments
  • Training
• Description of Device User Interface
  • Device User Interface
  • Summary of Operating Sequence
• Summary of Known Use Problems
• Analysis of Hazards Associated with Use of the Device
• Summary of Preliminary Analyses and Evaluations
• Description and Categorization of Critical and Non-critical Tasks
• Details of Human Factors Validation Testing
  • Study Objectives
  • Type of Testing
  • Study Test Environment
    • Realism
    • Study Locations
    • Study Devices and Materials
  • Participants
    • Number, Type, and Rationale
    • Participant Protection
  • Tasks and Scenarios
    • Session Overview
    • Use Scenarios and Tasks
  • Description of Data Collection and Methods
    • Recording Equipment
    • Recording Software
  • Data Analysis
    • Description of Root Cause Analysis
    • Description of Residual Risk Analysis
• Appendix A: Interview Guide
  • Testing Session
  • ITP Use Scenario 1: Simulated Use
    • Post-Scenario Interview
  • HCP Use Scenario 1: Simulated Use: No Lesion
    • Post-Scenario Interview
  • HCP Use Scenario 2: Simulated Use: Lesion
    • Post-Scenario Interview
  • HCP Use Scenario 3: Knowledge Assessment
    • Post-Scenario Interview
    • Thank and Exit
• Appendix B: Electronic IFU
• Appendix C: List of participants recruited for summative evaluation

List of Tables

• Table 1: Product Interface
• Table 2: Summary of Known Use Problems
• Table 3: Severity Ratings in URRA
• Table 4: Use-related Risk Analysis
• Table 5: Critical and Non-critical Tasks
• Table 6: HCP User Interface for Usability Testing
• Table 7: User Group Details
• Table 8: Participant IDs
• Table 9: Session Overview
• Table 10: Use Scenarios and Descriptions for ITPs
• Table 11: Use Scenarios and Descriptions for HCPs

List of Figures

• Figure 1: Visual Code Studio IDE Interface for ITPs
• Figure 2: Malignant Skin Lesions (Squamous Cell Carcinoma, Melanoma, & Basal Cell Carcinoma)

Terminology and Definitions

• AI: Artificial Intelligence
• API: Application Programming Interface
• CADe: Computer-assisted detection
• CC: Close call
• CFR: Code of Federal Regulations
• DO: Doctor of Osteopathic Medicine
• e.g.,: For example
• EHR: Electronic Health Record
• EMR: Electronic Medical Record
• FDA: Food and Drug Administration
• FHIR: Fast Healthcare Interoperability Resources
• HCP: Healthcare provider
• HD: High definition
• HF: Human factors
• i.e.,: That is
• ICD: International Classification of Diseases
• ICF: Informed consent form
• IDE: Integrated Development Environment
• IFU: Instructions for Use
• IL: Illinois
• IRB: Institutional Review Board
• ITP: Information Technology Professionals
• MD: Doctor of Medicine
• ML: Machine Learning
• OK: Success
• PA: Pennsylvania
• PID: Participant identification code
• RCA: Root cause analysis
• UD: Use difficulty
• UE: Use error
• uFMEA: Use failure mode and effects analysis
• URRA: Use-related risk analysis
• US: United States

Applicable Standards and Guidance

This protocol is written in accordance with the following standards:

• ANSI/AAMI HE75:2009/(R)2018 Human Factors Engineering-Design of Medical Devices
• ANSI/AAMI/IEC 62366-1:2015+AMD1:2020 Medical Devices Part 1: Application of Usability Engineering to Medical Devices
• AAMI/IEC TIR62366-2:2016 Medical Devices Part 2: Guidance on the Application of Usability Engineering to Medical Devices
• ANSI/AAMI/ISO 14971:2019 Medical Devices—Application of risk management to medical devices
• FDA Final Guidance for Industry and FDA Staff: Applying Human Factors and Usability Engineering to Medical Devices (February 3, 2016)

In addition to the standards and guidance documents listed, draft guidance documents were reviewed and considered to understand the current thinking of the FDA on relevant topics.

Internal References

• R-TF-012-014
• REQ*001 The user receives quantifiable data on the intensity of clinical signs * QMS
• REQ*002 The user receives quantifiable data on the count of clinical signs * QMS
• REQ*003 The user receives quantifiable data on the extent of clinical signs * QMS
• REQ*004 The user receives an interpretative distribution representation of possible ICD categories represented in the pixels of the image * QMS
• REQ*005 The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner * QMS
• REQ*006 The data that users send and receive follows the FHIR healthcare interoperability standard * QMS
• REQ*007 If something does not work, the API returns meaningful information about the error * QMS
• REQ*008 Notify the user if the image does not represent a skin structure * QMS
• REQ*009 Notify the user if the quality of the image is insufficient * QMS
• REQ*010 The device detects if the image is of clinical or dermatoscopic modality * QMS
• REQ*012 Users can easily integrate the device into their system * QMS

Introduction

AI Labs Group (hereafter, the manufacturer) has developed a new software-aided adjunctive diagnostic application programming interface (API) intended to assess clinical atypical cutaneous lesions that are suspicious for skin cancer or other skin conditions. This software-only medical device is intended for use by healthcare information technology professionals (ITPs) to integrate into their hospital systems' electronic medical record (EMR) system, and healthcare practitioners (HCPs) with varying degrees of training in clinical diagnosis and management of skin lesions. Throughout this document, this API will be referred to as the device.

The device is a prescription device that incorporates Artificial Intelligence/Machine Learning (AI/ML) technology, including computer-assisted detection (CADe), which analyzes images or other physical characteristics of a skin lesion. The software provides quantifiable data on the intensity, count, and extent of clinical signs, and an interpretative distribution representation of possible International Classification of Diseases (ICD) classes to aid in determining whether a patient should be referred to a dermatologist.

Design Science will conduct a human factors (HF) validation study to determine if the device and its user interface can be used safely and effectively by all its intended users, for its intended uses, and in its intended use environments. The protocol that follows documents the methodology that will be used.

This validation study will take place in Philadelphia, PA and Evanston, IL from TBD. The study will include simulated-use testing with ITP and HCP participants, who are representative users, to evaluate their use of the product in representative clinical environments.

Description of Intended Device Users, Uses, Use Environments, and Training

Intended Device Users

The intended user population consists of the following distinct user groups:

• IT Professionals (ITPs): IT professionals are responsible for the integration of the medical device into the healthcare organization's EMR system. It is advisable that they have a basic knowledge of Fast Healthcare Interoperability Resources (FHIR) and the output of the device. They are individuals aged 18 years and older and can be from various educational backgrounds. Some of them may have vision, hearing, or dexterity impairments.
• Healthcare Providers (HCPs): Medical professionals (e.g., physicians), who have varying degrees of training in the clinical diagnosis and management of skin lesions, and will utilize the output from the software integration for diagnosis. All of them will have the qualifications and competencies native to their profession, and knowledge on how to take images with smartphones.

Intended Uses

The device is a computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organizations to gather data and improve their workflows. The data generated are intended to aid healthcare practitioners and organizations in their clinical decision-making process. It is not meant to confirm a clinical diagnosis nor replace the role of a dermatologist, but rather to obtain additional information to consider a decision.

The device is indicated for use on images of visible skin structure abnormalities to support the assessment of all diseases of the skin incorporating conditions affecting the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral, and genital), the dermis, the cutaneous vasculature, and the subcutaneous tissue (subcutis).

Intended Use Environments

The device is intended to be used in the setting of healthcare organizations and their IT departments, which commonly are situated inside hospitals or other clinical facilities. The device is intended to be integrated into the healthcare organization's system by IT professionals.

The environments are further described as follows:

• IT Office Environment: In the context of healthcare facilities, an IT office is typically designated for hospital IT staff to within the hospital or clinic. The setting is expected to have standard, controlled indoor room temperature, humidity, noise, and lighting conditions. However, during actual use, the intended use environment might vary in ways that impact users' abilities to use the product safely and effectively. ITPs may experience natural distractions from colleagues.
• Clinical Environment: A standard clinical environment is typically designed with a clear layout that has designated areas for patients to be assessed for cutaneous lesions where HCPs can take photos of patients and utilize the device. This setting is also expected to adhere to cleanliness and sanitation standards to prevent infections and include a sufficient inventory of medical supplies. The setting is expected to have standard, controlled indoor room temperature, humidity, noise, and lighting conditions. However, during actual use, the intended use environment might vary in ways that impact users' abilities to use the product safely and effectively. HCPs may experience natural distractions from colleagues and patients. Additionally, the use of gloves can affect HCP's interactions and manual dexterity.

Training

The manufacturer does not expect that users will receive training prior to using the device.

Description of Device User Interface

Device User Interface

This software-only medical device consists of a software-aided adjunctive diagnostic application programming interface (API). As such, the user interface of the device software-only medical device consists of the API endpoints, documentation, and data structures, specifically, the electronic instruction for use (IFU) for both user groups as well as API endpoints. These endpoints are used by ITPs for integrating into the hospital IT system. HPCs will then use the user interface of the hospital IT system to interact with the device software-only medical device.

The components of the device API user interface are detailed in Table 1.

Interface Item	Written Description
API Endpoints	URL structures, methods and request structure, response structure, and authentication
API Documentation	Electronic IFUs
Data Structure	JSON payload formats, including field names and FHIR nomenclature

Table 1: Product Interface

Summary of Operating Sequence

The device installation sequence to be conducted by ITPs consists of the following:

1. Obtain credentials
2. Gain access token
3. Build JSON with data
4. Send JSON to device
5. Receive JSON with device
6. Process and store JSON
7. Build Report

Refer to the User Guide for a detailed description of the primary operating sequence expected for the device.

For operation by healthcare providers, the device analyzes images taken of lesions and other skin abnormalities and produces a list of potential conditions.

The primary operating sequence consists of the following:

1. Take a picture of the lesion with a smart phone
   1. Image should be close to the lesion, focused, and well lit
   2. The lesion should be the main item in the photo
2. Upload the image to the device client
3. Run the analysis
4. Review the data output

Refer to the User Guide for a detailed description of the primary operating sequence expected for the device.

Summary of Known Use Problems

The manufacturer has conducted research on known or expected use problems for similar products and product types.

Based on this research, Table 2 describes known use problems pertaining to software-aided adjunctive diagnostic APIs, along with how these are addressed. All of these, along with additional consideration for industry-wide issues, have been identified and documented in the device's risk assessment. These problems are also included in the evaluated during the validation study through observations.

Type of Use Problem	Description	Applicability to Product	Design Mitigation
User authentication and security	APIs in healthcare settings must adhere to strict security standards. Past experiences have highlighted issues related to complex authentication processes and maintaining compliance with security protocols	Applicable	Implementation of data encryption, robust authentication mechanisms such as OAuth or JWT (REQ_005), monitoring of security threats, cybersecurity info in IFU (REQ_012)
Error handling and reporting	Inadequate and/or unclear error messages or lack of real-time error reporting can significantly impact the user's ability to diagnose and fix issues promptly	Applicable	Clear error messages (REQ_007), comprehensive IFU details on correct endpoints (REQ_012), technical support, RESTful API implementation (REQ_005)
Interoperability	Ensuring interoperability with various healthcare information systems, including EHR systems, has been a challenge. Compatibility issues can lead to data mismatches or loss, critical in healthcare applications	Applicable	Elastic demand design, constant backups, advanced security/software (REQ_005), REST features for status feedback, error codes, automatic awareness of downtime, support
Integration challenges	Difficulties encountered during the integration process, primarily due to compatibility issues with existing systems	Applicable	Clear error messages (REQ_007), comprehensive IFU with correct endpoints (REQ_012), technical support, RESTful API implementation (REQ_005)
Insufficient documentation	Inadequate, unclear and/or incomplete guidelines, leading to incorrect implementation and usage errors	Applicable	IFU-provided information (REQ_012), clear error messages (REQ_007), technical support for troubleshooting

Table 2: Summary of Known Use Problems

Analysis of Hazards Associated with Use of the Device

The manufacturer has conducted a comprehensive task analysis on the use of the device, which led to a list of potential use errors and a comprehensive use-related risk analysis (URRA). This process involved analyzing known use problems with similar devices, identifying the user interface characteristics related to safety, identifying potential use errors, and identifying known and foreseeable hazards and hazardous situations.

Please refer to R-TF-012-014 for the full use-related risk analysis (URRA).

Table 3 defines the possible severity ratings applied in the URRA.

Table 4 lists the applicable use steps and their severities. It also summarizes the URRA, including tasks, potential use errors, hazards and harms, severities, and risk mitigation measures.

Severity Classification	Harm Severity Descriptions by Risk Type	Severity Rating
Catastrophic	Results in patient death.	5
Critical	Results in permanent impairment or life-threatening injury.	4
Serious	Results in injury or impairment requiring professional medical intervention.	3
Minor	Results in temporary injury or impairment not requiring professional medical intervention.	2
Negligible	Inconvenience or temporary discomfort.	1

Table 3: Severity Ratings in URRA

Filter by type:AllUsabilityCybersecurityProductRegulatoryArtificial Intelligence(15)

ID? Stable unique identifier in the format "R-XXX" (letter R, dash, three alphanumeric characters). Persist this across revisions to preserve traceability and version history.	Hazard or Use Error? Hazard or Use Error. A hazard is a potential source of harm (ISO 14971). A use error is an act or an omission by the user that causes a device response different from the one intended by the manufacturer or expected by the user (IEC 62366-1). Use the term that fits the risk's context.	Type? One or more lenses that apply to this risk: Usability (human factors), Cybersecurity, Product (device safety), Regulatory, Artificial Intelligence. Select multiple when the risk cuts across lenses.	Hazardous Situation or Vulnerability? If Type includes Cybersecurity, record the specific vulnerability that an attacker can exploit. Otherwise, record the hazardous situation, which is the circumstance that exposes people, property, or the environment to one or more hazards (ISO 14971).	Foreseeable sequence of events? Foreseeable sequence of events. List the ordered, realistic steps—including reasonably foreseeable misuse—that can lead from the initiator to a hazardous situation and then to harm (ISO 14971).	Harm? Potential injury or damage to health, or damage to property or the environment (ISO 14971). For AI‑related risks, also consider impacts on fundamental rights such as discrimination or undue bias when applicable (EU AI Act).	Risk or Threat? Concise statement of the risk or the threat scenario. For risks in the ISO 14971 sense, think "combination of the probability of occurrence of harm and the severity of that harm." For cybersecurity, frame as "adversary capability exploiting a vulnerability leads to impact."	Security (CIAA)? Security properties affected: Confidentiality, Integrity, Availability, and Authenticity (CIAA). Required when Type includes Cybersecurity.	User group? Intended user group for usability risks. Examples: Healthcare Professional (HCP) or Information Technology Professional (ITP). Optional for other risk types.	User task? Specific task from the validated use scenarios that the user is performing when the risk can occur. Required for usability risks.	Cause Requirement(s)? Requirement codes (PRS-*) whose absence, defect, or violation could plausibly cause or contribute to the sequence of events.	Affected Asset, Part or People? Assets, components, users, patients, or organizations that are impacted by the risk. Examples: patient, device, integration, managing organization, data set.	Likelihood (Initial)? Initial likelihood rating from 1 to 5. Interpret as the estimated probability that the hazardous situation will lead to harm before any controls. For cybersecurity, interpret as exploitability given exposure and attack preconditions.	Severity (Initial)? Initial severity rating from 1 to 5. Interpret as the magnitude of the foreseeable harm outcome before any controls.	RPN (Initial)? Initial Risk Priority Number from 1 to 25, calculated as Severity (Initial) multiplied by Likelihood (Initial). Use for prioritization; it does not replace ISO 14971 risk‑acceptability criteria.	Control Opt (ABC)? Chosen risk control option or options in the ISO 14971 priority order: A. Inherently safe design and construction; B. Protective measures in the device or in the manufacturing process; C. Information for safety and, when appropriate, training to users.	Implemented mitigation measures? Implemented risk control measures and safeguards: design features, protective mechanisms, procedures, information for safety, training, and technical security controls.	Mitigation or Control Requirement(s)? Requirement codes (PRS-*) that implement the selected risk control measures to maintain full traceability.	Responsible? Role accountable for planning, implementing, and verifying the risk control measures.	Verification of implementation of risk control measures? Objective evidence that risk control measures have been implemented as specified. Examples: test protocols and reports, design reviews, configuration and release records.	Severity (Controlled)? Controlled severity rating from 1 to 5 after controls. Severity often remains unchanged unless the control reduces the potential magnitude of harm.	Likelihood (Controlled)? Controlled likelihood rating from 1 to 5 after controls. For cybersecurity, reflect the residual exploitability after mitigations.	RPN (Controlled)? Controlled Risk Priority Number from 1 to 25, calculated as Severity (Controlled) multiplied by Likelihood (Controlled).	Residual risk evaluation? Residual risk band derived from the controlled RPN. 1 to 5: Acceptable. 6 to 12: As far as possible. 13 to 25: Not acceptable.	Verification of effectiveness of risk control measures? Objective evidence that risk control measures are effective in reducing risk in the intended context. Examples: usability validation per IEC 62366‑1, clinical evaluation where applicable, cybersecurity verification and validation, penetration testing.	Benefit-risk analysis? Benefit–risk analysis rationale. Required when the residual risk band is As far as possible or Not acceptable. Summarize the clinical or organizational benefits that outweigh the residual risk and explain why further risk reduction is not reasonably practicable.	Risks arise from risk control measures?? Indicate whether the implemented controls introduce new hazards, hazardous situations, vulnerabilities, or use errors. If yes, create linked risk rows and evaluate them.	Is risk control complete?? Confirm that all planned risk control measures have been implemented, verified, and shown effective, and that any secondary risks have been addressed.	Overall residual risk acceptability? Overall residual risk acceptability. Record the management decision per the risk management plan's criteria, including any aggregated considerations and disclosure of residual risk. This is not derived solely from row completion.	Threat Model Ref(s)? References to the relevant threat‑model elements for cybersecurity risks. Examples: STRIDE items, attack trees, misuse cases, architecture nodes.	Post-Market Plan Ref(s)? References to production and post‑production monitoring activities that will detect, trend, or respond to this risk after release. Examples: human‑factors postmarket follow‑up, field security monitoring, incident response, complaint trending.
R-HBD	Misrepresentation of magnitude returned by the device	🤳 Usability⚙️ Product	The care provider's system represent a value as if was representing a different magnitude.	The device analyses an image of the skin structure and provides the outputs to HCPs Due to a software bug, data formatting issue, or improper interpretation by the healthcare system, the magnitude is incorrectly represented in the output	Misdiagnosis; delay in diagnosis/patient's follow up/treatment	The name of the endpoints of the device do not follow a standard		👨‍💻ITP	Use a stable internet connection	PRS-1XUPRS-5LJ	Managing Organisation	3	3	9⚠️	AC	The endpoints of the device follow HL7's FHIR interoperability standard and we developed a integration manual within the IFU that explains the values, as well as a Swagger documentation	PRS-1XUPRS-5LJ	Technical director	IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information. HL7's FHIR standard compliance is verified at the TEST_013_The data that users send and receive follows the FHIR healthcare interoperability standard.	3	1	3✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable
R-DAG	Incorrect diagnosis or follow up	🤳 Usability🏛️ Regulatory	The medical device outputs a wrong result	The device processes an image of the skin structure Due to a software malfunction, algorithmic error, data corruption, or integration issue, the device outputs incorrect clinical information The HCP, unaware of the malfunction, relies on the device output	This could lead to misdiagnosis; delays in treatment and worsening of the patient's health status.	The interpretive distribution assigns a low probability to the correct ICD class among the potential ICD classes.		👩‍⚕️HCP	User logs into the system.	PRS-1V6PRS-1XUPRS-5LJPRS-8QJPRS-9J5	Patient	4	3	12⚠️	AC	Information about device outputs are detailed in the IFU. The medical device returns metadata about the output that helps supervising it, such as explainability media and other metrics. The device returns an interpretative distribution representation of possible ICD categories, not just one single condition. AI models undergo retraining using expanded dataset of images.	PRS-1V6PRS-1XUPRS-5LJPRS-8QJPRS-9J5	Technical director	Process for verification is defined in GP-012 Design, redesign and development. Implementation of device output information in the IFU verified in TEST_011. Verification of the implementation of metadata about the output of the device to help supervising the output: TEST_001, TEST_002, TEST_003. Verification of the implementation of interpretative distribution representation of possible ICD categories verified in TEST_004.	3	2	6⚠️	⚠️As far as possible	R-TF-012-015 Summative evaluation report_2024_001 R-TF-015-003 Clinical Evaluation Report (sections: Instructions for Use, Associated Design Product Requirement, Valid clinical association of the International Classification of Diseases (ICD) classes, Clinical performance)	Benefits outweigh the risks	FALSE	TRUE	✅Acceptable
R-SKK	Incorrect results shown to patient	🤳 Usability🔐 Cybersecurity🏛️ Regulatory🧠 Artificial Intelligence	The patient see erroneous results.	The device processes an image of the skin structure Due to a software malfunction, algorithmic error, data corruption, or integration issue, the device outputs incorrect clinical information The HCP, unaware of the malfunction, relies on the device output	The patient is affected and may suffer anxiety or delays visiting the HCP and their consequent treatment; worsening their health status.	The interpretive distribution assigns a low probability to the correct ICD class among the potential ICD classes.,Patient is using the device without the HCP monitoring	IntegrityAvailability	👩‍⚕️HCP	User takes a photo of the patient's lesion.	PRS-1V6PRS-1XUPRS-4QWPRS-5LJPRS-8QJPRS-9J5	Patient	4	3	12⚠️	AC	Information about device outputs are detailed in the IFU. The medical device returns metadata about the output that helps supervising it, such as explainability media and other metrics. The device returns an interpretative distribution representation of possible ICD categories, not just one single condition. AI models undergo retarining using expanded dataset of images.	PRS-1V6PRS-1XUPRS-4QWPRS-5LJPRS-8QJPRS-9J5	Technical director	Process for verification is defined in GP-012 Design, redesign and development. Implementation of device output information in the IFU verified in TEST_011. Verification of the implementation of metadata about the output of the device to help supervising the output: TEST_001, TEST_002, TEST_003. Verification of the implementation of interpretative distribution representation of possible ICD categories verified in TEST_004.	3	1	3✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001 R-TF-015-003 Clinical Evaluation Report (sections: Instructions for Use, Associated Design Product Requirement, Valid clinical association of the International Classification of Diseases (ICD) classes, Clinical performance)	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable	T-024-006-AML-001T-024-006-DAT-002	T-024-007-AUD-001T-024-007-CVE-002
R-E7Z	Inaccessible skin areas	🤳 Usability	The device cannot analyse certain skin areas	The HCP or the patient attempts to capture an image of the patient's skin to analyze potential conditions Certain areas of the skin (e.g., underarm, groin, scalp, or other folds) are difficult to access or visualize due to their location or the patient's body position The device analyses images with poor quality or not containing enough details of the skin abnormality intended to be captured The HCP relies on the available analysis results from accessible areas	Misdiagnosis; delays in treatment and worsening of the patient's health status.	Inability to access or take a picture of the skin structure due to its location in an unreachable body site and lack of aid in the process		👩‍⚕️HCP	User takes a photo of the patient's lesion.	PRS-1V6PRS-1XUPRS-5LJPRS-7XK	Patient	3	3	9⚠️	AC	A requirement of the device defines the creation of a processor whose purpose is to ensure that the image have enough quality. In other words, an algorithm, similar to the ones used to classify diseases, is used to check the validity of the image and provides an image quality score. The device returns meaningful messages to the users about the quality score of the images. This allows care providers to re-take a photo. The IFU contain the `How to take pictures` section with recommendation on how to take pictures with high quality plus in the Contraindications section of the IFU we state the following: We advise the user not to use the device if skin structures are not accessible by a camera, such as being located in a skin fold or is otherwise covered. We also offer training to the users to optimize the imaging process so that it is optimal for the device's operation.	PRS-1V6PRS-1XUPRS-5LJPRS-7XKPRS-9F2	Technical director	TEST_009_Notify the user if the quality of the image is insufficient TEST_007_If something does not work, the API returns meaningful information about the error TEST_011_We facilitate the integration of the device into the users' system R-TF-001-006 IFU and label validation	3	1	3✅	✅Acceptable	R-TF-015-003 Clinical Evaluation Report (sections: Associated Design Product Requirement, Associated Design Verification Test, Clinical performance) R-TF-012-015 Summative evaluation report_2024_001	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable
R-TA9	Inadequate camera usage or settings	⚙️ Product🤳 Usability	Poor image quality due to inadequate resolution, lighting, focus or camera settings	A camera is used to capture images of the skin structures Due to improper settings (such as inadequate lighting, incorrect resolution, or focus issues) or the use of a low-quality camera, the captured images are unclear or fail to capture sufficient detail These suboptimal images are input into the medical device for analysis The device's algorithm processes the poor-quality images, leading to decreased accuracy The medical device outputs potentially misleading or incomplete results	Misdiagnosis; delays in proper treatment and worsening of the patient's health status.	Inadequate image processing algorithms		👨‍💻ITP	Input authorization details	PRS-1V6PRS-1XUPRS-2ZBPRS-3YHPRS-5LJPRS-7XK	Managing Organisation	4	3	12⚠️	AC	A requirement of the device defines the creation of a processor whose purpose is to ensure that the image have enough quality. In other words, an algorithm, similar to the ones used to classify diseases, is used to check the validity of the image and provides an image quality score. The device returns meaningful messages about the quality score of the images. This allows care providers to re-take a photo. The IFU contain a dedicated section on how to take pictures (section name: `How to take pictures`) and technical specifications of the camera (section: `Technical specifications`) We also offer training to the users to optimize the imaging process so that it is optimal for the device's operation.	PRS-1V6PRS-1XUPRS-2ZBPRS-3YHPRS-5LJPRS-7XK	Technical director	Verification is defined in TEST_009_Notify the user if the quality of the image is insufficient. TEST_007 verifies REQ_007. TEST_011 and R-TF-001-006 IFU and label validation verify REQ_012	3	1	3✅	✅Acceptable	R-TF-015-003 Clinical Evaluation Report (sections: Validation and testing of machine learning models, Post-market Clinical investigations (with focus on clinical studies whose primary endpoint is improving diagnostic accuracy)) R-TF-012-015 Summative evaluation report_2024_001	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable
R-UI5	Inadequate instructions for use: product information for cybersecurity is not included in the IFU	🤳 Usability🔐 Cybersecurity🏛️ Regulatory	Presence of vulnerabilities that may compromise the integrity of the system and patient data	The instructions for use (IFU) do not provide necessary cybersecurity guidelines, such as how to secure device communication, implement encryption, or manage authentication protocols ITPs, unaware of the specific cybersecurity requirements for the medical device, may integrate the device without implementing appropriate security measures The device is integrated into the healthcare system with weak or default security settings, leading to vulnerabilities in data transmission and system access Due to the inadequate cybersecurity setup, malicious actors may exploit the system, gaining unauthorized access to patient data or the medical device's functionalities Patient data, such as medical images or diagnostic outputs, could be intercepted or altered during transmission, leading to compromised or inaccurate results	Unauthorized access to sensitive patient information; incorrect diagnosis; loss of trust	Inadequate information provided by the manufacturer	ConfidentialityIntegrityAvailabilityAuthenticity	👨‍💻ITP	Input authorization details	PRS-1V6PRS-1XUPRS-2KQPRS-7Z8PRS-9F2	Manufacturer	4	3	12⚠️	C	We specify in the IFU the product information for cybersecurity in the section `Security requirements and recommendations`	PRS-1V6PRS-1XUPRS-2KQPRS-7Z8PRS-9F2	Technical director	IFU verification documented in TEST_011 and in R-TF-001-006 IFU and label validation 2023_001	3	1	3✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable	T-024-006-DOC-001T-024-006-AUT-004	T-024-007-EDU-001T-024-007-VUL-004
R-5L4	Inadequate lighting conditions during image capture	🤳 Usability⚙️ Product	The medical device receives an input that does not have sufficient quality	Images of skin structures are taken in an environment with poor lighting Due to the inadequate lighting, the image lacks sufficient contrast, clarity, or detail, making it difficult for the medical device to properly analyze the image The low-quality image is submitted to the medical device for processing The medical device, unable to accurately process the poorly lit image, generates an incorrect or low-confidence output The HCP receives a list of potential skin diseases with incorrect or misleading probabilities due to the low-quality image	Misdiagnosis; delays in proper treatment and worsening of the patient's health status.	Inadequate image processing algorithms		👩‍⚕️HCP	User takes a photo of the patient's lesion.	PRS-1V6PRS-1XUPRS-2ZBPRS-5LJPRS-7XK	Patient	4	3	12⚠️	AC	A requirement of the device defines the creation of a processor whose purpose is to ensure that the image have enough quality. In other words, an algorithm, similar to the ones used to classify diseases, is used to check the validity of the image and provides an image quality score. The device returns meaningful messages about the quality score of the images. This allows care providers to re-take a photo. The IFU contain a dedicated section on how to take pictures (section name: `How to take pictures`) and technical specifications of the camera (section: `Technical specifications`) We also offer training to the users to optimize the imaging process so that it is optimal for the device's operation.	PRS-1V6PRS-1XUPRS-2ZBPRS-5LJPRS-7XK	Technical director	Verification is defined in TEST_009_Notify the user if the quality of the image is insufficient. TEST_007 verifies REQ_007. TEST_011 and R-TF-001-006 IFU and label validation verify REQ_012	3	2	6⚠️	⚠️As far as possible	R-TF-015-003 Clinical Evaluation Report (sections: Validation and testing of machine learning models, Post-market Clinical investigations (with focus on clinical studies whose primary endpoint is improving diagnostic accuracy)) R-TF-012-015 Summative evaluation report_2024_001	Benefits outweigh the risks	FALSE	TRUE	✅Acceptable
R-2S3	Integration failure or errors	🤳 Usability⚙️ Product	Failure to communicate with other systems	The ITP begins the process of integrating the medical device's API into the healthcare organisation's existing systems Due to system incompatibilities, incorrect configuration, or software bugs, the integration process encounters errors. These could include issues like incorrect API calls, data format mismatches, or failure of the device to communicate properly with the existing infrastructure The errors cause the integration to be either incomplete or improperly implemented, meaning the device may not fully communicate with other systems or may only function intermittently Due to the faulty integration, device's outputs may not be transferred correctly between the device and other healthcare systems, leading to missing, corrupted, or delayed data The HCPs may receive incorrect, incomplete, or delayed results, or in some cases, no results at all	Misdiagnosis; delayed treatment; loss of trust in the device	Inadequate information provided by the manufacturer		👨‍💻ITP	Input authorization details	PRS-1V6PRS-1XUPRS-5LJ	Managing Organisation	4	3	12⚠️	C	We specify the intended user and the required qualification in the IFU Additionally, we include at the IFU the instructions and information required by the ITPs to perform the integration of the device within their system	PRS-1V6PRS-1XUPRS-5LJPRS-9F2	Technical director	IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information.	3	1	3✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable
R-HAX	Incorrect interpretation of device outputs	🤳 Usability🏛️ Regulatory	The HCP validates the wrong skin condition, even if the device outputs the correct result	The ITP integrates the device into the healthcare organisation's system The ITP implements the interface for the HCP The medical device analyses images and provides outputs The HCP misinterprets the device's outputs due to lack of clarity in the presentation of data or due to misunderstanding of device's outputs The HCP decides to ignore the device's results	Incorrect or delayed diagnosis; inappropriate treatment or follow-up; loss of confidence in the device	Inadequate information provided by the manufacturer		👩‍⚕️HCP	User takes a photo of the patient's lesion.	PRS-1XU	Patient	3	3	9⚠️	C	The IFU explains the medical device's intended purpose The IFU explain the device's outputs The IFU contain a specific section (`User interface`) in which we explain the minimum requirements for the user interface that the ITP will implement	PRS-1XU	Technical director	IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information.	3	1	3✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable
R-O5Y	Complicated instructions for use: the instructions for use are too complicated and more intricate than they need to be	🤳 Usability🏛️ Regulatory	Misinterpretation of IFU	ITP starts the integration of the medical device into the healthcare organisation's system ITP consults IFU to follow the proper integration steps ITP does not fully understand the integration process due to complex instructions ITP may incorrectly integrate the device The device produces inaccurate or unreliable outputs due to improper integration	User discomfort and dissatisfaction; misdiagnosis; delay in diagnosis/patient's follow up/treatment	Inadequate information provided by the manufacturer, lack of clear regulatory guidance or failure to meet labeling standards		👩‍⚕️HCP	User takes a photo of the patient's lesion.	PRS-0MCPRS-1XUPRS-3YHPRS-5LJ	HCP	4	3	12⚠️	C	IFU has been written according to the applicable regulations: MDR 2017/745 and ISO 15223-1	PRS-0MCPRS-1XUPRS-3YHPRS-5LJ	Technical director	IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information.	3	1	3✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001 Internal/external audits	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable
R-UK2	Inadequate warnings in the IFU	🤳 Usability🏛️ Regulatory	Lack of critical safety information required for the correct use of the device	Users rely on the labeling and warnings provided, which are incomplete or unclear HCPs may use the device in situations where it is not indicated The device may produce unreliable or misleading outputs due to inappropriate usage	Misdiagnosis; delay in diagnosis/patient's follow up/treatment	Inadequate information provided by the manufacturer, lack of clear regulatory guidance or failure to meet labeling standards		👩‍⚕️HCP	User takes a photo of the patient's lesion.	PRS-0MCPRS-1XUPRS-3YH	Patient	3	3	9⚠️	C	IFU has been written according to the applicable regulations: MDR 2017/745 and ISO 15223-1	PRS-0MCPRS-1XUPRS-3YH	Technical director	IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information.	3	1	3✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001 Internal/external audits	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable
R-GTY	Instructions for use are not available at the time of use due to downtime	🤳 Usability🏛️ Regulatory	User cannot consult the IFU	User wants to consult the IFU User cannot reach the IFU Improper use of the device	User discomfort and dissatisfaction; delays in diagnosis; treatment and follow up	IFU are only electronically available, connectivity issue, server issues		👨‍💻ITP	Input authorization details	PRS-1V6PRS-1XUPRS-4QWPRS-5LJ	Managing Organisation	3	2	6⚠️	A	If the issue is access to the internet, the use would also not be able to use the device, so there is no risk of using the device without access to the instructions. Furthermore, the IFU can be downloaded by PDF. Moreover, the IFU is hosted on a independent instance to improve the resiliency of the information system, this means that downtime in the device does not imply downtime in the IFU. The device sends messages to the user when there is any problem with the communication between the device and the user end, so the user always receives basic instructions when something is wrong. Furthermore, the procedure SP-001-001 - eIFU management explains the process to fulfil customer's request for paper IFU	PRS-1V6PRS-1XUPRS-4QWPRS-5LJPRS-9F2	Technical director	The selection of independent instances is performed according to GP-012 Design, redesign and development IFU are verified as registered at the TEST_011_We facilitate the integration of the device into the users' system. Process to provide customers with IFU in paper format is explained in SP-001-001 - eIFU management	2	1	2✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable
R-109	Electronic instructions for use are not compatible with different devices	🤳 Usability🏛️ Regulatory	Intended user cannot consult IFU	The manufacturer creates electronic instructions for use that are intended to guide users in operating the medical device The manufacturer fails to conduct thorough compatibility assessments of the eIFU across various devices and platforms, such as different operating systems, software versions, or hardware configurations Users attempt to access the eIFU on their devices and fails Users are unable to access the critical information needed to operate the device safely and effectively	User discomfort and dissatisfaction; delays in diagnosis; treatment and follow up	Electronic IFU are developed in a non-universal platform or technology.		👨‍💻ITP	Input authorization details	PRS-0MCPRS-1V6PRS-1XUPRS-5LJPRS-7Z8	Managing organisation	3	2	6⚠️	A	The electronic instructions for use are accessible via a web app that is accessible via any browser with any operating system. The instructions do not contain features, graphics or materials that are not universally accessible. It is also relevant to mention that the electronic access to the IFU is actually our recommended method of interacting with them, due to the intrinsic nature of the device Users can request IFU in paper format	PRS-0MCPRS-1V6PRS-1XUPRS-5LJPRS-7Z8	Technical director	The selection of independent instances is performed according to GP-012 Design, redesign and development IFU are verified as registered at the TEST_011_We facilitate the integration of the device into the users' system	2	1	2✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable
R-4Z5	Lack of version control or traceability	🤳 Usability🏛️ Regulatory	The ITP cannot identify the version of the device being used	The manufacturer develops a new version of the software that includes updates, bug fixes, and improved functionality for the medical device The manufacturer neglects to include the software version update in the labeling The device is distributed to healthcare facilities without the crucial software version information clearly stated in the labeling Users may experience compatibility issues if the device's software version is not aligned with the systems or applications it is meant to integrate with, but they are unaware of the specific version in use Users might fail to update or maintain the software properly because they do not have clear information about the software version, which may lead to security vulnerabilities or degraded performance	User discomfort and dissatisfaction; delays in diagnosis; treatment and follow up	Inadequate information provided by the manufacturer		👨‍💻ITP	Input authorization details	PRS-1V6PRS-1XUPRS-5LJPRS-9F2	Managing organisation	3	2	6⚠️	AC	We include within one of the requirements defined during the design stage that one of the outputs of the device must be the version being used and this information is included in the IFU	PRS-1V6PRS-1XUPRS-5LJPRS-9F2	Technical director	IFU are verified as registered at the TEST_011_We facilitate the integration of the device into the users' system	2	1	2✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable
R-BXD	Insufficient knowledge to display electronic IFU	🤳 Usability🏛️ Regulatory	Fail to properly display the instructions for use	The ITPs or HCPs responsible for the device fail to understand how to access or display the eIFU Despite not having access to the eIFU, they proceed with integration or use, assuming they can work around the issue The medical device is improperly integrated into the system or outputs are misinterpreted	User discomfort; dissatisfaction. Misdiagnosis; delays in diagnosis/proper treatment and worsening of the patient's health status.	Lack of information/requirement on how to access eIFU		👨‍💻ITP	Input authorization details	PRS-1V6PRS-1XU	Managing organisation	3	3	9⚠️	AC	IFU is designed in such a way that it is accessible via a dedicated and secure URL and it is also available in the website. The only requirement for accessing the eIFU is having internet connection. The users can access the IFU via any web browsers with any operations system. Upon user's request, we provide the user with IFU in paper format according to the internal procedure SP-001-001 eIFU management	PRS-1V6PRS-1XUPRS-9F2	Technical director	IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information. SP-001-001 eIFU management	3	1	3✅	✅Acceptable	R-TF-012-015 Summative evaluation report_2024_001	Not applicable (acceptable risk)	FALSE	TRUE	✅Acceptable

---

Table 4: Use-related Risk Analysis

Summary of Preliminary Analyses and Evaluations

The manufacturer conducted various preliminary analyses and evaluations during product development. These helped inform the user interface design so it could be optimized with regards to safety and effectiveness.

Description and Categorization of Critical and Non-critical Tasks

A critical task is a user task which, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user, noting that harm is defined to include compromised medical care. Serious harm/injury is defined by 21 CFR 803.3(w) to mean an injury or illness that (1) is life-threatening, (2) results in permanent impairment of a body function or permanent damage to a body structure, or (3) necessitates medical or surgical intervention to preclude permanent impairment of a body function or permanent damage to a body structure. Per the severity rating classification of the URRA in Table 3, harms associated with severities of 3 or above will be considered critical tasks. For this study, critical tasks that logically occur in sequence when using the device will be grouped into use scenarios. All tasks (critical and non-critical) that are part of the natural workflow of a use scenario will be evaluated. However, non-critical tasks that are not necessary for the completion of critical task-based use scenarios are out of scope for evaluation. Table 5 lists the critical and non-critical tasks associated with the use of the device, the severity of each task based on the potential harm described in Table 3 above, as well as the use scenarios used in this study to assess each task.

User Group	Phase	Use Task	Severity	Classification	Use Scenario Used to Evaluate
ITP	Get access token	Input authorization details	3	Critical	ITP Use Scenario 1
ITP	Get access token	Input correct URL for login endpoint	3	Critical	ITP Use Scenario 1
ITP	Get access token	Use a stable internet connection	3	Critical	ITP Use Scenario 1
ITP	Build JSON with data	Format the data	3	Critical	ITP Use Scenario 1
ITP	Build JSON with data	Enter mandatory fields	3	Critical	ITP Use Scenario 1
ITP		Enter data	3	Critical	ITP Use Scenario 1
ITP		Send images to the API	3	Critical	ITP Use Scenario 1
ITP		Send required data to the API	3	Critical	ITP Use Scenario 1
ITP	Send JSON to device	Send request to the endpoint	3	Critical	ITP Use Scenario 1
ITP	Send JSON to device	Input correct HTTP method	3	Critical	ITP Use Scenario 1
ITP	Send JSON to device	Send access token with the request	3	Critical	ITP Use Scenario 1
ITP	Receive JSON from device	Process the request	3	Critical	ITP Use Scenario 1
ITP	Process and store JSON	Extract the data	3	Critical	ITP Use Scenario 1
ITP	Build report	Generate report	3	Critical	ITP Use Scenario 1
HCP	Take pictures	User takes a photo of the patient's lesion	3	Critical	HCP Use Scenario 1
HCP	Authenticate in the system	User logs into the system	3	Critical	HCP Use Scenario 1; HCP Use Scenario 2
HCP	Upload pictures	User uploads photos to the client	3	Critical	HCP Use Scenario 1; HCP Use Scenario 2
HCP	Read and interpret the report	User understands the format of the device's output	3	Critical	HCP Use Scenario 3
HCP	Read and interpret the report	User correctly reports the probability of certain conditions based on the ICD categories	3	Critical	HCP Use Scenario 3
HCP	Read and interpret the report	User correctly interprets the report	3	Critical	HCP Use Scenario 3
HCP	Read and interpret the report	The user understands that the report is not a certain diagnosis	3	Critical	HCP Use Scenario 3

Table 5: Critical and Non-critical Tasks

Details of Human Factors Validation Testing

Study Objectives

The goal of this study is to assess whether the device and its user interface, including its instructions for use, can be used safely and effectively by its intended users, for its intended uses, and in its intended use environments. Intended users will simulate a typical application case with the device based on its intended use.

Not finished

To achieve this goal, this validation usability study will be conducted on TBD, at TBD and consists of the simulated use of the device by representative users. The study will consist of a 60-minute one-on-one (moderator and participant) usability testing session, during which the study moderator will give participants, representing potential users, a series of tasks to perform and questions to answer with the device. Throughout the sessions, the study moderator will observe and record participants' objective performance and subjective feedback and ask follow-up questions regarding use of the device. The study will evaluate all observed use problems (i.e., use errors [UEs], close calls [CCs], and use difficulties [UDs]) as well as their associated root causes, see sections 8.6 Description of Data Collection and Methods and 8.7.1 Description of Root Cause Analysis for a summary of the data collection and evaluation method.

The study results will then be used to determine if the residual risk associated with product use is acceptable. While it is rarely feasible to eliminate all risk, it should be reduced to an acceptable level, where feasible and practical, such that the benefits of product use outweigh any residual risks.

Type of Testing

This HF study will involve simulated-use testing to explore how users may interact with the product in a natural and independent manner (to the extent possible). Representative users will use the product in realistic scenarios under simulated conditions.

This type of testing is sufficient to assess the adequacy of the user interface, as relevant aspects of actual product use can be simulated, including the conditions of use, product characteristics, and environments of use.

Study Test Environment

Realism

The study test environment will simulate the intended use environment for each user group, as follows:

• Clinical Environment: Healthcare providers will use the study device in a simulated clinical environment (i.e., a hospital room or doctor's office), consisting of a quiet, well-lit room inside a hospital where HCPs can perform diagnostic assessments. No effort will be made to mitigate natural distractions, such as a phone ringing, that may occur. In this environment, HCPs will take photos of a patient's skin lesions, and submit them to a computer integrated with the device software to produce a report.
• Clinical IT Office Environment: IT professionals will use the study device in a simulated clinical office environment (i.e., a hospital IT desk or clinic's office), consisting of a quiet, well-lit room inside a hospital where ITPs execute their work. No effort will be made to mitigate natural distractions, such as a phone ringing, that may occur. In this environment, ITPs will be working on a computer to integrate the device API into their hospital's IT system.

Study Locations

Not finished

This study will be conducted at Design Science's facilities in Philadelphia, PA on TBD and its facilities in Evanston, IL on TBD.

Study Devices and Materials

The devices used in this study will be production-equivalent, including its electronic IFU. As described in the section "Training", the manufacturer does not expect that users will receive training prior to using the device.

Study Devices and Materials - HCPs

To interact with the API, HCPs will be working with a user interface representative of what they may see during actual use following the integration of the API in their hospital system that they will access via a URL on a computer.

While each hospital system may display a different final user interface once the API is integrated, they will all incorporate the same functional elements that are evaluated in this study, namely the portal to upload images, and the subsequent diagnostic report data, i.e., detected conditions, preliminary analysis (malignancy, sensitivity, specificity), and other related data. While stylistic differences exist between various hospital systems, e.g., color schemes, fonts, arrangement of elements, they will all represent the same functions.

As this software-only medical device API only consists of the functions, the GUI design is outside of the scope of this validation study, and only the functional layer of the interface, i.e., the API interaction, will be assessed. Therefore, testing the API with a generic version of the user interface will be sufficient to evaluate the API. This generic interface is shown in Table 6.

User Interface Item	Written Description	Image
Software	Initial screen where HCPs will upload the photo of the lesion	img]:w-1/2>
Diagnostic report	Details of the diagnostic report following submitting the photo of lesion	img]:w-1/2>

Table 6: HCP User Interface for Usability Testing

Based on section "Description of Device User Interface", the following materials will be present in the study environment to simulate items that HCPs may use during their diagnostic assessment of a patient:

• Smartphone with the capability of taking high quality photos supplied by the manufacturer for usability testing
• Computer with the device software integrated into a representative EHR/EMR system
• Gloves
• High quality digital images of skin lesions supplied by the client for participants to upload to the device

Study Devices and Materials - ITPs

ITPs will be utilizing a computer with a representative EHR/EMR they would use in their hospital, a URL of the API, an Integrated Development Environment (IDE) installed, and the electronic IFU. They will conduct their typical practice to integrate the API into the hospital system with the materials provided. ITPs use IDEs to write, edit, debug, and run code. As such, an ITP may use programs such as Visual Code Studio to integrate the API. Therefore, this program is sufficient for usability testing. Figure 1 displays the user interface for Visual Code Studio, a representative IDE that ITPs would use in their typical practice.

Figure 1: Visual Code Studio IDE Interface for ITPs

Based on section 3 Description of Device User Interface, the following materials will be present in the study environment to simulate items that ITPs may use to complete their work:

• Computer with a representative EHR/EMR system and preferred IDE installed

Participants

Number, Type, and Rationale

The study will include 30+6 participants (i.e., 15 IT professionals, and 15 healthcare providers) who are representative of the intended users. There will be an additional 3 participants recruited for each user group to account for potential no-shows, disqualifications, and cancellations; however, testing will end after the fifteenth participant for each user group. Refer to Attachment B-Participant Screener for full inclusion/exclusion criteria.

Table 7 describes the user groups intended user information.

User Group	# of Participants	Training Status	# of Sessions	Characteristics
ITPs	n = 15 + 3	Untrained	1	- Ages 18 years or older - Have experience with FHIR - Have experience integrating services and API into a hospital IT system - At least 1 year of experience working for a hospital IT department
HCPs	n = 15 + 3	Untrained	1	- Ages 18 years or older - Licensed physician (MDs, DOs) - Varying degrees of training in clinical diagnosis and management of skin lesions - At least 1 year of experience as an HCP

Table 7: User Group Details

Inclusion and Exclusion Criteria

In addition to the defined criteria, to reduce bias and ensure representative users have been selected as participants, all participants must

• not be affiliated with market research, product development, pharmaceutical companies, or medical device companies,
• not have completed any medical market research related to software-aided adjunctive diagnostic devices within the last 3 months,
• bring photo identification to the study,
• have the capacity to understand and sign an informed consent form (ICF),
• be able and willing to participate in a 60-minute session and complete study assessments,
• be willing to be video recorded during the study,
• be able to speak, read, and comprehend English, and
• be a US resident.

Participant Protection

Identity Protection

The study moderator will assign a unique identification code (i.e., participant ID) to each participant as a means of referencing participants, so that the manufacturer is not able to associate participant names or any other unique personal identifiers with the study data.

The letter component of the participant ID corresponds to the user groups as presented in Table 8. The numerical component of the participant ID (ranging from 01-15) refers to the order in which each participant from each user group participated in the study.

Participant ID	User Group
I##	IT Professionals
H##	Healthcare Provider

Table 8: Participant IDs

Institutional Review Board Oversight

This human factors study will not be submitted to an Institutional Review Board (IRB) due to the following reasons:

• Risk to the participants is minimal. Testing is non-invasive and structured in a simulated-use environment.
• There will be no needles, treatment, or drug product present. Participants will be interacting with a software-only device.
• Participants will review and sign an informed consent form (ICF) and business confidentiality agreement providing consent to participate in the evaluation.

Tasks and Scenarios

Session Overview

The structure of each 60-minute testing session is detailed in Table 9.

Session Phase	Description
Informed Consent Review	The study moderator will review the ICF with the participant, ensure that they understand the form, agree to participate, and agree to be recorded.
Background Questions	The study moderator will ask the participant a series of background questions to confirm that they meet the study inclusion criteria and collect relevant data.
Overview	An overall explanation of the testing procedure will be given.
ITP Use Scenario 1: Simulated Use Post Scenario Interview (including Root Cause Interview)	Each of the use scenarios will be administered, followed by an interview.
Conclusion	Participants will be given a chance to express any additional questions or comments.

User Group: ITP

Session Phase	Description
Informed Consent Review	The study moderator will review the ICF with the participant, ensure that they understand the form, agree to participate, and agree to be recorded.
Background Questions	The study moderator will ask the participant a series of background questions to confirm that they meet the study inclusion criteria and collect relevant data.
Overview	An overall explanation of the testing procedure will be given.
HCP Use Scenario 1: Simulated Use: No Lesion Post Scenario Interview (including Root Cause Interview)	Each of the use scenarios will be administered, followed by an interview.
HCP Use Scenario 2: Simulated Use: Lesion Post Scenario Interview (including Root Cause Interview)	Each of the use scenarios will be administered, followed by an interview.
HCP Use Scenario 3: Knowledge Assessment Post Scenario Interview (including Root Cause Interview)	Each of the use scenarios will be administered, followed by an interview.
Conclusion	Participants will be given a chance to express any additional questions or comments.

User Group: HCP

Table 9: Session Overview

Use Scenarios and Tasks

All participants will complete all use scenarios described in Table 10 and Table 11 in that order. Prior to the completion of the use scenarios, the study moderator will introduce participants to the study environment and give a brief introduction to each scenario. Critical tasks must be completed successfully to demonstrate the safety and effectiveness of the device.

During the first scenario, participants will be asked to imagine that they have just received the device and must use it for the very first time, just as they would in a real-life situation.

ITPs will have the electronic IFU containing installation instructions for the device software available to use if they chose so. Also, a computer will be set up with a typical representation of hospital IT systems. This would include an installed EHR/EMR program and a representative IDE. They will be provided with a URL with the device API and will be tasked with integrating it with the hospital IT system on the computer. In order to assess the API, the study moderator will observe participants' behaviors and actions and note any use problems that occur throughout the integration process (e.g., difficulty logging in, managing tokens, understanding error messages etc.). The study computer will enable a screensharing feature that will allow the study team to observe the participant's progress. Furthermore, as participants interact with the API during simulated use, a log that is visible to the manufacturer is automatically generated that tracks their progress to confirm the completion of the assigned tasks. A representative from the manufacturer's team will be monitoring the log during the sessions to assist in identifying and characterizing any observed use problems.

Phase	Task	Classification	Success Criteria
Build JSON with data	Enter data	Critical	User does not enter data in incorrect types (e.g., string instead of integer)
Build JSON with data	Enter mandatory fields	Critical	User enters mandatory fields required for the request
Build JSON with data	Format the data	Critical	User formats the data correctly, does not include the wrong JSON structure or incorrect field names
Build report	Generate report	Critical	User understands the data format, and the data are displayed consistently.
Build report	Send images to the API	Critical	User codes for images to be sent in a supported format, or correctly encoded in Base64
Build report	Send required data to the API	Critical	User does not include extraneous data not required for the request
Get access token	Input authorization details	Critical	User inputs correct username, password, or other required authentication details
Get access token	Input correct URL for login endpoint	Critical	User enters the correct URL for the login endpoint
Get access token	Use a stable internet connection	Critical	User uses a stable internet connection
Process and store JSON	Extract the data	Critical	User extracts the data with the JSON decoder. The extracted data are compatible with existing systems and do not overwrite existing data.
Receive JSON from device	Process the request	Critical	The user completes the response without corrupting it, misinterpreting the HTTP status, deleting or modifying it, or incorrectly mapping the response data to the clinical workflow or patient records.
Send JSON to device	Input correct HTTP method	Critical	User uses the correct HTTP method
Send JSON to device	Send access token with the request	Critical	User includes the access token in the header. The access token is not expired, and the request is only sent once
Send JSON to device	Send request to the endpoint	Critical	User sends the request to the correct endpoint

ITP Use Scenario 1: Simulated Use

Table 10: Use Scenarios and Descriptions for ITPs

HCPs will be tasked with using the API through a representative GUI to assist in diagnosing a patient that has come in to assess a skin lesion. To simulate taking photos, the HCP will be asked to take a photo of the hand of a member of the research team such as the study moderator or a second supporting analyst. The HCP will then submit it to the device that has been installed on a computer, running the report, and interpreting the results to see if they are consistent with the lesion. The HCP will be tasked with following this workflow to assess skin that does not bear a malignant lesion using the device only one time. The study moderator will then observe participants' behaviors and actions and note any use problems that occur during the usage of the software.

Following this simulated-use scenario (Scenario 1), a post scenario interview will be performed to better evaluate safe and effective use. To this end, participants will be asked how they felt the scenario went as well as whether they made any mistakes, almost made any mistakes, or had any difficulties. After this self-reflection, the moderator will probe for better understanding and delve into a root cause interview, where open-ended questions will be asked until the underlying cause of each use problem can be identified. This will include participant-reported and moderator-observed use problems from Scenario 1. For some tasks, successful completion may or may not be observable during the simulated-use scenario due to the nature of the task. For example, a task may require the participant to correctly interpret the data produced by the product (e.g., read the list report output). If task performance is not observable, the moderator may follow up in the root cause interview and ask open-ended, naturally worded follow up questions about the task to accurately assess the participant's task performance.

Following this, the HCPs will be asked to complete their workflow once again and will be directed to use the device to produce a report for a malignant skin lesion (Scenario 2). A digital file containing photos of a skin lesion will be provided to HCPs. The HCP will then upload these photos to the device that has been installed on a computer, running the report, and interpreting the results to see if they are consistent with the lesion. The HCP will be tasked with following this workflow to assess a malignant lesion using the device only one time. The study moderator will then observe participants' behaviors and actions and note any use problems that occur during the usage of the software.

For the HCP workflow, some important aspects of use, such as interpreting and understanding the report, cannot be assessed through task performance and instead are assessed through direct questioning of the participant to ascertain their understanding of information. This knowledge is tested by questioning the test participants using open-ended and naturally worded questions (Scenario 3). The knowledge assessment will be administered to determine their understanding of the information in the report output. The malignant skin lesions participants may assess are shown in Figure 2.

Figure 2: Malignant Skin Lesions (Squamous Cell Carcinoma, Melanoma, & Basal Cell Carcinoma)

In some instances, asking a directed question can bias the participant to provide the correct answer. In these cases, less specific wording will be used for the initial question to not influence the participant. Only if the participant's knowledge, or lack thereof, cannot be sufficiently assessed through their response to the initial question, optional and more focused follow up questions will be asked. These optional, open-ended, naturally worded follow-up questions will be asked in a tiered approach.

Participants will be free to refer to any materials when answering but will not be specifically guided to any aspect of the product interface (e.g., the Electronic IFU). The study moderator will note participants' answers and record any use problems. There will then be another post scenario interview, beginning with participants being asked how they felt answering the questions and whether there was anything they identified as unclear or confusing when answering. This will again be followed by any clarifying questions and a root cause interview on all remaining use problems to evaluate safe and effective use.

Use Scenario	Phase	Task	Classification	Success Criteria
HCP Use Scenario 1: Simulated Use: No Lesion	Authenticate in the system	User logs into the system.	Critical	The user enters the correct username and password and logs in successfully
HCP Use Scenario 1: Simulated Use: No Lesion	Authenticate in the system	User logs into the system.	Critical	The user enters the correct username and password and logs in successfully
HCP Use Scenario 1: Simulated Use: No Lesion	Take pictures	User takes a photo of the patient's lesion.	Critical	The photo has good quality, captures the relevant skin structure, is well lit, focused, with adequate distance
HCP Use Scenario 1: Simulated Use: No Lesion	Upload pictures	User uploads photos to the Legit.Health plus client.	Critical	The user uploads a high-quality image
HCP Use Scenario 2: Simulated Use: Lesion	Authenticate in the system	User logs into the system.	Critical	The user enters the correct username and password and logs in successfully
HCP Use Scenario 2: Simulated Use: Lesion	Upload pictures	User uploads photos to the Legit.Health plus client.	Critical	The user uploads a high-quality image
HCP Use Scenario 2: Simulated Use: Lesion	Upload pictures	User uploads photos to the Legit.Health plus client.	Critical	The user uploads a high-quality image
HCP Use Scenario 3: Knowledge Assessment	Read and interpret the report	The user understands that the report is not a certain diagnosis.	Critical	The user acknowledges that the output report is not a standalone diagnosis.
HCP Use Scenario 3: Knowledge Assessment	Read and interpret the report	User correctly interprets the report output.	Critical	The user correctly identifies the quantification of disease intensity, extent, or count of clinical signs.
HCP Use Scenario 3: Knowledge Assessment	Read and interpret the report	User correctly reports the probability of certain conditions based on the ICD categories.	Critical	The user identifies the probability of conditions listed in the report
HCP Use Scenario 3: Knowledge Assessment	Read and interpret the report	User understands the format of the device's output	Critical	The user identifies the information in the report

HCP Use Scenarios 1, 2 and 3

Table 11: Use Scenarios and Descriptions for HCPs

Description of Data Collection and Methods

The data to be collected are as follows:

• Observational Data: Participants will be given an opportunity to use the devices independently and in as realistic a manner as possible, without guidance, coaching, praise, or critique from the study moderator.
• Interview Data: The study moderator will ask participants open-ended, neutrally-worded questions to understand any use problems participants may have with the study devices and tasks. Particular care will be taken not to blame participants or bias them in later tasks.
  • Subjective assessment of use: This data consists of participants' subjective feedback on the root cause of potentially safety-related use errors and use difficulties, and opinions on all tasks for any residual risk, as well as the adequacy of instructions.
• Knowledge Task Data: Participants will be given an opportunity to explain their understanding of product use independently. This type of data is necessary as some critical tasks cannot be directly observed through simulated use; these critical tasks may involve users' understanding of information (e.g., warnings).

The following coding system will be used for simulated-use and knowledge task data collection and analysis:

• Success (OK): This coding describes a situation in which the participant correctly performs a task, or answers a knowledge task question, according to the steps in the User Guide or an acceptable alternative.
• Use error (UE): This coding describes a situation in which the participant performs a task incorrectly (i.e., not in accordance with the intended manner of use), does not complete a necessary task, or answers a knowledge task question incorrectly or incompletely.
• Close call (CC): This coding describes a situation in which the participant performs or nearly performs a task incorrectly (i.e., not in accordance with the intended manner of use), or answers a knowledge task question incorrectly or incompletely, but then resolves the issue prior to any harm being done without assistance.
• Use difficulty (UD): This coding describes a situation in which the participant struggles to some extent while completing a task or answering a knowledge task question (e.g., confusion, taking longer than expected, difficulty manipulating a device's components), but eventually can completes the task or answers the question successfully without assistance.

Recording Equipment

Audio and video will be recorded for all study sessions using synchronized, multi-channel, HD-resolution video.

Recording Software

Study data will be recorded by the study team into OpenClinica. OpenClinica is a web-based platform that allows the study moderator to directly record their scores, observations, and notes into an electronic version of the study protocol. All data entered into OpenClinica and any subsequent changes made by the study team are visible via an audit log. Any changes made to the data after it is entered requires a rationale for the change to be documented.

OpenClinica is compliant with the FDA's Title 21 CFR Part 11.

Data Analysis

Observational data during simulated-use and knowledge task data will be collected and any use problems will be recorded as moderator observations. Based on this, as previously described, an interview will be conducted to better discern any use problems (i.e., UEs, CCs, and UDs), the root cause of such problems, and the associated residual risk. Such interview data will be recorded as participant comments.

Description of Root Cause Analysis

Root cause analysis (RCA) will involve examining the data to determine the underlying cause of each use problem, with a focus on investigating contributing factors to each problem and determining ways to prevent them in the future. Moderator observations and participant comments will be analyzed holistically during this process to evaluate the safety and effectiveness of product use.

Description of Residual Risk Analysis

All risks that remain after human factors validation testing will be analyzed to determine whether they can be reduced or eliminated. This analysis will determine whether design modifications are needed, would be possible and might be effective at reducing the associated risks to acceptable levels. This analysis will include a risk-based discussion of the observed use events and their root causes as well as the effectiveness of current risk mitigations.

If user interface design modifications are required to further mitigate risk, the residual risk analysis will include a description of the recommended design changes and a discussion regarding whether additional HF validation testing is required.

The residual risk analysis will be used to evaluate the safety and effectiveness of product use and determine if the residual risk associated with product use is acceptable. While it is rarely feasible to eliminate all risks, it should be reduced to an acceptable level, where feasible and practical, such that the benefits of product use outweigh any residual risks.

Appendix A: Interview Guide

Thanks for coming in today. In a few moments, we'll start the session, but before we begin, I want to review a few things with you. You have already signed the informed consent. Do you have any additional questions about the document? I would also like to make sure you:

• Promise not to tell anyone about what you see today.
• Give us permission to audio and video record this session, so long as none of the footage is ever made public.
• Agree that the study's Sponsor can make use of any information you provide during the study without any restrictions.

[Answer any questions and then proceed.]

Ok, I'm going to turn on the video and audio recording now.

Testing Session

Today, I am going to ask you to go through a series of use scenarios with a software-only device that analyzes images of visible skin structure abnormalities to support the assessment of all diseases of the skin. I want to get your feedback on the device and will also ask you some questions regarding the use of the device.

For the purpose of our study, we are simulating a real use scenario, so please imagine that you are in a real situation and perform all tasks as if you were at work.

If for some reason you feel that you are unable to complete a task the way you would at work due to this simulated environment, please let me know.

At times, you may have questions for me that I may or may not be able to answer. If I am unable to answer, it is because I want to observe how you would handle a real-life situation when problems arise.

Please remember that you are here to help us evaluate the product. We are not evaluating you or your abilities in any way. You are the expert here to help us improve the product today.

Finally, please keep in mind, I did not design anything you're seeing today, so you do not have to worry about my feelings regardless of whether your feedback is positive or negative. We want your honest opinions, good or bad. So, if you see any problems with the devices I will be showing you that you think I should know about, please mention them.

Do you have any questions before we start?

ITP Use Scenario 1: Simulated Use

Moderator to prepare computer with hospital EHR system installed, and link to electronic IFU.

For our first task, I would like you to imagine that your hospital is looking to incorporate the device into their system. I would like you to complete all the steps necessary to install and set up the device for use. You can refer to any of the materials provided at any time.

When you're ready, please begin and take your time.

Observation form

To see the observation form, please refer to T-TF-025-005 Summative evaluation observation form

Post-Scenario Interview

How did that go?

Did you make any mistakes or almost make any mistakes during that scenario?

Did you have any difficulty at any point during that scenario?

[Follow up on participant-observed use problems and root-causes.]

[Follow up on study moderator-observed use problems and root-causes.]

HCP Use Scenario 1: Simulated Use: No Lesion

Moderator to prepare smartphone, computer with hospital EHR/EMR system installed, the device installed, and manikin with simulated skin lesions

For our first task, I would like you to imagine that an IT professional has implemented the device into your hospital EHR system. Your patient has visited your clinic because of concerns over the skin of the back of their hand (Moderator to direct participant to a researcher to take a photo of their hand). I would like you to complete all the steps necessary to assess the area by generating a report using the device. You can refer to any of the materials provided at any time.

When you're ready, please begin and take your time.

Observation form

To see the observation form, please refer to T-TF-025-005 Summative evaluation observation form

Post-Scenario Interview

How did that go?

Did you make any mistakes or almost make any mistakes during that scenario?

Did you have any difficulty at any point during that scenario?

[Follow up on participant-observed use problems and root-causes.]

[Follow up on study moderator-observed use problems and root-causes after Scenario 3.]

HCP Use Scenario 2: Simulated Use: Lesion

[Moderator to prepare smartphone, computer with hospital EHR/EMR system installed, the device installed, and manikin with simulated skin lesions]

For our next task, I would like you to imagine that your patient has visited your clinic again because of concerns over a skin lesion that has appeared on their body. Imagine that these are the images you have taken of the lesion (Moderator to provide participant with images of the skin lesions provided by the sponsor) I would like you to complete all the steps necessary to assess the lesion by generating a report using the device. You can refer to any of the materials provided at any time.

When you're ready, please begin and take your time.

Observation form

To see the observation form, please refer to T-TF-025-005 Summative evaluation observation form

Post-Scenario Interview

How did that go?

Did you make any mistakes or almost make any mistakes during that scenario?

Did you have any difficulty at any point during that scenario?

[Follow up on participant-observed use problems and root-causes.]

[Follow up on study moderator-observed use problems and root-causes after Scenario 3.]

HCP Use Scenario 3: Knowledge Assessment

Now I'm going to ask you a few questions about the device user interface. Also, this is not a test of any kind; you may refer to any of the materials provided at any point to help you answer the questions.

Questionnaire

To see the questionnaire, please refer to T-TF-025-006 Summative evaluation questionnaire

Post-Scenario Interview

How was answering those questions?

Did you notice anything related to the product and its associated materials that was unclear or confusing when answering those questions?

Follow up on all (study moderator-observed and participant-observed) use problems and root-causes up to this point (that is, from Scenario 3).

Thank and Exit

Do you have any additional questions or comments about anything that you have seen or done here today?

That is the end of our session. Thank you for your time and feedback. [Stop recording and walk participant out.]

Were there any protocol deviations?

If so, record:

• Yes
• No

Appendix B: Electronic IFU

Electronic IFU is available at the following URL: https://apidocs-draft.legit.health.

Appendix C: List of participants recruited for summative evaluation

ID	Nationality	User group	Sex	Age	Profession	Years of experience
I01	US	ITP	M	32	IT Professional	5
I02	US	ITP		F	29	3
I03	US	ITP	M	45	IT Professional	10