R-TF-025-004 Summative evaluation protocol

Table of contents

• List of Tables
• List of Figures
• Terminology and Definitions
• Applicable Standards and Guidance
• Internal References
• Introduction
• Description of Intended Device Users, Uses, Use Environments, and Training
  • Intended Device Users
  • Intended Uses
  • Intended Use Environments
  • Training
• Description of Device User Interface
  • Device User Interface
  • Summary of Operating Sequence
• Summary of Known Use Problems
• Analysis of Hazards Associated with Use of the Device
• Summary of Preliminary Analyses and Evaluations
• Description and Categorization of Critical and Non-critical Tasks
• Details of Human Factors Validation Testing
  • Study Objectives
  • Type of Testing
  • Study Test Environment
    • Realism
    • Study Locations
    • Study Devices and Materials
  • Participants
    • Number, Type, and Rationale
    • Participant Protection
  • Tasks and Scenarios
    • Session Overview
    • Use Scenarios and Tasks
  • Description of Data Collection and Methods
    • Recording Equipment
    • Recording Software
  • Data Analysis
    • Description of Root Cause Analysis
    • Description of Residual Risk Analysis
• Appendix A: Interview Guide
  • Testing Session
  • ITP Use Scenario 1: Simulated Use
    • Post-Scenario Interview
  • HCP Use Scenario 1: Simulated Use: No Lesion
    • Post-Scenario Interview
  • HCP Use Scenario 2: Simulated Use: Lesion
    • Post-Scenario Interview
  • HCP Use Scenario 3: Knowledge Assessment:
    • Post-Scenario Interview
    • Thank and Exit
• Appendix B: Electronic IFU

List of Tables

• Table 1: Product Interface
• Table 2: Summary of Known Use Problems
• Table 3: Severity Ratings in URRA
• Table 4: Use-related Risk Analysis
• Table 5: Critical and Non-critical Tasks
• Table 6: HCP User Interface for Usability Testing
• Table 7: User Group Details
• Table 8: Participant IDs
• Table 9: Session Overview
• Table 10: Use Scenarios and Descriptions for ITPs
• Table 11: Use Scenarios and Descriptions for HCPs

List of Figures

• Figure 1: Visual Code Studio IDE Interface for ITPs
• Figure 2: Malignant Skin Lesions (Squamous Cell Carcinoma, Melanoma, & Basal Cell Carcinoma)

Terminology and Definitions

• AI: Artificial Intelligence
• API: Application Programming Interface
• CADe: Computer-assisted detection
• CC: Close call
• CFR: Code of Federal Regulations
• DO: Doctor of Osteopathic Medicine
• e.g.,: For example
• EHR: Electronic Health Record
• EMR: Electronic Medical Record
• FDA: Food and Drug Administration
• FHIR: Fast Healthcare Interoperability Resources
• HCP: Healthcare provider
• HD: High definition
• HF: Human factors
• i.e.,: That is
• ICD: International Classification of Diseases
• ICF: Informed consent form
• IDE: Integrated Development Environment
• IFU: Instructions for Use
• IL: Illinois
• IRB: Institutional Review Board
• ITP: Information Technology Professionals
• MD: Doctor of Medicine
• ML: Machine Learning
• OK: Success
• PA: Pennsylvania
• PID: Participant identification code
• RCA: Root cause analysis
• UD: Use difficulty
• UE: Use error
• uFMEA: Use failure mode and effects analysis
• URRA: Use-related risk analysis
• US: United States

Applicable Standards and Guidance

This protocol is written in accordance with the following standards:

• ANSI/AAMI HE75:2009/(R)2018 Human Factors Engineering-Design of Medical Devices
• ANSI/AAMI/IEC 62366-1:2015+AMD1:2020 Medical Devices Part 1: Application of Usability Engineering to Medical Devices
• AAMI/IEC TIR62366-2:2016 Medical Devices Part 2: Guidance on the Application of Usability Engineering to Medical Devices
• ANSI/AAMI/ISO 14971:2019 Medical Devices—Application of risk management to medical devices
• FDA Final Guidance for Industry and FDA Staff: Applying Human Factors and Usability Engineering to Medical Devices (February 3, 2016)

In addition to the standards and guidance documents listed, draft guidance documents were reviewed and considered to understand the current thinking of the FDA on relevant topics.

Internal References

• R-TF-012-014
• REQ*001 The user receives quantifiable data on the intensity of clinical signs * QMS
• REQ*002 The user receives quantifiable data on the count of clinical signs * QMS
• REQ*003 The user receives quantifiable data on the extent of clinical signs * QMS
• REQ*004 The user receives an interpretative distribution representation of possible ICD categories represented in the pixels of the image * QMS
• REQ*005 The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner * QMS
• REQ*006 The data that users send and receive follows the FHIR healthcare interoperability standard * QMS
• REQ*007 If something does not work, the API returns meaningful information about the error * QMS
• REQ*008 Notify the user if the image does not represent a skin structure * QMS
• REQ*009 Notify the user if the quality of the image is insufficient * QMS
• REQ*010 The device detects if the image is of clinical or dermatoscopic modality * QMS
• REQ*012 Users can easily integrate the device into their system * QMS

Introduction

AI Labs Group (hereafter, the Manufacturer) has developed a new software-aided adjunctive diagnostic application programming interface (API) intended to assess clinical atypical cutaneous lesions that are suspicious for skin cancer or other skin conditions. This software-only medical device is intended for use by healthcare information technology professionals (ITPs) to integrate into their hospital systems' electronic medical record (EMR) system, and healthcare practitioners (HCPs) with varying degrees of training in clinical diagnosis and management of skin lesions. Throughout this document, this API will be referred to as the Manufacturer.

The device is a prescription device that incorporates Artificial Intelligence/Machine Learning (AI/ML) technology, including computer-assisted detection (CADe), which analyzes images or other physical characteristics of a skin lesion. The software provides quantifiable data on the intensity, count, and extent of clinical signs, and an interpretative distribution representation of possible International Classification of Diseases (ICD) classes to aid in determining whether a patient should be referred to a dermatologist.

Design Science will conduct a human factors (HF) validation study to determine if the device and its user interface can be used safely and effectively by all its intended users, for its intended uses, and in its intended use environments. The protocol that follows documents the methodology that will be used.

This validation study will take place in Philadelphia, PA and Evanston, IL from TBD. The study will include simulated-use testing with ITP and HCP participants, who are representative users, to evaluate their use of the product in representative clinical environments.

Description of Intended Device Users, Uses, Use Environments, and Training

Intended Device Users

The intended user population consists of the following distinct user groups:

• IT Professionals (ITPs): IT professionals are responsible for the integration of the medical device into the healthcare organization's EMR system. It is advisable that they have a basic knowledge of Fast Healthcare Interoperability Resources (FHIR) and the output of the device. They are individuals aged 18 years and older and can be from various educational backgrounds. Some of them may have vision, hearing, or dexterity impairments.
• Healthcare Providers (HCPs): Medical professionals (e.g., physicians), who have varying degrees of training in the clinical diagnosis and management of skin lesions, and will utilize the output from the software integration for diagnosis. All of them will have the qualifications and competencies native to their profession, and knowledge on how to take images with smartphones.

Intended Uses

The device is a computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organizations to gather data and improve their workflows. The data generated are intended to aid healthcare practitioners and organizations in their clinical decision-making process. It is not meant to confirm a clinical diagnosis nor replace the role of a dermatologist, but rather to obtain additional information to consider a decision.

The device is indicated for use on images of visible skin structure abnormalities to support the assessment of all diseases of the skin incorporating conditions affecting the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral, and genital), the dermis, the cutaneous vasculature, and the subcutaneous tissue (subcutis).

Intended Use Environments

The device is intended to be used in the setting of healthcare organizations and their IT departments, which commonly are situated inside hospitals or other clinical facilities. The device is intended to be integrated into the healthcare organization's system by IT professionals.

The environments are further described as follows:

• IT Office Environment: In the context of healthcare facilities, an IT office is typically designated for hospital IT staff to within the hospital or clinic. The setting is expected to have standard, controlled indoor room temperature, humidity, noise, and lighting conditions. However, during actual use, the intended use environment might vary in ways that impact users' abilities to use the product safely and effectively. ITPs may experience natural distractions from colleagues.
• Clinical Environment: A standard clinical environment is typically designed with a clear layout that has designated areas for patients to be assessed for cutaneous lesions where HCPs can take photos of patients and utilize the device. This setting is also expected to adhere to cleanliness and sanitation standards to prevent infections and include a sufficient inventory of medical supplies. The setting is expected to have standard, controlled indoor room temperature, humidity, noise, and lighting conditions. However, during actual use, the intended use environment might vary in ways that impact users' abilities to use the product safely and effectively. HCPs may experience natural distractions from colleagues and patients. Additionally, the use of gloves can affect HCP's interactions and manual dexterity.

Training

The manufacturer does not expect that users will receive training prior to using the device.

Description of Device User Interface

Device User Interface

This software-only medical device consists of a software-aided adjunctive diagnostic application programming interface (API). As such, the user interface of the device software-only medical device consists of the API endpoints, documentation, and data structures, specifically, the electronic instruction for use (IFU) for both user groups as well as API endpoints. These endpoints are used by ITPs for integrating into the hospital IT system. HPCs will then use the user interface of the hospital IT system to interact with the device software-only medical device.

The components of the device API user interface are detailed in Table 1.

Interface Item	Written Description
API Endpoints	URL structures, methods and request structure, response structure, and authentication
API Documentation	Electronic IFUs
Data Structure	JSON payload formats, including field names and FHIR nomenclature

Table 1: Product Interface

Summary of Operating Sequence

The device installation sequence to be conducted by ITPs consists of the following:

1. Obtain credentials
2. Gain access token
3. Build JSON with data
4. Send JSON to device
5. Receive JSON with device
6. Process and store JSON
7. Build Report

Refer to the User Guide for a detailed description of the primary operating sequence expected for the device.

For operation by healthcare providers, the device analyzes images taken of lesions and other skin abnormalities and produces a list of potential conditions.

The primary operating sequence consists of the following:

1. Take a picture of the lesion with a smart phone
   1. Image should be close to the lesion, focused, and well lit
   2. The lesion should be the main item in the photo
2. Upload the image to the device client
3. Run the analysis
4. Review the data output

Refer to the User Guide for a detailed description of the primary operating sequence expected for the device.

Summary of Known Use Problems

The manufacturer has conducted research on known or expected use problems for similar products and product types.

Based on this research, Table 2 describes known use problems pertaining to software-aided adjunctive diagnostic APIs, along with how these are addressed. All of these, along with additional consideration for industry-wide issues, have been identified and documented in the device's risk assessment. These problems are also included in the evaluated during the validation study through observations.

Type of Use Problem	Description	Applicability to Product	Design Mitigation
User authentication and security	APIs in healthcare settings must adhere to strict security standards. Past experiences have highlighted issues related to complex authentication processes and maintaining compliance with security protocols	Applicable	Implementation of data encryption, robust authentication mechanisms such as OAuth or JWT (REQ_005), monitoring of security threats, cybersecurity info in IFU (REQ_012)
Error handling and reporting	Inadequate and/or unclear error messages or lack of real-time error reporting can significantly impact the user's ability to diagnose and fix issues promptly	Applicable	Clear error messages (REQ_007), comprehensive IFU details on correct endpoints (REQ_012), technical support, RESTful API implementation (REQ_005)
Interoperability	Ensuring interoperability with various healthcare information systems, including EHR systems, has been a challenge. Compatibility issues can lead to data mismatches or loss, critical in healthcare applications	Applicable	Elastic demand design, constant backups, advanced security/software (REQ_005), REST features for status feedback, error codes, automatic awareness of downtime, support
Integration challenges	Difficulties encountered during the integration process, primarily due to compatibility issues with existing systems	Applicable	Clear error messages (REQ_007), comprehensive IFU with correct endpoints (REQ_012), technical support, RESTful API implementation (REQ_005)
Insufficient documentation	Inadequate, unclear and/or incomplete guidelines, leading to incorrect implementation and usage errors	Applicable	IFU-provided information (REQ_012), clear error messages (REQ_007), technical support for troubleshooting

Table 2: Summary of Known Use Problems

Analysis of Hazards Associated with Use of the Device

The manufacturer has conducted a comprehensive task analysis on the use of the device, which led to a list of potential use errors and a comprehensive use-related risk analysis (URRA). This process involved analyzing known use problems with similar devices, identifying the user interface characteristics related to safety, identifying potential use errors, and identifying known and foreseeable hazards and hazardous situations.

Please refer to R-TF-012-014 for the full use-related risk analysis (URRA).

Table 3 defines the possible severity ratings applied in the URRA.

Table 4 lists the applicable use steps and their severities. It also summarizes the URRA, including tasks, potential use errors, hazards and harms, severities, and risk mitigation measures.

Severity Classification	Harm Severity Descriptions by Risk Type	Severity Rating
Catastrophic	Results in patient death.	5
Critical	Results in permanent impairment or life-threatening injury.	4
Serious	Results in injury or impairment requiring professional medical intervention.	3
Minor	Results in temporary injury or impairment not requiring professional medical intervention.	2
Negligible	Inconvenience or temporary discomfort.	1

Table 3: Severity Ratings in URRA

No risks found with type "Usability"

Table 4: Use-related Risk Analysis

Summary of Preliminary Analyses and Evaluations

The manufacturer conducted various preliminary analyses and evaluations during product development. These helped inform the user interface design so it could be optimized with regards to safety and effectiveness.

Description and Categorization of Critical and Non-critical Tasks

A critical task is a user task which, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user, noting that harm is defined to include compromised medical care. Serious harm/injury is defined by 21 CFR 803.3(w) to mean an injury or illness that (1) is life-threatening, (2) results in permanent impairment of a body function or permanent damage to a body structure, or (3) necessitates medical or surgical intervention to preclude permanent impairment of a body function or permanent damage to a body structure. Per the severity rating classification of the URRA in Table 3, harms associated with severities of 3 or above will be considered critical tasks. For this study, critical tasks that logically occur in sequence when using the device will be grouped into use scenarios. All tasks (critical and non-critical) that are part of the natural workflow of a use scenario will be evaluated. However, non-critical tasks that are not necessary for the completion of critical task-based use scenarios are out of scope for evaluation. Table 5 lists the critical and non-critical tasks associated with the use of the device, the severity of each task based on the potential harm described in Table 3 above, as well as the use scenarios used in this study to assess each task.

Table 5: Critical and Non-critical Tasks

User Group	Phase	Use Task	Severity	Classification	Use Scenario Used to Evaluate
ITP	Get access token	Input authorization details	3	Critical	ITP Use Scenario 1
ITP	Get access token	Input correct URL for login endpoint	3	Critical	ITP Use Scenario 1
ITP	Get access token	Use a stable internet connection	3	Critical	ITP Use Scenario 1
ITP	Build JSON with data	Format the data	3	Critical	ITP Use Scenario 1
ITP	Build JSON with data	Enter mandatory fields	3	Critical	ITP Use Scenario 1
ITP		Enter data	3	Critical	ITP Use Scenario 1
ITP		Send images to the API	3	Critical	ITP Use Scenario 1
ITP		Send required data to the API	3	Critical	ITP Use Scenario 1
ITP	Send JSON to device	Send request to the endpoint	3	Critical	ITP Use Scenario 1
ITP	Send JSON to device	Input correct HTTP method	3	Critical	ITP Use Scenario 1
ITP	Send JSON to device	Send access token with the request	3	Critical	ITP Use Scenario 1
ITP	Receive JSON from device	Process the request	3	Critical	ITP Use Scenario 1
ITP	Process and store JSON	Extract the data	3	Critical	ITP Use Scenario 1
ITP	Build report	Generate report	3	Critical	ITP Use Scenario 1
HCP	Take pictures	User takes a photo of the patient's lesion	3	Critical	HCP Use Scenario 1
HCP	Authenticate in the system	User logs into the system	3	Critical	HCP Use Scenario 1; HCP Use Scenario 2
HCP	Upload pictures	User uploads photos to the client	3	Critical	HCP Use Scenario 1; HCP Use Scenario 2
HCP	Read and interpret the report	User understands the format of the device's output	3	Critical	HCP Use Scenario 3
HCP	Read and interpret the report	User correctly reports the probability of certain conditions based on the ICD categories	3	Critical	HCP Use Scenario 3
HCP	Read and interpret the report	User correctly interprets the report	3	Critical	HCP Use Scenario 3
HCP	Read and interpret the report	The user understands that the report is not a certain diagnosis	3	Critical	HCP Use Scenario 3

Details of Human Factors Validation Testing

Study Objectives

The goal of this study is to assess whether the device and its user interface, including its instructions for use, can be used safely and effectively by its intended users, for its intended uses, and in its intended use environments. Intended users will simulate a typical application case with the device based on its intended use.

Not finished

To achieve this goal, this validation usability study will be conducted on TBD, at TBD and consists of the simulated use of the device by representative users. The study will consist of a 60-minute one-on-one (moderator and participant) usability testing session, during which the study moderator will give participants, representing potential users, a series of tasks to perform and questions to answer with the device. Throughout the sessions, the study moderator will observe and record participants' objective performance and subjective feedback and ask follow-up questions regarding use of the device. The study will evaluate all observed use problems (i.e., use errors [UEs], close calls [CCs], and use difficulties [UDs]) as well as their associated root causes, see sections 8.6 Description of Data Collection and Methods and 8.7.1 Description of Root Cause Analysis for a summary of the data collection and evaluation method.

The study results will then be used to determine if the residual risk associated with product use is acceptable. While it is rarely feasible to eliminate all risk, it should be reduced to an acceptable level, where feasible and practical, such that the benefits of product use outweigh any residual risks.

Type of Testing

This HF study will involve simulated-use testing to explore how users may interact with the product in a natural and independent manner (to the extent possible). Representative users will use the product in realistic scenarios under simulated conditions.

This type of testing is sufficient to assess the adequacy of the user interface, as relevant aspects of actual product use can be simulated, including the conditions of use, product characteristics, and environments of use.

Study Test Environment

Realism

The study test environment will simulate the intended use environment for each user group, as follows:

• Clinical Environment: Healthcare providers will use the study device in a simulated clinical environment (i.e., a hospital room or doctor's office), consisting of a quiet, well-lit room inside a hospital where HCPs can perform diagnostic assessments. No effort will be made to mitigate natural distractions, such as a phone ringing, that may occur. In this environment, HCPs will take photos of a patient's skin lesions, and submit them to a computer integrated with the device software to produce a report.
• Clinical IT Office Environment: IT professionals will use the study device in a simulated clinical office environment (i.e., a hospital IT desk or clinic's office), consisting of a quiet, well-lit room inside a hospital where ITPs execute their work. No effort will be made to mitigate natural distractions, such as a phone ringing, that may occur. In this environment, ITPs will be working on a computer to integrate the device API into their hospital's IT system.

Study Locations

Not finished

This study will be conducted at Design Science's facilities in Philadelphia, PA on TBD and its facilities in Evanston, IL on TBD.

Study Devices and Materials

The devices used in this study will be production-equivalent, including its electronic IFU. As described in the section "Training", the manufacturer does not expect that users will receive training prior to using the device.

Study Devices and Materials - HCPs

To interact with the API, HCPs will be working with a user interface representative of what they may see during actual use following the integration of the API in their hospital system that they will access via a URL on a computer.

While each hospital system may display a different final user interface once the API is integrated, they will all incorporate the same functional elements that are evaluated in this study, namely the portal to upload images, and the subsequent diagnostic report data, i.e., detected conditions, preliminary analysis (malignancy, sensitivity, specificity), and other related data. While stylistic differences exist between various hospital systems, e.g., color schemes, fonts, arrangement of elements, they will all represent the same functions.

As this software-only medical device API only consists of the functions, the GUI design is outside of the scope of this validation study, and only the functional layer of the interface, i.e., the API interaction, will be assessed. Therefore, testing the API with a generic version of the user interface will be sufficient to evaluate the API. This generic interface is shown in Table 6.

Table 6: HCP User Interface for Usability Testing

User Interface Item	Written Description	Image
Software	Initial screen where HCPs will upload the photo of the lesion	img]:w-1/2>
Diagnostic report	Details of the diagnostic report following submitting the photo of lesion	img]:w-1/2>

Based on section "Description of Device User Interface", the following materials will be present in the study environment to simulate items that HCPs may use during their diagnostic assessment of a patient:

• Smartphone with the capability of taking high quality photos supplied by the manufacturer for usability testing
• Computer with the device software integrated into a representative EHR/EMR system
• Gloves
• High quality digital images of skin lesions supplied by the client for participants to upload to the device

Study Devices and Materials - ITPs

ITPs will be utilizing a computer with a representative EHR/EMR they would use in their hospital, a URL of the API, an Integrated Development Environment (IDE) installed, and the electronic IFU. They will conduct their typical practice to integrate the API into the hospital system with the materials provided. ITPs use IDEs to write, edit, debug, and run code. As such, an ITP may use programs such as Visual Code Studio to integrate the API. Therefore, this program is sufficient for usability testing. Figure 1 displays the user interface for Visual Code Studio, a representative IDE that ITPs would use in their typical practice.

Figure 1: Visual Code Studio IDE Interface for ITPs

Based on section 3 Description of Device User Interface, the following materials will be present in the study environment to simulate items that ITPs may use to complete their work:

• Computer with a representative EHR/EMR system and preferred IDE installed

Participants

Number, Type, and Rationale

The study will include 30+6 participants (i.e., 15 IT professionals, and 15 healthcare providers) who are representative of the intended users. There will be an additional 3 participants recruited for each user group to account for potential no-shows, disqualifications, and cancellations; however, testing will end after the fifteenth participant for each user group. Refer to Attachment B-Participant Screener for full inclusion/exclusion criteria.

Table 7 describes the user groups intended user information.

Table 7: User Group Details

User Group	# of Participants	Training Status	# of Sessions	Characteristics
ITPs	n = 15 + 3	Untrained	1	- Ages 18 years or older - Have experience with FHIR - Have experience integrating services and API into a hospital IT system - At least 1 year of experience working for a hospital IT department
HCPs	n = 15 + 3	Untrained	1	- Ages 18 years or older - Licensed physician (MDs, DOs) - Varying degrees of training in clinical diagnosis and management of skin lesions - At least 1 year of experience as an HCP

Inclusion and Exclusion Criteria

In addition to the defined criteria, to reduce bias and ensure representative users have been selected as participants, all participants must

• not be affiliated with market research, product development, pharmaceutical companies, or medical device companies,
• not have completed any medical market research related to software-aided adjunctive diagnostic devices within the last 3 months,
• bring photo identification to the study,
• have the capacity to understand and sign an informed consent form (ICF),
• be able and willing to participate in a 60-minute session and complete study assessments,
• be willing to be video recorded during the study,
• be able to speak, read, and comprehend English, and
• be a US resident.

Participant Protection

Identity Protection

The study moderator will assign a unique identification code (i.e., participant ID) to each participant as a means of referencing participants, so that the manufacturer is not able to associate participant names or any other unique personal identifiers with the study data.

The letter component of the participant ID corresponds to the user groups as presented in Table 8. The numerical component of the participant ID (ranging from 01-15) refers to the order in which each participant from each user group participated in the study.

Table 8: Participant IDs

Participant ID	User Group
I##	IT Professionals
H##	Healthcare Provider

Institutional Review Board Oversight

This human factors study will not be submitted to an Institutional Review Board (IRB) due to the following reasons:

• Risk to the participants is minimal. Testing is non-invasive and structured in a simulated-use environment.
• There will be no needles, treatment, or drug product present. Participants will be interacting with a software-only device.
• Participants will review and sign an informed consent form (ICF) and business confidentiality agreement providing consent to participate in the evaluation.

Tasks and Scenarios

Session Overview

The structure of each 60-minute testing session is detailed in Table 9.

Table 9: Session Overview

Session Phase	Description
Informed Consent Review	The study moderator will review the ICF with the participant, ensure that they understand the form, agree to participate, and agree to be recorded.
Background Questions	The study moderator will ask the participant a series of background questions to confirm that they meet the study inclusion criteria and collect relevant data.
Overview	An overall explanation of the testing procedure will be given.
ITP Use Scenario 1: Simulated Use Post Scenario Interview (including Root Cause Interview)	Each of the use scenarios will be administered, followed by an interview.
Conclusion	Participants will be given a chance to express any additional questions or comments.

User Group: ITP

Session Phase	Description
Informed Consent Review	The study moderator will review the ICF with the participant, ensure that they understand the form, agree to participate, and agree to be recorded.
Background Questions	The study moderator will ask the participant a series of background questions to confirm that they meet the study inclusion criteria and collect relevant data.
Overview	An overall explanation of the testing procedure will be given.
HCP Use Scenario 1: Simulated Use: No Lesion Post Scenario Interview (including Root Cause Interview)	Each of the use scenarios will be administered, followed by an interview.
HCP Use Scenario 2: Simulated Use: Lesion Post Scenario Interview (including Root Cause Interview)	Each of the use scenarios will be administered, followed by an interview.
HCP Use Scenario 3: Knowledge Assessment Post Scenario Interview (including Root Cause Interview)	Each of the use scenarios will be administered, followed by an interview.
Conclusion	Participants will be given a chance to express any additional questions or comments.

User Group: HCP

Use Scenarios and Tasks

All participants will complete all use scenarios described in Table 10 and Table 11 in that order. Prior to the completion of the use scenarios, the study moderator will introduce participants to the study environment and give a brief introduction to each scenario. Critical tasks must be completed successfully to demonstrate the safety and effectiveness of the device.

During the first scenario, participants will be asked to imagine that they have just received the device and must use it for the very first time, just as they would in a real-life situation.

ITPs will have the electronic IFU containing installation instructions for the device software available to use if they chose so. Also, a computer will be set up with a typical representation of hospital IT systems. This would include an installed EHR/EMR program and a representative IDE. They will be provided with a URL with the device API and will be tasked with integrating it with the hospital IT system on the computer. In order to assess the API, the study moderator will observe participants' behaviors and actions and note any use problems that occur throughout the integration process (e.g., difficulty logging in, managing tokens, understanding error messages etc.). The study computer will enable a screensharing feature that will allow the study team to observe the participant's progress. Furthermore, as participants interact with the API during simulated use, a log that is visible to the manufacturer is automatically generated that tracks their progress to confirm the completion of the assigned tasks. A representative from the manufacturer's team will be monitoring the log during the sessions to assist in identifying and characterizing any observed use problems.

Table 10: Use Scenarios and Descriptions for ITPs

Phase	Task	Classification	Success Criteria
Get access token	Input authorization details	Critical	User inputs correct username, password, or other required authentication details
	Input correct URL for login endpoint	Critical	User enters the correct URL for the login endpoint
	Use a stable internet connection	Critical	User uses a stable internet connection
Build JSON with data	Format the data	Critical	User formats the data correctly, does not include the wrong JSON structure or incorrect field names
	Enter mandatory fields	Critical	User enters mandatory fields required for the request
	Enter data	Critical	User does not enter data in incorrect types (e.g., string instead of integer)
Build report	Send images to the API	Critical	User codes for images to be sent in a supported format, or correctly encoded in Base64
	Send required data to the API	Critical	User does not include extraneous data not required for the request
Send JSON to device	Send request to the endpoint	Critical	User sends the request to the correct endpoint
	Input correct HTTP method	Critical	User uses the correct HTTP method
	Send access token with the request	Critical	User includes the access token in the header. The access token is not expired, and the request is only sent once
Receive JSON from device	Process the request	Critical	The user completes the response without corrupting it, misinterpreting the HTTP status, deleting or modifying it, or incorrectly mapping the response data to the clinical workflow or patient records.
Process and store JSON	Extract the data	Critical	User extracts the data with the JSON decoder. The extracted data are compatible with existing systems and do not overwrite existing data.
Build report	Generate report	Critical	User understands the data format, and the data are displayed consistently.

ITP Use Scenario 1: Simulated Use

HCPs will be tasked with using the API through a representative GUI to assist in diagnosing a patient that has come in to assess a skin lesion. To simulate taking photos, the HCP will be asked to take a photo of the hand of a member of the research team such as the study moderator or a second supporting analyst. The HCP will then submit it to the device that has been installed on a computer, running the report, and interpreting the results to see if they are consistent with the lesion. The HCP will be tasked with following this workflow to assess skin that does not bear a malignant lesion using the device only one time. The study moderator will then observe participants' behaviors and actions and note any use problems that occur during the usage of the software.

Following this simulated-use scenario (Scenario 1), a post scenario interview will be performed to better evaluate safe and effective use. To this end, participants will be asked how they felt the scenario went as well as whether they made any mistakes, almost made any mistakes, or had any difficulties. After this self-reflection, the moderator will probe for better understanding and delve into a root cause interview, where open-ended questions will be asked until the underlying cause of each use problem can be identified. This will include participant-reported and moderator-observed use problems from Scenario 1. For some tasks, successful completion may or may not be observable during the simulated-use scenario due to the nature of the task. For example, a task may require the participant to correctly interpret the data produced by the product (e.g., read the list report output). If task performance is not observable, the moderator may follow up in the root cause interview and ask open-ended, naturally worded follow up questions about the task to accurately assess the participant's task performance.

Following this, the HCPs will be asked to complete their workflow once again and will be directed to use the device to produce a report for a malignant skin lesion (Scenario 2). A digital file containing photos of a skin lesion will be provided to HCPs. The HCP will then upload these photos to the device that has been installed on a computer, running the report, and interpreting the results to see if they are consistent with the lesion. The HCP will be tasked with following this workflow to assess a malignant lesion using the device only one time. The study moderator will then observe participants' behaviors and actions and note any use problems that occur during the usage of the software.

For the HCP workflow, some important aspects of use, such as interpreting and understanding the report, cannot be assessed through task performance and instead are assessed through direct questioning of the participant to ascertain their understanding of information. This knowledge is tested by questioning the test participants using open-ended and naturally worded questions (Scenario 3). The knowledge assessment will be administered to determine their understanding of the information in the report output. The malignant skin lesions participants may assess are shown in Figure 2.

Figure 2: Malignant Skin Lesions (Squamous Cell Carcinoma, Melanoma, & Basal Cell Carcinoma)

In some instances, asking a directed question can bias the participant to provide the correct answer. In these cases, less specific wording will be used for the initial question to not influence the participant. Only if the participant's knowledge, or lack thereof, cannot be sufficiently assessed through their response to the initial question, optional and more focused follow up questions will be asked. These optional, open-ended, naturally worded follow-up questions will be asked in a tiered approach.

Participants will be free to refer to any materials when answering but will not be specifically guided to any aspect of the product interface (e.g., the Electronic IFU). The study moderator will note participants' answers and record any use problems. There will then be another post scenario interview, beginning with participants being asked how they felt answering the questions and whether there was anything they identified as unclear or confusing when answering. This will again be followed by any clarifying questions and a root cause interview on all remaining use problems to evaluate safe and effective use.

Table 11: Use Scenarios and Descriptions for HCPs

Use Scenario	Phase	Task	Classification	Success Criteria
HCP Use Scenario 1: Simulated Use: No Lesion	Take pictures	User takes a photo of the patient's lesion.	Critical	The photo has good quality, captures the relevant skin structure, is well lit, focused, with adequate distance
HCP Use Scenario 1: Simulated Use: No Lesion HCP Use Scenario 2: Simulated Use: Lesion	Authenticate in the system	User logs into the system.	Critical	The user enters the correct username and password and logs in successfully
HCP Use Scenario 1: Simulated Use: No Lesion HCP Use Scenario 2: Simulated Use: Lesion	Upload pictures	User uploads photos to the Legit.Health plus client.	Critical	The user uploads a high-quality image
HCP Use Scenario 1: Simulated Use: No Lesion HCP Use Scenario 2: Simulated Use: Lesion	Authenticate in the system	User logs into the system.	Critical	The user enters the correct username and password and logs in successfully
HCP Use Scenario 2: Simulated Use: Lesion	Upload pictures	User uploads photos to the Legit.Health plus client.	Critical	The user uploads a high-quality image
HCP Use Scenario 3: Knowledge Assessment	Read and interpret the report	User understands the format of the device's output	Critical	The user identifies the information in the report
		User correctly reports the probability of certain conditions based on the ICD categories.	Critical	The user identifies the probability of conditions listed in the report
		User correctly interprets the report output.	Critical	The user correctly identifies the quantification of disease intensity, extent, or count of clinical signs.
		The user understands that the report is not a certain diagnosis.	Critical	The user acknowledges that the output report is not a standalone diagnosis.

HCP Use Scenarios 1, 2 and 3

Description of Data Collection and Methods

The data to be collected are as follows:

• Observational Data: Participants will be given an opportunity to use the devices independently and in as realistic a manner as possible, without guidance, coaching, praise, or critique from the study moderator.
• Interview Data: The study moderator will ask participants open-ended, neutrally-worded questions to understand any use problems participants may have with the study devices and tasks. Particular care will be taken not to blame participants or bias them in later tasks.
  • Subjective assessment of use: This data consists of participants' subjective feedback on the root cause of potentially safety-related use errors and use difficulties, and opinions on all tasks for any residual risk, as well as the adequacy of instructions.
• Knowledge Task Data: Participants will be given an opportunity to explain their understanding of product use independently. This type of data is necessary as some critical tasks cannot be directly observed through simulated use; these critical tasks may involve users' understanding of information (e.g., warnings).

The following coding system will be used for simulated-use and knowledge task data collection and analysis:

• Success (OK): This coding describes a situation in which the participant correctly performs a task, or answers a knowledge task question, according to the steps in the User Guide or an acceptable alternative.
• Use error (UE): This coding describes a situation in which the participant performs a task incorrectly (i.e., not in accordance with the intended manner of use), does not complete a necessary task, or answers a knowledge task question incorrectly or incompletely.
• Close call (CC): This coding describes a situation in which the participant performs or nearly performs a task incorrectly (i.e., not in accordance with the intended manner of use), or answers a knowledge task question incorrectly or incompletely, but then resolves the issue prior to any harm being done without assistance.
• Use difficulty (UD): This coding describes a situation in which the participant struggles to some extent while completing a task or answering a knowledge task question (e.g., confusion, taking longer than expected, difficulty manipulating a device's components), but eventually can completes the task or answers the question successfully without assistance.

Recording Equipment

Audio and video will be recorded for all study sessions using synchronized, multi-channel, HD-resolution video.

Recording Software

Study data will be recorded by the study team into OpenClinica. OpenClinica is a web-based platform that allows the study moderator to directly record their scores, observations, and notes into an electronic version of the study protocol. All data entered into OpenClinica and any subsequent changes made by the study team are visible via an audit log. Any changes made to the data after it is entered requires a rationale for the change to be documented.

OpenClinica is compliant with the FDA's Title 21 CFR Part 11.

Data Analysis

Observational data during simulated-use and knowledge task data will be collected and any use problems will be recorded as moderator observations. Based on this, as previously described, an interview will be conducted to better discern any use problems (i.e., UEs, CCs, and UDs), the root cause of such problems, and the associated residual risk. Such interview data will be recorded as participant comments.

Description of Root Cause Analysis

Root cause analysis (RCA) will involve examining the data to determine the underlying cause of each use problem, with a focus on investigating contributing factors to each problem and determining ways to prevent them in the future. Moderator observations and participant comments will be analyzed holistically during this process to evaluate the safety and effectiveness of product use.

Description of Residual Risk Analysis

All risks that remain after human factors validation testing will be analyzed to determine whether they can be reduced or eliminated. This analysis will determine whether design modifications are needed, would be possible and might be effective at reducing the associated risks to acceptable levels. This analysis will include a risk-based discussion of the observed use events and their root causes as well as the effectiveness of current risk mitigations.

If user interface design modifications are required to further mitigate risk, the residual risk analysis will include a description of the recommended design changes and a discussion regarding whether additional HF validation testing is required.

The residual risk analysis will be used to evaluate the safety and effectiveness of product use and determine if the residual risk associated with product use is acceptable. While it is rarely feasible to eliminate all risks, it should be reduced to an acceptable level, where feasible and practical, such that the benefits of product use outweigh any residual risks.

Appendix A: Interview Guide

Thanks for coming in today. In a few moments, we'll start the session, but before we begin, I want to review a few things with you. You have already signed the informed consent. Do you have any additional questions about the document? I would also like to make sure you:

• Promise not to tell anyone about what you see today.
• Give us permission to audio and video record this session, so long as none of the footage is ever made public.
• Agree that the study's Sponsor can make use of any information you provide during the study without any restrictions.

[Answer any questions and then proceed.]

Ok, I'm going to turn on the video and audio recording now.

Testing Session

Today, I am going to ask you to go through a series of use scenarios with a software-only device that analyzes images of visible skin structure abnormalities to support the assessment of all diseases of the skin. I want to get your feedback on the device and will also ask you some questions regarding the use of the device.

For the purpose of our study, we are simulating a real use scenario, so please imagine that you are in a real situation and perform all tasks as if you were at work.

If for some reason you feel that you are unable to complete a task the way you would at work due to this simulated environment, please let me know.

At times, you may have questions for me that I may or may not be able to answer. If I am unable to answer, it is because I want to observe how you would handle a real-life situation when problems arise.

Please remember that you are here to help us evaluate the product. We are not evaluating you or your abilities in any way. You are the expert here to help us improve the product today.

Finally, please keep in mind, I did not design anything you're seeing today, so you do not have to worry about my feelings regardless of whether your feedback is positive or negative. We want your honest opinions, good or bad. So, if you see any problems with the devices I will be showing you that you think I should know about, please mention them.

Do you have any questions before we start?

ITP Use Scenario 1: Simulated Use

Moderator to prepare computer with hospital EHR system installed, and link to electronic IFU.

For our first task, I would like you to imagine that your hospital is looking to incorporate the device into their system. I would like you to complete all the steps necessary to install and set up the device for use. You can refer to any of the materials provided at any time.

When you're ready, please begin and take your time.

Task	Outcome	Observations
Input authorization details	OK UD CC UE
Input correct URL for login endpoint	OK UD CC UE
Format the data	OK UD CC UE
Enter mandatory fields	OK UD CC UE
Enter data	OK UD CC UE
Send images to the API	OK UD CC UE
Send request to the endpoint	OK UD CC UE
Input correct HTTP method	OK UD CC UE
Send access token with the request	OK UD CC UE
Process the request	OK UD CC UE
Extract the data	OK UD CC UE
Generate report	OK UD CC UE

Post-Scenario Interview

How did that go?

Did you make any mistakes or almost make any mistakes during that scenario?

Did you have any difficulty at any point during that scenario?

Follow up on participant-observed use problems and root-causes.

Follow up on study moderator-observed use problems and root-causes.

HCP Use Scenario 1: Simulated Use: No Lesion

Moderator to prepare smartphone, computer with hospital EHR/EMR system installed, the device installed, and manikin with simulated skin lesions

For our first task, I would like you to imagine that an IT professional has implemented the device into your hospital EHR system. Your patient has visited your clinic because of concerns over the skin of the back of their hand (Moderator to direct participant to a researcher to take a photo of their hand). I would like you to complete all the steps necessary to assess the area by generating a report using the device. You can refer to any of the materials provided at any time.

When you're ready, please begin and take your time.

Task	Outcome	Observations
User takes a photo of the patient's hand	OK UD CC UE
User logs into the system	OK UD CC UE
user uploads photos to the client	OK UD CC UE
User generates a report	OK UD CC UE

Post-Scenario Interview

How did that go?

Did you make any mistakes or almost make any mistakes during that scenario?

Did you have any difficulty at any point during that scenario?

Follow up on participant-observed use problems and root-causes.

Follow up on study moderator-observed use problems and root-causes after Scenario 3.

HCP Use Scenario 2: Simulated Use: Lesion

Moderator to prepare smartphone, computer with hospital EHR/EMR system installed, the device installed, and manikin with simulated skin lesions

For our next task, I would like you to imagine that your patient has visited your clinic again because of concerns over a skin lesion that has appeared on their body. Imagine that these are the images you have taken of the lesion (Moderator to provide participant with images of the skin lesions provided by the sponsor) I would like you to complete all the steps necessary to assess the lesion by generating a report using the device. You can refer to any of the materials provided at any time.

When you're ready, please begin and take your time.

Task	Outcome	Observations
User logs into the system	OK UD CC UE
User uploads photos to the client	OK UD CC UE
User generates a report	OK UD CC UE

Post-Scenario Interview

How did that go?

Did you make any mistakes or almost make any mistakes during that scenario?

Did you have any difficulty at any point during that scenario?

Follow up on participant-observed use problems and root-causes.

Follow up on study moderator-observed use problems and root-causes after Scenario 3.

HCP Use Scenario 3: Knowledge Assessment:

Now I'm going to ask you a few questions about the device user interface. Also, this is not a test of any kind; you may refer to any of the materials provided at any point to help you answer the questions.

Correct answers are bolded.

Q#	Question	Response
Q1	What information is produced by the Legit.Health Plus report? [Top 5 predictions, quantification of disease intensity, extent, or count of clinical signs]	OK UD CC UE
Q2	What is the probability of malignancy for the lesion you analyzed today? [The percentage that the report produces]	OK UD CC UE
Q3	What are the conditions that the Legit.Health Plus have detected? [The conditions that the report produces]	OK UD CC UE
Q4	Can this report output of the Legit.Health Plus act as a diagnosis? [No]	OK UD CC UE

Post-Scenario Interview

How was answering those questions?

Did you notice anything related to the product and its associated materials that was unclear or confusing when answering those questions?

Follow up on all (study moderator-observed and participant-observed) use problems and root-causes up to this point (that is, from Scenario 3).

Thank and Exit

Do you have any additional questions or comments about anything that you have seen or done here today?

That is the end of our session. Thank you for your time and feedback. [Stop recording and walk participant out.]

Were there any protocol deviations?

If so, record:

• Yes
• No

Appendix B: Electronic IFU

Electronic IFU is available at the following URL: https://apidocs-draft.legit.health.