R-012-014 Summative evaluation plan_2024_001

Objective

This summative evaluation study aims at validating through objective and empirical evidence that our software as medical device can be used in an easy and safe manner.

More specifically, the summative evaluation test aims to evaluate the risks associated with use errors to confirm they are properly addressed and therefore to confirm the safe use of the device.

This study is planned and executed according to the requirements set out in IEC 62366-1:2015 (Medical devices - Part 1: Application of usability engineering to medical devices).

Abbreviations and definitions

• Instruction for use: in accordance with the Medical Device Regulation, the term `instructions for use refers to the information provided by the manufacturer to inform the user of a device's intended purpose and proper use and of any precautions to be taken
• Intended user: Users are defined in IEC 62366-1:2015 as person interacting with (i.e. operating or handling) the medical device.
• Summative user testing: It (quantitative testing) provides an indirect assessment of the usability of the design. A group of testers are given a task, and based on their performance, the usability quotients of the design elements are measured.
• Tester: person who has been asked to perform the tasks described in the present protocol.
• eIFU : Electronic instruction for use
• IFU: Instruction for use
• Use environment: actual conditions and setting in which users interact with the medical device
• User interface: means by which the user and the medical device interact
• URRA: use related risk analysis
• SaMD: software as medical device.

Introduction

AI Labs Group S.L. is a European company with headquarters in Bilbao that designs, develops, manufactures software as a medical device to aid healthcare practitioners and organisations in their clinical decision-making process, thus enhancing the efficiency and accuracy of care delivery.

The device is computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. It is built as an API that follows the REST protocol and its principal function is to provide a wide range of clinical data from the analysed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organisations to gather data and improve their workflows.

The generated data is intended to aid healthcare practitioners and organisations in their clinical decision-making process, thus enhancing the efficiency and accuracy of care delivery. The device should never be used to confirm a clinical diagnosis. On the contrary, its result is one element of the overall clinical assessment. Indeed, the device is designed to be used when a healthcare practitioner chooses to obtain additional information to consider a decision.

Due to the nature of our device (software built as an API to be integrated in the healthcare organization system), we identified as user group of the summative evaluation test IT professionals because they are the ones interacting with the device.

More details about the analysis and choice of this user group are provided in the section Intended users above.

Description of intended users, uses, environments and training

Device Identification

	Information
Device name	Legit.Health Plus (hereinafter, the device)
Model and type	NA
Version	1.0.0.0
Basic UDI-DI	8437025550LegitCADx6X
Certificate number (if available)	MDR 792790
EMDN code(s)	Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software)
GMDN code	65975
Class	Class IIb
Classification rule	Rule 11
Novel product (True/False)	FALSE
Novel related clinical procedure (True/False)	FALSE
SRN	ES-MF-000025345

Intended use

The device is a computational software-only medical device intended to support health care providers in the assessment of skin structures, enhancing efficiency and accuracy of care delivery, by providing:

• quantification of intensity, count, extent of visible clinical signs
• interpretative distribution representation of possible International Classification of Diseases (ICD) classes.

Quantification of intensity, count and extent of visible clinical signs

The device provides quantifiable data on the intensity, count and extent of clinical signs such as erythema, desquamation, and induration, among others; including, but not limited to:

• erythema,
• desquamation,
• induration,
• crusting,
• dryness,
• oedema,
• oozing,
• excoriation,
• swelling,
• lichenification,
• exudation,
• depth,
• edges,
• undermining,
• pustulation,
• hair loss,
• type of necrotic tissue,
• amount of necrotic tissue,
• type of exudate,
• peripheral tissue edema,
• peripheral tissue induration,
• granulation tissue,
• epithelialization,
• nodule count,
• papule count,
• pustule count,
• cyst count,
• comedone count,
• abscess count,
• draining tunnel count,
• lesion count

Image-based recognition of visible ICD classes

The device is intended to provide an interpretative distribution representation of possible International Classification of Diseases (ICD) classes that might be represented in the pixels content of the image.

Device description

The device is computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organisations to gather data and improve their workflows.

The generated data is intended to aid healthcare practitioners and organizations in their clinical decision-making process, thus enhancing the efficiency and accuracy of care delivery.

The device should never be used to confirm a clinical diagnosis. On the contrary, its result is one element of the overall clinical assessment. Indeed, the device is designed to be used when a healthcare practitioner chooses to obtain additional information to consider a decision.

Intended medical indication

The device is indicated for use on images of visible skin structure abnormalities to support the assessment of all diseases of the skin incorporating conditions affecting the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral and genital), the dermis, the cutaneous vasculature and the subcutaneous tissue (subcutis).

Intended patient population

The device is intended for use on images of skin from patients presenting visible skin structure abnormalities, across all age groups, skin types, and demographics.

Intended user

The medical device is intended for use by healthcare providers to aid in the assessment of skin structures.

User qualification and competencies

In this section we specificy the specific qualifications and competencies needed for users of the device, to properly use the device, provided that they already belong to their professional category. In other words, when describing the qualifications of HCPs, it is assumed that healthcare professionals (HCPs) already have the qualifications and competencies native to their profession.

Healthcare professionals

No official qualifications are needes, but it is advisable if HCPs have some competencies:

• Knowledge on how to take images with smartphones.

IT professionals

IT professionals are responsible for the integration of the medical device into the healthcare organisation's system.

No specific official qualifications are needed, but it is advisable that IT professionals using the device have the following competencies:

• Basic knowledge of FHIR
• Understanding of the output of the device.

Use environment

The device is intended to be used in the setting of healthcare organisations and their IT departments, which commonly are situated inside hospitals or other clinical facilities.

The device is intended to be integrated into the healthcare organisation's system by IT professionals.

Operating principle

The device is computational medical tool leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures.

Body structures

The device is intended to use on the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral and genital), the dermis, the cutaneous vasculature and the subcutaneous tissue (subcutis).

In fact, the device is intended to use on visible skin structures. As such, it can only quantify clinical signs that are visible, and distribute the probabilities across ICD classes that are visible.

We have conducted a thorough user analysis to identify the primary user groups who will interact with our software as a medical device (SaMD). Our SaMD is an API designed to be integrated into healthcare organization systems, requiring a focused approach to usability testing. Based on our analysis, we have determined that IT professionals are the exclusive user group for the following reasons:

• The API is a backend tool intended to be integrated into larger healthcare systems and is not directly used by clinical staff, patients, or other non-technical stakeholders. The primary interaction with the API is through its integration, configuration, and maintenance, tasks that are exclusively performed by IT professionals.
• The integration and management of the API require technical knowledge. IT professionals possesses the required expertise and skills to interact with and utilize the API effectively, ensuring proper implementation and troubleshooting.
• The API is designed to function within the context of a healthcare organization's IT infrastructure. The primary concerns during the summative evaluation include compatibility, integration efficiency, system performance and security. IT professionals are uniquely positioned to evaluate these aspects due to their familiarity with the existing systems and their technical requirements.
• Usability considerations focus on technical integration, ease of implementation, system reliability, clarity of error messages, all of which are within the purview of IT professionals. This user group can provide the most relevant feedback on these aspects.
• Role of HCPs and patients: while HCPs and patients are important stakeholders, their interaction with the device is mediated through systems managed by IT professionals, not through direct use of the API itself.

Given these points, we have concluded that IT professionals represent the most appropriate and relevant user group for our summative evaluation test. By focusing on this user group, we can ensure that the API meets the necessary technical and integration requirements, ultimately facilitating a smooth implementation within healthcare systems.

Intended use environment

The device is intended to be used in healthcare organization facilities.

The summative evaluation will be perfomed in the healthcare organisations'offices where the intended users (IT professionals) usually execute their work.

It is expected that users will set aside specific time to use the product in an environment with minimal noise and distractions.

Training

User training is not required for the safe use of the device.

Description of device user interface

Device user interface

The user interface of the device comprises 3 elements:

• API Endpoints: URL structures, expected HTTP methods, and status codes.
• API Documentation: IFU, use cases, sample requests, and responses.
• Data Structure: JSON payload formats, including field names and FHIR nomenclature.

These are the testable design and technical requirements of the user interface of the device.

What is the "interface" of an API?

When testing the usability of an API-centric medical device such as ours, the "interface" comprises its endpoints, documentation, error messages, and data formats. Usability for such systems often hinges on the clarity of the documentation, the intuitiveness of the API design, and the robustness of the system in handling various data scenarios. This usability plan will ensure that all these aspects are considered and evaluated.

Summary of operating sequence

We performed a task analysis to identify the operating sequence of the device and the tasks to be performed by the IT professionals; the results are listed below:

• Get access token
• Read IFU
• Build JSON with data
• Send JSON to the device
• Receive JSON from the device
• Process and store JSON
• Build report.

Summary of known use problems

We conducted research on known or expected use problems for similar products and product types and previous models of the device. Based on this research, the following use problems have been identified to pertain to the device:

• Integration challenges: difficulties encountered during the integration process, primarily due to compatibility issues with existing systems.
• Insufficient documentation: inadequate, unclear and/or incomplete guidelines, leading to incorrect implementation and usage errors.
• User authentication and security: APIs in healthcare settings must adhere to strict security standards. Past experiences have highlighted issues related to complex authentication processes and maintaining compliance with security protocols.
• Error handling and reporting: inadequate and/or unclear error messages or lack of real-time error reporting can significantly impact the user's ability to diagnose and fix issues promptly.
• Interoperability: ensuring interoperability with various healthcare information systems, including Electronic Health Records (EHR) systems, has been a notable challenge. Compatibility issues can lead to data mismatches or loss, which are critical in healthcare applications.

Analysis of hazards associated with the use of the device

We conducted a task analysis on the use of the device which led to a comprehensive use related risk analysis (URRA) and list of potential use errors. This process involved analyzing known use problems with similar devices, identifying the user interface characteristics related to safety, identifying potential use errors, and identifying known and foreseeable hazards and hazardous situations, foreseeable sequence of events and associated harms.

Each task has been classified as Critical or Non-critical based on the following definition: critical task is a user task which, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user, noting that harm is defined to include compromised medical care.

Based on this definition, we classify user tasks as critical when the severity score is between 3 and 5, while we classify user tasks as non-critical when the severity score is between 0 and 2.

The severity and probability scores listed in the tables below (together with their definitions) are the ones defined in our internal procedure GP-013 Risk management and used for the user related risk analysis:

Score	Severity	Meaning
5	Critical	Loss or degradation of primary function or death
4	Serious	Permanent impairment or irreversible injury
3	Major	Injury or impairment requiring medical or surgical intervention
2	Minor	Temporary injury or impairment not requiring medical or surgical intervention
1	Negligible	Temporary inconvenience or disruption, no harm to the patient or user.
0	None	None

Score	Probablity	Meaning
5	Frequent	It is very probable that happens. When the dangerous situation occurs, there is a possibility >75% that it could lead to damage to the patient.
4	Probable	It is probable that happens. When the dangerous situation occurs, there is a possibility between >50% ≤75% that it could lead to damage to the patient.
3	Occasional	It is possible that happens often. When the dangerous situation occurs, there is a possibility between >10% ≤50% that it could lead to damage to the patient.
2	Remote	It is probable that happens some time. When the dangerous situation occurs, there is a possibility between >0,1% ≤10% that it could lead to damage to the patient.
1	Improbable	It not impossible that happens but exists some possibility it is that happens . When the dangerous situation occurs, there is a possibility ≤ 0,1% that it could lead to damage to the patient.
0	Impossible	It is impossible that happens

The use related risk analysis is provided below:

Risk ID	Use Step/ Task Statement	User Group	Task Classification	Use Error (Potential Failure Modes)	Potential Result of Use Error or Hazard (Potential Failure Modes)	Foreseeable Sequence of Events (Potential Failure Modes)	Hazardous Situation (Potential Failure Modes)	Potential Harm (Potential Failure Modes)	Severity (Risk Pre-Mitigation)	Probability (Risk Pre-Mitigation)	Risk Priority Number (Risk Pre-Mitigation)	Risk control measures (Risk control)
UR-001	Get access token	ITP	Critical	User may input incorrect username, password, or other required authentication details	Repeated failed login attempts could lead to account lockout or delays in data access, impacting timely patient care	- The IT professional attempts to access the API to integrate it into the healthcare organization's system - The IT professional inputs the required authentication details - The IT professional inputs incorrect authentication details (e.g., mistyped username or password)	The user is unable to obtain an access token due to the incorrect input of authentication details, leading to a failure to authenticate and access the API	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device	3	4	12	Clear error messages, provide technical support to assist users in troubleshooting and resolving authentication issues
UR-002	Get access token	ITP	Critical	User may enter an incorrect URL for the login endpoint	The device may fail to authenticate, preventing access to the protected endpoints and delaying clinical decision-making	- The IT professional attempts to access the API to integrate it into the healthcare organization's system - The IT professional configures the system with the URL for the login endpoint to obtain an access token - The IT professional enters an incorrect URL for the login endpoint - The system attempts to contact the incorrect URL, resulting in a failed request	The system fails to obtain an access token due to the incorrect URL, preventing authentication and access to the API	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden	3	3	9	Clear error messages, provide technical support to assist users in troubleshooting, provide comprehensive information in the IFU
UR-003	Get access token	ITP	Critical	User may use poor or unstable internet connection	Inability to retrieve or validate the access token, causing delays in accessing the device's features	- The IT professional attempts to access the API to integrate it into the healthcare organization's system - The system sends a request to the login endpoint to obtain an access token - The internet connection is poor or unstable, leading to intermittent connectivity or a complete loss of connection - The request to the login endpoint fails or times out due to the unstable connection - The system or the user may attempt to retry the request multiple times, experiencing repeated failures	The system fails to obtain an access token due to the unstable connection, preventing authentication and access to the API	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Robust error handlings, state-of-the-art techniques of security and software availability, provide information in the IFU about minimum IT requirements, provide technical support to assist users in troubleshooting
UR-004	Build JSON with data	ITP	Critical	User might format the data incorrectly, such as wrong JSON structure or incorrect field names	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data. - The IT professional enters the data into the JSON object - The IT professional makes an error, resulting in incorrect JSON structure (e.g., missing brackets, commas) or incorrect field names (e.g., misspelled or mismatched with the API specification) - The incorrectly formatted JSON is submitted to the API endpoint - The API detects the formatting error and responds with an error message	The system fails to process the incorrectly formatted JSON, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Clear error messages, comprehensive information in the IFU that details the required JSON structure, field names, examples of correctly formatted data, provide technical support for troubleshooting
UR-005	Build JSON with data	ITP	Critical	User might omit mandatory fields required for the request	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data - The IT professional enters the data into the JSON object - The IT professional omits one or more mandatory fields required by the API specification - The incomplete JSON is submitted to the API endpoint - The API detects the missing mandatory fields and responds with an error message	The system fails to process the incomplete JSON, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Clear error messages, comprehensive information in the IFU that details the required JSON structure, field names, examples of correctly formatted data and mandatory fields, provide technical support for troubleshooting
UR-006	Build JSON with data	ITP	Critical	User may enter data in incorrect types (e.g., string instead of integer)	If the device is unable to parse the incorrect data type to the data type defined by the schema of the request, then the user's request will be rejected. If the error message is not clear enough, the user may not be able to send a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data - The IT professional enters the data into the JSON object - The IT professional enters data in incorrect types, such as a string instead of an integer or vice versa - The incorrectly typed JSON is submitted to the API endpoint - The API detects the type error and responds with an error message	The system fails to process the JSON due to incorrect data types, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	3	9	Clear error messages, comprehensive information in the IFU that details the required JSON structure, field names, examples of correctly formatted data, correct data types and mandatory fields, provide technical support for troubleshooting
UR-007	Build JSON with data	ITP	Critical	User may send images in an unsupported format (other than Base64) or incorrectly encoded in Base64	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data, including images - The IT professional encodes the images to be included in the JSON object - The IT professional either uses an unsupported format (other than Base64) or incorrectly encodes the images in Base64 - The incorrectly formatted JSON, containing the improperly encoded images, is submitted to the API endpoint - The API detects the unsupported or incorrect image encoding and responds with an error message	The system fails to process the JSON due to unsupported image formats or incorrect Base64 encoding, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Validation checks, clear error messages, comprehensive information in the IFU, provide technical support for troubleshooting
UR-008	Build JSON with data	ITP	Critical	User may include extraneous data not required for the request	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data - The IT professional enters the data into the JSON object - The IT professional includes extraneous data that is not required or specified by the API documentation - The JSON object, containing the extraneous data, is submitted to the API endpoint - The API processes the request and responds with an error message	The system may fail to process the JSON correctly	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	3	9	Clear error messages, comprehensive information in the IFU, provide technical support for troubleshooting
UR-009	Send JSON to device	ITP	Critical	User might send the request to the wrong endpoint or to a non-existent URL	Failed to connect to the device, delaying the decision-making process and potentially impacting patient care	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional incorrectly enters the URL or endpoint - The JSON request is sent - The device responds with an error	The system fails to process the JSON due to sending the request to an incorrect or non-existent endpoint, leading to a failure in data transmission	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Clear error messages, comprehensive information in the IFU that details the correct endpoints, provide technical support for troubleshooting
UR-010	Send JSON to device	ITP	Critical	User may use the wrong HTTP method (e.g. GET instead of POST)	Failed to connect to the device, delaying the decision-making process and potentially impacting patient care	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional selects the HTTP method to be used for the request (e.g., GET, POST) - The IT professional mistakenly chooses the wrong HTTP method, such as selecting GET instead of the required POST - The JSON request is sent to the device using the incorrectly chosen HTTP method - The device responds with an error message	The system fails to process the JSON request due to using an incorrect HTTP method, leading to a rejection of the request by the device	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	3	9	Clear error messages, comprehensive information in the IFU that details the supported HTTP methods, provide technical support for troubleshooting
UR-011	Send JSON to device	ITP	Critical	User may not include the access token in the request	Authentication will fail, leading to interruptions in workflow and potential delays in the clinical assessment process	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional obtains the access token required for authentication - The IT professional constructs the JSON request, intending to include the required access token - The IT professional mistakenly omits the access token from the request - The JSON request lacking the access token is sent to the device - The device detects the missing access token and responds with an error message	The system fails to authenticate the user due to the absence of the access token, leading to a rejection of the request by the device	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data security risk	3	4	12	Clear error messages, inclusion of the requirement for including access token in all requests in the IFU, provide technical support for troubleshooting
UR-012	Send JSON to device	ITP	Critical	User may include the access token in the body of the request instead of headers	Authentication will fail, leading to interruptions in workflow and potential delays in the clinical assessment process	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional obtains the access token required for authentication - The IT professional constructs the JSON request and mistakenly includes the access token in the body of the request payload instead of the request headers - The JSON request with the access token in the body is sent to the device - The device receives the request but does not recognize the access token in the body as it is expecting it in the headers	The device may fail to authenticate the request due to the access token being included in the wrong part of the request, leading to potential unauthorized access or data security risks.	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data security risk	3	4	12	Clear error messages, inclusion of correct method of including access token in all requests in the IFU, provide technical support for troubleshooting
UR-013	Send JSON to device	ITP	Critical	User might attempt to send an expired token	Authentication will fail, leading to interruptions in workflow and potential delays in the clinical assessment process	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional obtains the access token required for authentication - The access token obtained by the IT professional expires due to reaching its predefined expiration time - The IT professional attempts to use the expired access token to authenticate the JSON request - The JSON request with the expired access token is sent - The device receives the request with the expired token and responds with an error message	The device may fail to authenticate the request due to the access token being included in the wrong part of the request, leading to potential unauthorized access or data security risks.	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data security risk	3	3	9	Clear error messages, include timeframe of token expiration and clear instructions on how to obtaining new token in the IFU, provide technical support for troubleshooting
UR-014	Send JSON to device	ITP	Critical	User might send the exact same request multiple times	Redundant processing, wasting resources and possibly causing confusion in clinical assessments	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional sends the JSON request - Due to network instability, or misunderstanding, the IT professional resends the same JSON request multiple times - The device receives multiple identical requests from the same user within a short period - The device processes each request as a new submission	The device processes multiple identical requests, potentially leading to duplicate actions or data entries	Misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data duplication	3	3	9	Clear error messages, provide technical support for troubleshooting
UR-015	Receive JSON from device	ITP	Critical	Network issues, device downtime, or incorrect handling of the request	Delays in obtaining clinical data, which can impact the timeliness and accuracy of clinical assessments	- The IT professional or system initiates a request to receive JSON data from the device - During the communication, network issues (such as instability, high latency, or disconnection) or device downtime occurs or the IT professional incorrectly handles the request, leading to improper processing or failure to receive the expected data - The device either fails to respond, sends a corrupted response, or the client misinterprets the response due to handling errors - The client system either waits indefinitely, retries the request, or processes an incomplete/corrupt JSON response	The system does not receive the necessary JSON data correctly or in a timely manner, leading to a failure in processing the required information	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, incomplete data processing	3	4	12	State-of-the-art techniques of security and software availability, clear error messages, implementation of recovery strategy, monitoring of device downtime, provide technical support for troubleshooting, information on how to handle request provided in the IFU
UR-016	Receive JSON from device	ITP	Critical	The response might be incomplete or corrupted due to data transmission issues	Incomplete or corrupted data can lead to incorrect clinical assessments	- The IT professional or system initiates a request to receive JSON data from the device - The device processes the request and sends the JSON data back to the client system - During the data transmission, network instability, interference, or other issues cause the data to be incomplete or corrupted - The client system receives the JSON response, but the data is either incomplete, or corrupted - The client system attempts to process the received JSON data, leading to potential errors or failures	The system processes incomplete or corrupted data, leading to potential errors in the application's functioning and data interpretation	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	State-of-the-art techniques of security and software availability, clear error messages, provide technical support for troubleshooting
UR-017	Receive JSON from device	ITP	Critical	Misinterpretation of the HTTP status code of the response	If a user misinterprets a successful status code (e.g., 200 OK) as an error, they might not process the received data. Conversely, interpreting an error status code (e.g., 404 Not Found, 500 Internal device Error) as successful could lead to the belief that the data has been processed correctly, when it hasn't. This delay or failure in data processing can impact timely clinical decisions.	- The IT professional initiates a request to receive JSON data from the device - The device processes the request and sends back a response with an HTTP status code and JSON data - The client system receives the response from the device, including the HTTP status code and JSON payload - The client system or IT professional misinterprets the HTTP status code (e.g., interpreting a 500 Internal Server Error as a 200 OK, or vice versa) - Based on the misinterpreted status code, the client system or IT professional takes incorrect actions, such as processing incomplete data or failing to retry a failed request	The system proceeds with incorrect assumptions about the success or failure of the request, leading to potential errors in data handling and application behavior	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	Comprehensive information on HTTP status codes and their meanings provided in the IFU, clear error messages, provide technical support for troubleshooting
UR-018	Receive JSON from device	ITP	Critical	User may accidentally delete or modify the request during the process	Clinical information generated by the device is lost. The user would have to send the same request, leading to a delay in the decision-making process	- The IT professional or system initiates a request to receive JSON data from the device - The client system prepares the request for data transmission - The IT professional accidentally deletes or modifies the request - The client system attempts to handle the response, which may be an error message or an unexpected result due to the incorrect request	The system does not receive the expected JSON data due to the accidental modification or deletion of the request, leading to potential errors in the application's functioning and data interpretation	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	Clear error messages, provide technical support for troubleshooting
UR-019	Receive JSON from device	ITP	Critical	User might incorrectly map the response data to the clinical workflow or patient records	Incorrect data mapping can result in misidentification or misassociation of clinical results with the wrong patient, leading to potential harm	- The IT professional or system initiates a request to receive JSON data from the device - The device processes the request and sends back a JSON response containing the clinical data - The client system receives the JSON response from the device - The IT professional processes the JSON response and maps the data to the clinical workflow or patient records - During this process, the user incorrectly maps the data (e.g., assigns data to the wrong patient record, uses incorrect data fields, or mismatches data types) - The mapped data is used in the clinical workflow or patient records, impacting clinical evaluations or decisions	Incorrect mapping of response data leads to erroneous data being integrated into the clinical workflow or patient records	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	Comprehensive information on how to interpret and map the JSON response data accurately provided in the IFU, error handling
UR-020	Process and store JSON	ITP	Critical	User may misinterpret the format or structure of the response data, leading to incorrect data extraction	Inaccurate or incomplete clinical data being stored, potentially leading to incorrect clinical decisions	- The client system receives the JSON response from the device - The IT professional begins processing the JSON response to extract relevant data - The user misinterprets the format or structure of the JSON data (e.g., misunderstanding the hierarchy, incorrect field names, or data types) - Due to the misinterpretation, the system extracts data incorrectly (e.g., extracting wrong values, missing important data, or misinterpreting data fields) - The incorrectly extracted data is stored in the database or integrated into the clinical workflow	Incorrectly extracted and stored data leads to errors in the clinical workflow or patient records	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	Detailed information on the JSON response format and structure to guide users in correctly interpreting the data provided in the IFU, clear error messages, provide technical support for troubleshooting
UR-021	Process and store JSON	ITP	Critical	Issues with the storage system, such as database failures or insufficient storage capacity	Loss of critical clinical data or inability to retrieve stored data when needed can delay patient care	- The client system receives the JSON response from the device - The client system processes the JSON data to prepare it for storage - The system attempts to store the processed data in the database - The storage system encounters issues such as database failures, insufficient storage capacity, or connection problems - The system may fail to store the data correctly, partially store the data, or not store the data at all - Based on the system's handling of the error, the user may take incorrect actions, such as assuming the data is stored correctly	Data loss or corruption due to issues with the storage system, leading to incomplete or missing clinical data	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	The deployment of the medical devices uses elastic demand design. The medical device makes constant backups. State-of-the-art techniques of security and software availability. Due to the inherent features of the REST protocol, when a user send a request and the device is down, the device returns a specific code informing of the state of the device, including downtime. This means that the user will be automatically aware of downtime, as well as any other states. Clear error messages, provide technical support for troubleshooting
UR-022	Process and store JSON	ITP	Critical	A decoder other than JSON is used to extract the information from the response	A non-JSON decoder might misinterpret the structure and content of the JSON response, leading to corrupted data. This could result in incorrect values being extracted, or the data might be entirely unusable. Misinterpreted data can lead to inaccurate clinical assessments and inappropriate clinical decisions	- The client system receives the JSON response from the device - The IT professional initiates the process to decode and extract data from the response - Instead of using a JSON decoder, a different type of decoder (e.g., XML, YAML, or binary decoder) is incorrectly selected to process the response - The incorrect decoder fails to properly interpret the JSON structure, leading to errors such as corrupted data, incomplete data extraction, or extraction of incorrect values - The incorrectly decoded data is stored in the database or integrated into the clinical workflow	Incorrectly decoded and stored data leads to errors in the clinical workflow or patient records	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	Comprehensive information on the proper procedures for decoding JSON responses provided in the IFU, error handling
UR-023	Process and store JSON	ITP	Critical	New data overwriting existing data due to incorrect handling of unique identifiers or version control	Loss of historical data, which can be crucial for longitudinal patient studies and follow-up care	- The client system receives the JSON response from the device - The IT professional initiates the process to handle and store the received data - Due to a programming error or oversight, the system incorrectly interprets or mishandles the unique identifiers or version control information - The system erroneously concludes that the new data should overwrite existing data, based on the mishandled identifiers or version control - The system proceeds to store the new data, potentially overwriting or modifying existing data in the database - The storage operation completes, with the new data potentially replacing or modifying existing records	Existing data may be inadvertently overwritten or modified, leading to data loss or corruption	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data loss, corruption	3	4	12	Implementation of REST protocol to inherently avoid bad practices in programming such as data re-writing (every request is independent and cannot be edited), error handling, provide technical support for troubleshooting
UR-024	Process and store JSON	ITP	Critical	User failing to implement adequate security measures to protect stored data	Unauthorized access to or breaches of sensitive patient data can lead to privacy violations and potential legal consequences	- The client system receives JSON data from the device and proceeds to process and store it in the database - The user fails to implement appropriate security measures to safeguard the stored data - Due to the lack of security measures, the stored data becomes vulnerable to various security threats such as unauthorized access, data breaches, or malicious attacks - The compromised data may be stolen, tampered with, or deleted by unauthorized parties	The stored data is exposed to risks of unauthorized access, manipulation, or theft due to inadequate security measures	Data breaches, security violations, data manipulation, loss of trust, regulatory and legal consequences	3	4	12	Implementation of data encryption, implementation of robust authentication mechanisms such as OAuth or JWT to ensure that only authorized users can access the device, monitoring of security threats, information about cybersecurity included in the IFU
UR-025	Process and store JSON	ITP	Critical	Storing data in inconsistent formats that are not compatible with existing systems	Difficulties in data integration, retrieval, and analysis, compromising clinical workflow efficiency	- The client system receives JSON data from the device and initiates the process to store it in the database - Due to a programming error or oversight, the system stores the data in an inconsistent format that is not compatible with existing systems or data models - The system completes the storage operation, resulting in the inconsistent format data being saved in the database - When attempting to retrieve the stored data, other systems or processes encounter issues due to the inconsistent format - The inconsistent format data cannot be properly interpreted or utilized by existing systems or processes, leading to compatibility problems - Other systems or processes may misinterpret the inconsistent format data, leading to errors, inaccuracies, or disruptions in data processing	The stored data is in inconsistent formats that are not compatible with existing systems or data models, leading to potential data misinterpretation or processing errors	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	The endpoints of the device follow HL7's FHIR interoperability standard, information about data formats is provided in the IFU, provide technical support for troubleshooting
UR-026	Build report	ITP	Critical	User might misunderstand the data format or the meaning of specific data points generated by the device	Incorrect representation of data in the UI, leading to misinformed clinical decisions	- The IT professional retrieves data from the device to build a report - The IT professional may misunderstand certain aspects of the data, such as its structure, semantics, or context - Based on the interpreted data, the IT professional proceeds to build the report - The misinterpretation of data format or meaning may lead to inaccuracies, errors, or omissions in the report content - The report is completed and may be shared with healthcare practitioners	The IT professional misunderstands the data format or the meaning of specific data points, leading to inaccuracies or errors in the generated report	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden	3	4	12	The endpoints of the device follow HL7's FHIR interoperability standard, comprehensive information about values returned by the device provided in the IFU, Swagger documentation
UR-027	Build report	ITP	Critical	Data might be displayed inconsistently, such as varying units of measurement, inconsistent decimal places	Confusion or misinterpretation by IT professionals, potentially resulting in clinical errors when data are shown to HCP	- The IT professional retrieves data from the device to build a report - Due to oversight or programming error, the data is formatted inconsistently, such as using varying units of measurement or inconsistent decimal places - The inconsistent data is used to generate the report - The report displays the data inconsistently, with varying units of measurement or inconsistent decimal places - Healthcare practitioners may find it challenging to interpret the inconsistently displayed data accurately	The report displays data inconsistently, leading to potential misinterpretation or misapplication by healthcare practitioners	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden	3	3	9	The endpoints of the device follow HL7's FHIR interoperability standard, comprehensive information about values returned by the device provided in the IFU, Swagger documentation
UR-028	Read IFU	ITP	Critical	User may misunderstand the technical language or specific instructions provided in the IFU	Incorrect use of the device, leading to inaccurate data being generated, which could result in improper clinical decisions	- The user reads through the IFU document to understand the technical specifications, operating instructions, and safety precautions associated with the software - Due to complexity, ambiguity, or lack of clarity in the technical language or specific instructions, the user may misunderstand certain aspects of the IFU content - The user may inadvertently apply incorrect procedures, settings, or precautions based on the misunderstood information from the IFU	Misunderstanding the technical language or specific instructions provided in the IFU may lead to potential harm or errors in the use of the software	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	4	12	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), provide technical support
UR-029	Read IFU	ITP	Critical	User might skip through the IFU or skip sections, missing critical information about the device's operation or limitations	Incomplete understanding of device functionality, which may cause incorrect usage and potentially unreliable clinical data	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to time constraints, lack of attention, or overconfidence, the user may skim through or skip sections of the IFU without thoroughly reviewing all content - The user may miss critical information about the device's operation, maintenance requirements, safety precautions, or limitations - The user proceeds to execute procedures or interact with the software based on incomplete or misunderstood information from the IFU	Incomplete understanding of device operation or limitations, posing risks during device use	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	3	9	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, provide technical support
UR-030	Read IFU	ITP	Critical	User may not notice or fully comprehend safety warnings and precautions	Increased risk of harm to patients or practitioners due to ignored safety protocols, possibly leading to injury or incorrect clinical decisions	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to distractions, time constraints, or other factors, the user may not fully concentrate on reading the IFU content - As a result of lack of attention, the user may overlook or skim over safety warnings and precautions outlined in the IFU - The user proceeds to execute procedures or interact with the software or device without fully considering or understanding the safety warnings and precautions	User may not notice or fully comprehend safety warnings and precautions provided in the IFU, leading to potential harm or errors in device use	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	3	9	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, provide technical support
UR-031	Read IFU	ITP	Critical	User might not fully grasp the instructions if the IFU is not available in their preferred language or if the language used is too complex	Miscommunication of device usage steps, leading to errors in operation and data interpretation	- The user attempts to access the Instructions for Use (IFU) document provided with the software - If the IFU is not available in the user's preferred language, or if the language used is too complex for their comprehension level, the user may struggle to understand the instructions - As a result of language barriers or complexity, the user may only partially understand the instructions provided in the IFU - Despite incomplete understanding, the user proceeds to execute procedures or interact with the software based on their interpretation of the IFU instructions	Inadequate understanding of IFU instructions due to language barriers or complexity may lead to potential harm or errors in device use	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	4	12	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, translation of the IFU according to internal procedure (GP-022 Document translation), provide technical support
UR-032	Read IFU	ITP	Critical	User might misinterpret the device's intended use or its limitations, thinking it is a final diagnosis report instead of a support to diagnosis for the HCP	The IT staff builds a user interface in which the device results do not reach the HCP but are returned to the patient as a definitive diagnosis	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to lack of attention, incomplete comprehension, or preconceived notions, the user may misinterpret the device's intended use or its limitations - The user may mistakenly believe that the device provides a final diagnosis rather than serving as a support tool for healthcare practitioners (HCPs) during the diagnostic proces - Based on the misinterpretation, the user may assume that the device's outputs are conclusive diagnostic reports without further validation or confirmation by HCPs - Diagnosti report sent to patient instead of HCP	Misinterpreting the device's intended use or its limitations as final diagnostic reports may lead to potential harm or errors in clinical decision-making	Misdiagnosis, inappropriate treatments, legal concerns	3	4	12	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, information about intended use and users provided in the first section of the IFU, provide technical support
UR-033	Read IFU	ITP	Critical	User may not adhere to the recommended sequence of operations provided in the IFU	Incorrect use of the device, leading to potential errors in image analysis and data generation	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to lack of attention, oversight, or misunderstanding, the user may misinterpret or overlook the recommended sequence of operations outlined in the IFU - By deviating from the recommended sequence of operations, the user may fail to properly integrate the device	Non-adherence to the recommended sequence of operations provided in the IFU may lead to potential harm or errors in device use	Misdiagnosis, compromised performance, user frustration	3	3	9	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, provide technical support

Details of summative evaluation testing

Overall study approach

The study will be conducted remotely during the month of June 2024, and consists of the simulated use of device by the selected IT professionals.

The study will be conducted during multiple sessions, meaning that the usability test will not be conducted simultaneously among all the selected participants.

Each participant will perform the test in a single section during which each participant will perform the tasks listed below, after reading the IFU:

• Authentication endpoint tests
  • Credential validation
  • Token expiry and refresh
• Diagnosis support endpoint tests
  • The simplest workflow
  • Beyond the boundaries of the standard operating flow
  • Missing or extraneous field in the request
  • Submission of multiple images for a more reliable diagnosis
  • Handling corrupted images
  • Excessive number of images
• Severity assessment endpoint tests
  • Evaluating the severity of skin conditions of different natures
  • Scoring system requiring a supporting questionnaire
  • Inappropriate scoring system for the condition
  • Scoring system unspecified or misspelled
  • Sending more than one image.

We will send to each participant instructions to execute the test, which include the description of the tasks to be performed together with the expected outcomes.

In order to gather feedback from the participants, we will send to the selected participants an evaluation form to be filled in at the end of the test. More details about the questions provided in the evaluation form are available in the section Evaluation form of this plan.

Participants or testers

The study will include participants who are IT professionals with the characteristics described below.

User profile

• Programming Languages: Given that the API follows REST principles, knowledge of any web development language such as JavaScript, Python, or Java could be essential. These languages are commonly used to interact with REST APIs due to their extensive support for HTTP request methods.
• Technologies:
  • RESTful API Standards: Users should be familiar with RESTful design principles and how to consume APIs using standard HTTP methods (GET, POST, PUT, DELETE).
  • JSON: Knowledge of data formats like JSON for handling API responses is crucial since REST APIs typically communicate in this format.
• Tools:
  • Development Environments: Familiarity with integrated development environments (IDEs) like Visual Studio Code, Eclipse, or others that facilitate API development and testing.
  • Version Control Systems: Understanding of version control systems like Git, which are essential for managing changes to the API's codebase, especially in team environments.
  • API Testing Tools: Proficiency with tools like Postman or Swagger could be beneficial These tools are used for testing API endpoints, understanding the API structure, and generating documentation.
• Security Practices: Knowledge of basic security practices for API integration, such as handling authentication (e.g., API keys), ensuring data encryption, and safeguarding against common vulnerabilities (SQL injection, cross-site scripting).
• Compliance and Standards: Familiarity with healthcare-related IT standards and regulations (e.g., FHIR, HIPAA) that might impact how the API is implemented and used, especially in terms of data handling and patient privacy.
• Experience: at least 5 years of experience as a programmer in projects that involve consuming HTTP APIs.
• Understand the outputs provided by the API.

Consideration about sample size

For the summative evaluation test, we have determined a sample size of 5 participants to be adequate for detecting most use errors. This decision is informed by industry guidelines, IEC TR 62366-2:2016, Annex K Usability test sample size, which discusses considerations for sample size in usability testing. According to Annex K, a smaller sample size can be sufficient when formative evaluations have already indicated a high level of task success and usability. Our formative evaluation results demonstrated successful completion of tasks by users, which supports the rationale that a sample size of 5 participants will effectively uncover the majority of potential use errors. More specifically, Annex K suggests that testing with 5 participants can uncover approximately 85% of usability problems. This number is expected to provide sufficient insights into the usability and safety of the device while ensuring that the evaluation is both effective and efficient. Since we define a small sample size, we will select the participants to the summative evaluation test very carefully in order to represent the diversity of the intended user group, especially in terms of years of experience. This approach ensures a comprehensive assessment while maintaining the practical constraints of testing.

Study device and materials

The device used in this study will be a production equivalent device (including the electronic IFU). The device is commercially representative and ensures that the performance replicates actual device performance.

The participants will be provided with:

• instructions document which includes description of the tasks to be performed (R-TF-012-016 Software usability test guide)
• access to the electronic IFU
• URL of the API
• set of images to be used to run specific test cases
• evaluation form.

Electronic IFU version

The electronic IFU that will be used during the test are available at the following link.

Version of the eIFU: 17edb45.

User tasks

All participants will complete all tasks listed below and described in the R-TF-012-016 Software usability test guide.

For each task we identified success criteria as shown below:

Use scenario	Task	Success criteria
Scenario 1: Authentication	Credential validation	Successful login when authenticating with correct credentials
Scenario 1: Authentication	Credential validation	Failed login attempts return a 401 Unauthorized status code with error message Invalid credentials when attempting authentication with incorrect credentials
Scenario 1: Authentication	Token expiry and refresh	API identifies expired tokens and returns a 401 Unauthorized status code with error message Invalid credentials
Scenario 1: Authentication	Token expiry and refresh	Successful refresh of token
Scenario 2: Diagnosis support	The simplest workflow (user sends an image to check patient's skin abnormality and stores the results)	API responds with a 200 OK status code for the request sent
Scenario 2: Diagnosis support	The simplest workflow (user sends an image to check patient's skin abnormality and stores the results)	API returns accurate diagnosis for the sent image following the FHIR interoperability standard
Scenario 2: Diagnosis support	The simplest workflow (user sends an image to check patient's skin abnormality and stores the results)	JSON successfully stored to disk
Scenario 2: Diagnosis support	Beyond the boundaries of the standard operating flow (user sends low quality image(s) and/or image(s) outside the dermatological domain to receive a diagnosis)	API responds with a 200 OK status code for all the requests sent
Scenario 2: Diagnosis support	Beyond the boundaries of the standard operating flow (user sends low quality image(s) and/or image(s) outside the dermatological domain to receive a diagnosis)	API indicates the low-quality image has not passed the quality check
Scenario 2: Diagnosis support	Beyond the boundaries of the standard operating flow (user sends low quality image(s) and/or image(s) outside the dermatological domain to receive a diagnosis)	Non-dermatological image does not result in false positive, the API returns the category Non dermatological
Scenario 2: Diagnosis support	Missing or extraneous field in the request	API fails in processing request with a 422 Unprocessable content status code
Scenario 2: Diagnosis support	Missing or extraneous field in the request	API returns clear error messages for each of the 3 body requests: for body request 1, API returns Field required; for body request 2, API returns Extra inputs are not permitted; for body request 3, API returns Extra inputs are not permitted
Scenario 2: Diagnosis support	Submission of multiple images for a more reliable diagnosis	API responds with a 200 OK status code for the request sent
Scenario 2: Diagnosis support	Submission of multiple images for a more reliable diagnosis	The imagingStudySeries key of the JSON contains 2 JSON object, one for each image sent. Each object contains the results of the analysis following the FHIR interoperability standard
Scenario 2: Diagnosis support	Submission of multiple images for a more reliable diagnosis	The conclusions key of the JSON contains a differential diagnosis resulting from aggregating the results of the diagnosis for each image (endpoint follows the FHIR interoperability standard)
Scenario 2: Diagnosis support	Handling corrupted images	API responds with a 422 Unprocessable content staus code with error message Value error, only base64 data is allowed
Scenario 2: Diagnosis support	Excessive number of images	API responds with a 422 Unprocessable content staus code with error message List should have at most 5 items after validation, not 6
Scenario 3: Severity assessment	Evaluating the severity of skin conditions of different natures	API responds with a 200 OK status code for the request sent
Scenario 3: Severity assessment	Evaluating the severity of skin conditions of different natures	Calculated score for each scoring system specified in the request is returned
Scenario 3: Severity assessment	Evaluating the severity of skin conditions of different natures	For skin conditions where lesions can be counted, JSON contained in the response includes the processed image
Scenario 3: Severity assessment	Scoring system requiring a supporting questionnaire	API responds with a 200 OK status code for the request sent
Scenario 3: Severity assessment	Scoring system requiring a supporting questionnaire	If mandatory questionnaire is not included in the request, API returns 500 Internal server error
Scenario 3: Severity assessment	Inappropriate scoring system for the condition	API responds with a 200 OK status code for the request sent
Scenario 3: Severity assessment	Inappropriate scoring system for the condition	JSON contained in the response includes the processed image with the wrong scoring system applied
Scenario 3: Severity assessment	Scoring system unspecified or misspelled	When sending an empty list, API responds with 500 Internal server error
Scenario 3: Severity assessment	Scoring system unspecified or misspelled	When sending a misspelled scoring system, API returns 422 Unprocessable content status code with error message including a list of exact names of all available scoring systems in the device
Scenario 3: Severity assessment	Sending more than one image	API returns 422 Unprocessable content status code with error message Input should be a valid dictionary or object to extract fields from

We also define success criteria for the IFU provided to participants to evaluate their completeness and understandability:

• At least 80% of participants are able to successfully complete the tasks using the IFU
• At least 80% of participants answered with Strongly agree or Agree to the question about rating the IFU in terms of completeness and understandability in the evaluation form.

As we defined that users will not receive training on the device prior to use, this study will only include untrained participants to reflect the conditions of real use of the device.

Data coding

The data to be collected are as follows:

• Objective Data: this data consists of participants' results of the implementation of the tasks defined for the test.
• Subjective assessment of use: this data consists of participants' subjective feedback on the root cause of potentially safety-related use errors and use difficulties, and opinions on all tasks for any residual risk, as well as the adequacy of instructions.

The following coding system will be used for critical simulated use and knowledge task data collection and analysis:

• Success (OK): This coding describes a situation in which the participant correctly performs a task according to the steps in the R-TF-012-016 Software usability test guide and in the IFU.
• Use error (UE): This coding describes a situation in which the participant performs a task incorrectly (i.e., not in accordance with the intended manner of use), does not complete a necessary task.
• Use difficulty (UD): This coding describes a situation in which the participant struggles to some extent while completing a task (e.g., confusion, taking longer than expected), but eventually can complete the task.

Data analysis

The collected data will be analyzed using the definitions of success criteria of each task presented in the table above, to determine if the study objectives have been met.

During this analysis, both objective data and subjective data will be analyses to draw conclusions on the usability of the device that will be documented in T-TF-12-015 Summative evaluation report.

Description and analysis of all use errors, use difficulties that could cause serious harm, root causes of the problems, and implications for additional risk elimination or reduction will be used to determine if there are patterns of use events that pose unacceptable risk to the patient or user.

Evaluation form

In order to collect feedback from participants, we will provide them with an evaluation form to be filled in after the execution of the test.

The questions contained in the evaluation form are the following:

1. What is your professional role? (e.g., Software Developer, Tester, DevOps)

2. How many years of experience do you have in that role?

3. When authenticating with the API, how easy was it to understand and use the authentication process?

• Very difficult
• Difficult
• Neutral
• Easy
• Very easy

4. Were the error messages clear and helpful when the login attempts failed?

• Not clear at all
• Somewhat unclear
• Neutral
• Clear
• Very clear

5. How secure do you consider the authentication process to be?

• Not secure
• Somewhat insecure
• Neutral
• Secure
• Very secure

6. How effective was the token management, including expiration and refresh?

• Very ineffective
• Ineffective
• Neutral
• Effective
• Very effective

7. Did you experience any issues in obtaining the access token?

• Yes
• No

If yes, please describe the issues encountered

8. When you uploaded images labelled with the corresponding skin disease, did the API return the correct preliminary differential diagnosis (e.g.Melanoma for images labelled as such)?

• Strongly disagree
• Disagree
• Neutral
• Agree
• Strongly agree

9. How well did the API handle images of different quality?

• Very poorly
• Poorly
• Neutral
• Well
• Very well

10. How would you rate the response time of the /diagnosis-support endpoint?

• Very slow
• Slow
• Average
• Fast
• Very fast

11. How well did the API manage multiple concurrent requests?

• Very poorly
• Poorly
• Neutral
• Well
• Very well

12. How secure do you consider the data transmission process to be?

• Not secure
• Somewhat insecure
• Neutral
• Secure
• Very secure

13. Do you agree the Instructions for Use are clear, comprehensive, and easy to understand?

• Strongly disagree
• Disagree
• Neutral
• Agree
• Strongly agree

14. Do you agree that the provided documentation was sufficient for using the API effectively?

• Strongly disagree
• Disagree
• Neutral
• Agree
• Strongly agree

15. Do you agree that the safety information included in the Instructions for Use are clear, comprehensive, and easy to understand?

• Strongly disagree
• Disagree
• Neutral
• Agree
• Strongly agree

16. Were the IFU always accessible during the test?

• Yes
• No

If not, please give more details about issues encountered

17. Overall, how would you rate the clarity of error messages?

• Not clear at all
• Somewhat unclear
• Neutral
• Clear
• Very clear

18. Overall, how satisfied are you with the medical device's API connectivity?

• Very unsatisfied
• Unsatisfied
• Neutral
• Satisfied
• Very satisfied

19. Please describe any difficulties encountered during the execution of the test (if any).

20. Please provide any recommendations for improvements or other comments.

Record signature meaning

• Author: JD-004
• Review: JD-003
• Approval: JD-005