R-TF-012-015 Summative evaluation report 2024_001

Objective

This summative evaluation study aims at validating through objective and empirical evidence that our software as medical device can be used in an easy and safe manner. More specifically, the summative evaluation test aims to evaluate the risks associated with use errors to confirm they are properly addressed and therefore to confirm the safe use of the device.

This study is planned and executed according to the requirements set out in IEC 62366-1:2015 (Medical devices - Part 1: Application of usability engineering to medical devices).

Abbreviations and definitions

• Instruction for use: following the Medical Device Regulation, the term `instructions for use refers to the information provided by the manufacturer to inform the user of a device's intended purpose and proper use and of any precautions to be taken
• Intended user: Users are defined in IEC 62366-1:2015 as persons interacting with (i.e. operating or handling) the medical device.
• Summative user testing: It (quantitative testing) provides an indirect assessment of the usability of the design. A group of testers are given a task, and based on their performance, the usability quotients of the design elements are measured.
• Tester: a person who has been asked to perform the tasks described in the present protocol.
• eIFU : Electronic instruction for use
• IFU: Instruction for use
• URRA: Use related risk analysis

Device description

Device Identification

	Information
Device name	Legit.Health Plus (hereinafter, the device)
Model and type	NA
Version	1.1.0.0
Basic UDI-DI	8437025550LegitCADx6X
Certificate number (if available)	MDR 792790
EMDN code(s)	Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software)
GMDN code	65975
Class	Class IIb
Classification rule	Rule 11
Novel product (True/False)	FALSE
Novel related clinical procedure (True/False)	FALSE
SRN	ES-MF-000025345

Intended use

The device is a computational software-only medical device intended to support health care providers in the assessment of skin structures, enhancing efficiency and accuracy of care delivery, by providing:

• quantification of intensity, count, extent of visible clinical signs
• interpretative distribution representation of possible International Classification of Diseases (ICD) categories.

Quantification of intensity, count and extent of visible clinical signs

The device provides quantifiable data on the intensity, count and extent of clinical signs such as erythema, desquamation, and induration, among others; including, but not limited to:

• erythema,
• desquamation,
• induration,
• crusting,
• xerosis (dryness),
• swelling (oedema),
• oozing,
• excoriation,
• lichenification,
• exudation,
• wound depth,
• wound border,
• undermining,
• hair loss,
• necrotic tissue,
• granulation tissue,
• epithelialization,
• nodule,
• papule
• pustule,
• cyst,
• comedone,
• abscess,
• draining tunnel,
• inflammatory lesion,
• exposed wound, bone and/or adjacent tissues,
• slough or biofilm,
• maceration,
• external material over the lesion,
• hypopigmentation or depigmentation,
• hyperpigmentation,
• scar,
• ictericia

Image-based recognition of visible ICD categories

The device is intended to provide an interpretative distribution representation of possible International Classification of Diseases (ICD) categories that might be represented in the pixels content of the image.

Device description

The device is a computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organisations to gather data and improve their workflows.

The generated data is intended to aid healthcare practitioners and organizations in their clinical decision-making process, thus enhancing the efficiency and accuracy of care delivery.

The device should never be used to confirm a clinical diagnosis. On the contrary, its result is one element of the overall clinical assessment. Indeed, the device is designed to be used when a healthcare practitioner chooses to obtain additional information to consider a decision.

Intended medical indication

The device is indicated for use on images of visible skin structure abnormalities to support the assessment of all diseases of the skin incorporating conditions affecting the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral and genital), the dermis, the cutaneous vasculature and the subcutaneous tissue (subcutis).

Intended patient population

The device is intended for use on images of skin from patients presenting visible skin structure abnormalities, across all age groups, skin types, and demographics.

Intended user

The medical device is intended for use by healthcare providers to aid in the assessment of skin structures.

User qualification and competencies

In this section we specificy the specific qualifications and competencies needed for users of the device, to properly use the device, provided that they already belong to their professional category. In other words, when describing the qualifications of HCPs, it is assumed that healthcare professionals (HCPs) already have the qualifications and competencies native to their profession.

Healthcare professionals

No official qualifications are needes, but it is advisable if HCPs have some competencies:

• Knowledge on how to take images with smartphones.

IT professionals

IT professionals are responsible for the integration of the medical device into the healthcare organisation's system.

No specific official qualifications are needed, but it is advisable that IT professionals using the device have the following competencies:

• Basic knowledge of FHIR
• Understanding of the output of the device.

Use environment

The device is intended to be used in the setting of healthcare organisations and their IT departments, which commonly are situated inside hospitals or other clinical facilities.

The device is intended to be integrated into the healthcare organisation's system by IT professionals.

Operating principle

The device is computational medical tool leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures.

Body structures

The device is intended to use on the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral and genital), the dermis, the cutaneous vasculature and the subcutaneous tissue (subcutis).

In fact, the device is intended to use on visible skin structures. As such, it can only quantify clinical signs that are visible, and distribute the probabilities across ICD categories that are visible.

Risk management

We conducted a task analysis on the use of the device which led to a comprehensive use-related risk analysis and list of potential use errors. This process involved analyzing known use problems with similar devices and/or previous versions of the device, identifying the user interface characteristics related to safety, identifying potential use errors, and identifying known and foreseeable hazards and hazardous situations.

The use related risk analysis is shown below:

Risk ID	Use Step or Task Statement	User Group	Task Classification	Use Error (Potential Failure Modes)	Potential Result of Use Error or Hazard (Potential Failure Modes)	Foreseeable Sequence of Events (Potential Failure Modes)	Hazardous Situation (Potential Failure Modes)	Potential Harm (Potential Failure Modes)	Severity (Risk Pre-Mitigation)	Probability (Risk Pre-Mitigation)	Risk Priority Number (Risk Pre-Mitigation)	Risk control measures (Risk control)
UR-001	Get access token	ITP	Critical	User may input incorrect username, password, or other required authentication details	Repeated failed login attempts could lead to account lockout or delays in data access, impacting timely patient care	- The IT professional attempts to access the API to integrate it into the healthcare organization's system - The IT professional inputs the required authentication details - The IT professional inputs incorrect authentication details (e.g., mistyped username or password)	The user is unable to obtain an access token due to the incorrect input of authentication details, leading to a failure to authenticate and access the API	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device	3	4	12	Clear error messages, provide technical support to assist users in troubleshooting and resolving authentication issues
UR-002	Get access token	ITP	Critical	User may enter an incorrect URL for the login endpoint	The device may fail to authenticate, preventing access to the protected endpoints and delaying clinical decision-making	- The IT professional attempts to access the API to integrate it into the healthcare organization's system - The IT professional configures the system with the URL for the login endpoint to obtain an access token - The IT professional enters an incorrect URL for the login endpoint - The system attempts to contact the incorrect URL, resulting in a failed request	The system fails to obtain an access token due to the incorrect URL, preventing authentication and access to the API	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden	3	3	9	Clear error messages, provide technical support to assist users in troubleshooting, provide information in the IFU
UR-003	Get access token	ITP	Critical	User may use poor or unstable internet connection	Inability to retrieve or validate the access token, causing delays in accessing the device’s features	- The IT professional attempts to access the API to integrate it into the healthcare organization's system - The system sends a request to the login endpoint to obtain an access token - The internet connection is poor or unstable, leading to intermittent connectivity or a complete loss of connection - The request to the login endpoint fails or times out due to the unstable connection - The system or the user may attempt to retry the request multiple times, experiencing repeated failures	The system fails to obtain an access token due to the unstable connection, preventing authentication and access to the API	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Robust error handlings, state-of-the-art techniques of security and software availability, provide information in the IFU about minimum IT requirements, provide technical support to assist users in troubleshooting
UR-004	Build JSON with data	ITP	Critical	User might format the data incorrectly, such as wrong JSON structure or incorrect field names	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data. - The IT professional enters the data into the JSON object - The IT professional makes an error, resulting in incorrect JSON structure (e.g., missing brackets, commas) or incorrect field names (e.g., misspelled or mismatched with the API specification) - The incorrectly formatted JSON is submitted to the API endpoint - The API detects the formatting error and responds with an error message	The system fails to process the incorrectly formatted JSON, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Clear error messages, comprehensive information in the IFU that details the required JSON structure, field names, examples of correctly formatted data, provide technical support for troubleshooting
UR-005	Build JSON with data	ITP	Critical	User might omit mandatory fields required for the request	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data - The IT professional enters the data into the JSON object - The IT professional omits one or more mandatory fields required by the API specification - The incomplete JSON is submitted to the API endpoint - The API detects the missing mandatory fields and responds with an error message	The system fails to process the incomplete JSON, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Clear error messages, comprehensive information in the IFU that details the required JSON structure, field names, examples of correctly formatted data and mandatory fields, provide technical support for troubleshooting
UR-006	Build JSON with data	ITP	Critical	User may enter data in incorrect types (e.g., string instead of integer)	If the device is unable to parse the incorrect data type to the data type defined by the schema of the request, then the user's request will be rejected. If the error message is not clear enough, the user may not be able to send a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data - The IT professional enters the data into the JSON object - The IT professional enters data in incorrect types, such as a string instead of an integer or vice versa - The incorrectly typed JSON is submitted to the API endpoint - The API detects the type error and responds with an error message	The system fails to process the JSON due to incorrect data types, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	3	9	Clear error messages, comprehensive information in the IFU that details the required JSON structure, field names, examples of correctly formatted data, correct data types and mandatory fields, provide technical support for troubleshooting
UR-007	Build JSON with data	ITP	Critical	User may send images in an unsupported format (other than Base64) or incorrectly encoded in Base64	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data, including images - The IT professional encodes the images to be included in the JSON object - The IT professional either uses an unsupported format (other than Base64) or incorrectly encodes the images in Base64 - The incorrectly formatted JSON, containing the improperly encoded images, is submitted to the API endpoint - The API detects the unsupported or incorrect image encoding and responds with an error message	The system fails to process the JSON due to unsupported image formats or incorrect Base64 encoding, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Validation checks, clear error messages, information in the IFU, provide technical support for troubleshooting
UR-008	Build JSON with data	ITP	Critical	User may include extraneous data not required for the request	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data - The IT professional enters the data into the JSON object - The IT professional includes extraneous data that is not required or specified by the API documentation - The JSON object, containing the extraneous data, is submitted to the API endpoint - The API processes the request and responds with an error message	The system may fail to process the JSON correctly	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	3	9	Clear error messages, comprehensive information in the IFU, provide technical support for troubleshooting
UR-009	Send JSON to device	ITP	Critical	User might send the request to the wrong endpoint or to a non-existent URL	Failed to connect to the device, delaying the decision-making process and potentially impacting patient care	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional incorrectly enters the URL or endpoint - The JSON request is sent - The device responds with an error	The system fails to process the JSON due to sending the request to an incorrect or non-existent endpoint, leading to a failure in data transmission	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Clear error messages, comprehensive information in the IFU that details the correct endpoints, provide technical support for troubleshooting
UR-010	Send JSON to device	ITP	Critical	User may use the wrong HTTP method (e.g. GET instead of POST)	Failed to connect to the device, delaying the decision-making process and potentially impacting patient care	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional selects the HTTP method to be used for the request (e.g., GET, POST) - The IT professional mistakenly chooses the wrong HTTP method, such as selecting GET instead of the required POST - The JSON request is sent to the device using the incorrectly chosen HTTP method - The device responds with an error message	The system fails to process the JSON request due to using an incorrect HTTP method, leading to a rejection of the request by the device	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	3	9	Clear error messages, comprehensive information in the IFU that details the supported HTTP methods, provide technical support for troubleshooting
UR-011	Send JSON to device	ITP	Critical	User may not include the access token in the request	Authentication will fail, leading to interruptions in workflow and potential delays in the clinical assessment process	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional obtains the access token required for authentication - The IT professional constructs the JSON request, intending to include the required access token - The IT professional mistakenly omits the access token from the request - The JSON request lacking the access token is sent to the device - The device detects the missing access token and responds with an error message	The system fails to authenticate the user due to the absence of the access token, leading to a rejection of the request by the device	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data security risk	3	4	12	Clear error messages, inclusion of the requirement for including access token in all requests in the IFU, provide technical support for troubleshooting
UR-012	Send JSON to device	ITP	Critical	User may include the access token in the body of the request instead of headers	Authentication will fail, leading to interruptions in workflow and potential delays in the clinical assessment process	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional obtains the access token required for authentication - The IT professional constructs the JSON request and mistakenly includes the access token in the body of the request payload instead of the request headers - The JSON request with the access token in the body is sent to the device - The device receives the request but does not recognize the access token in the body as it is expecting it in the headers	The device may fail to authenticate the request due to the access token being included in the wrong part of the request, leading to potential unauthorized access or data security risks.	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data security risk	3	4	12	Clear error messages, inclusion of correct method of including access token in all requests in the IFU, provide technical support for troubleshooting
UR-013	Send JSON to device	ITP	Critical	User might attempt to send an expired token	Authentication will fail, leading to interruptions in workflow and potential delays in the clinical assessment process	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional obtains the access token required for authentication - The access token obtained by the IT professional expires due to reaching its predefined expiration time - The IT professional attempts to use the expired access token to authenticate the JSON request - The JSON request with the expired access token is sent - The device receives the request with the expired token and responds with an error message	The device may fail to authenticate the request due to the access token being included in the wrong part of the request, leading to potential unauthorized access or data security risks.	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data security risk	3	3	9	Clear error messages, instructions on how to obtaining new token in the IFU, provide technical support for troubleshooting
UR-014	Send JSON to device	ITP	Critical	User might send the exact same request multiple times	Redundant processing, wasting resources and possibly causing confusion in clinical assessments	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional sends the JSON request - Due to network instability, or misunderstanding, the IT professional resends the same JSON request multiple times - The device receives multiple identical requests from the same user within a short period - The device processes each request as a new submission	The device processes multiple identical requests, potentially leading to duplicate actions or data entries	Misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data duplication	3	3	9	Clear error messages, provide technical support for troubleshooting
UR-015	Receive JSON from device	ITP	Critical	Network issues, device downtime, or incorrect handling of the request	Delays in obtaining clinical data, which can impact the timeliness and accuracy of clinical assessments	- The IT professional or system initiates a request to receive JSON data from the device - During the communication, network issues (such as instability, high latency, or disconnection) or device downtime occurs or the IT professional incorrectly handles the request, leading to improper processing or failure to receive the expected data - The device either fails to respond, sends a corrupted response, or the client misinterprets the response due to handling errors - The client system either waits indefinitely, retries the request, or processes an incomplete/corrupt JSON response	The system does not receive the necessary JSON data correctly or in a timely manner, leading to a failure in processing the required information	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, incomplete data processing	3	4	12	State-of-the-art techniques of security and software availability, clear error messages, implementation of recovery strategy, monitoring of device downtime, provide technical support for troubleshooting, information on how to handle request provided in the IFU
UR-016	Receive JSON from device	ITP	Critical	The response might be incomplete or corrupted due to data transmission issues	Incomplete or corrupted data can lead to incorrect clinical assessments	- The IT professional or system initiates a request to receive JSON data from the device - The device processes the request and sends the JSON data back to the client system - During the data transmission, network instability, interference, or other issues cause the data to be incomplete or corrupted - The client system receives the JSON response, but the data is either incomplete, or corrupted - The client system attempts to process the received JSON data, leading to potential errors or failures	The system processes incomplete or corrupted data, leading to potential errors in the application's functioning and data interpretation	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	State-of-the-art techniques of security and software availability, clear error messages, provide technical support for troubleshooting
UR-017	Receive JSON from device	ITP	Critical	Misinterpretation of the HTTP status code of the response	If a user misinterprets a successful status code (e.g., 200 OK) as an error, they might not process the received data. Conversely, interpreting an error status code (e.g., 404 Not Found, 500 Internal device Error) as successful could lead to the belief that the data has been processed correctly, when it hasn't. This delay or failure in data processing can impact timely clinical decisions.	- The IT professional initiates a request to receive JSON data from the device - The device processes the request and sends back a response with an HTTP status code and JSON data - The client system receives the response from the device, including the HTTP status code and JSON payload - The client system or IT professional misinterprets the HTTP status code (e.g., interpreting a 500 Internal Server Error as a 200 OK, or vice versa) - Based on the misinterpreted status code, the client system or IT professional takes incorrect actions, such as processing incomplete data or failing to retry a failed request	The system proceeds with incorrect assumptions about the success or failure of the request, leading to potential errors in data handling and application behavior	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	Information provided in the IFU, clear error messages, provide technical support for troubleshooting
UR-018	Receive JSON from device	ITP	Critical	User may accidentally delete or modify the request during the process	Clinical information generated by the device is lost. The user would have to send the same request, leading to a delay in the decision-making process	- The IT professional or system initiates a request to receive JSON data from the device - The client system prepares the request for data transmission - The IT professional accidentally deletes or modifies the request - The client system attempts to handle the response, which may be an error message or an unexpected result due to the incorrect request	The system does not receive the expected JSON data due to the accidental modification or deletion of the request, leading to potential errors in the application's functioning and data interpretation	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	Clear error messages, provide technical support for troubleshooting
UR-019	Receive JSON from device	ITP	Critical	User might incorrectly map the response data to the clinical workflow or patient records	Incorrect data mapping can result in misidentification or misassociation of clinical results with the wrong patient, leading to potential harm	- The IT professional or system initiates a request to receive JSON data from the device - The device processes the request and sends back a JSON response containing the clinical data - The client system receives the JSON response from the device - The IT professional processes the JSON response and maps the data to the clinical workflow or patient records - During this process, the user incorrectly maps the data (e.g., assigns data to the wrong patient record, uses incorrect data fields, or mismatches data types) - The mapped data is used in the clinical workflow or patient records, impacting clinical evaluations or decisions	Incorrect mapping of response data leads to erroneous data being integrated into the clinical workflow or patient records	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	Information provided in the IFU, error handling
UR-020	Process and store JSON	ITP	Critical	User may misinterpret the format or structure of the response data, leading to incorrect data extraction	Inaccurate or incomplete clinical data being stored, potentially leading to incorrect clinical decisions	- The client system receives the JSON response from the device - The IT professional begins processing the JSON response to extract relevant data - The user misinterprets the format or structure of the JSON data (e.g., misunderstanding the hierarchy, incorrect field names, or data types) - Due to the misinterpretation, the system extracts data incorrectly (e.g., extracting wrong values, missing important data, or misinterpreting data fields) - The incorrectly extracted data is stored in the database or integrated into the clinical workflow	Incorrectly extracted and stored data leads to errors in the clinical workflow or patient records	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	Detailed information on the JSON response format and structure to guide users in correctly interpreting the data provided in the IFU, clear error messages, provide technical support for troubleshooting
UR-021	Process and store JSON	ITP	Critical	Issues with the storage system, such as database failures or insufficient storage capacity	Loss of critical clinical data or inability to retrieve stored data when needed can delay patient care	- The client system receives the JSON response from the device - The client system processes the JSON data to prepare it for storage - The system attempts to store the processed data in the database - The storage system encounters issues such as database failures, insufficient storage capacity, or connection problems - The system may fail to store the data correctly, partially store the data, or not store the data at all - Based on the system's handling of the error, the user may take incorrect actions, such as assuming the data is stored correctly	Data loss or corruption due to issues with the storage system, leading to incomplete or missing clinical data	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	The deployment of the medical devices uses elastic demand design. The medical device makes constant backups. State-of-the-art techniques of security and software availability. Due to the inherent features of the REST protocol, when a user send a request and the device is down, the device returns a specific code informing of the state of the device, including downtime. This means that the user will be automatically aware of downtime, as well as any other states. Clear error messages, provide technical support for troubleshooting
UR-022	Process and store JSON	ITP	Critical	A decoder other than JSON is used to extract the information from the response	A non-JSON decoder might misinterpret the structure and content of the JSON response, leading to corrupted data. This could result in incorrect values being extracted, or the data might be entirely unusable. Misinterpreted data can lead to inaccurate clinical assessments and inappropriate clinical decisions	- The client system receives the JSON response from the device - The IT professional initiates the process to decode and extract data from the response - Instead of using a JSON decoder, a different type of decoder (e.g., XML, YAML, or binary decoder) is incorrectly selected to process the response - The incorrect decoder fails to properly interpret the JSON structure, leading to errors such as corrupted data, incomplete data extraction, or extraction of incorrect values - The incorrectly decoded data is stored in the database or integrated into the clinical workflow	Incorrectly decoded and stored data leads to errors in the clinical workflow or patient records	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	Information in the IFU, error handling
UR-023	Process and store JSON	ITP	Critical	New data overwriting existing data due to incorrect handling of unique identifiers or version control	Loss of historical data, which can be crucial for longitudinal patient studies and follow-up care	- The client system receives the JSON response from the device - The IT professional initiates the process to handle and store the received data - Due to a programming error or oversight, the system incorrectly interprets or mishandles the unique identifiers or version control information - The system erroneously concludes that the new data should overwrite existing data, based on the mishandled identifiers or version control - The system proceeds to store the new data, potentially overwriting or modifying existing data in the database - The storage operation completes, with the new data potentially replacing or modifying existing records	Existing data may be inadvertently overwritten or modified, leading to data loss or corruption	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data loss, corruption	3	4	12	Implementation of REST protocol to inherently avoid bad practices in programming such as data re-writing (every request is independent and cannot be edited), error handling, provide technical support for troubleshooting
UR-024	Process and store JSON	ITP	Critical	User failing to implement adequate security measures to protect stored data	Unauthorized access to or breaches of sensitive patient data can lead to privacy violations and potential legal consequences	- The client system receives JSON data from the device and proceeds to process and store it in the database - The user fails to implement appropriate security measures to safeguard the stored data - Due to the lack of security measures, the stored data becomes vulnerable to various security threats such as unauthorized access, data breaches, or malicious attacks - The compromised data may be stolen, tampered with, or deleted by unauthorized parties	The stored data is exposed to risks of unauthorized access, manipulation, or theft due to inadequate security measures	Data breaches, security violations, data manipulation, loss of trust, regulatory and legal consequences	3	4	12	Implementation of data encryption, implementation of robust authentication mechanisms such as OAuth or JWT to ensure that only authorized users can access the device, monitoring of security threats, information about cybersecurity included in the IFU
UR-025	Process and store JSON	ITP	Critical	Storing data in inconsistent formats that are not compatible with existing systems	Difficulties in data integration, retrieval, and analysis, compromising clinical workflow efficiency	- The client system receives JSON data from the device and initiates the process to store it in the database - Due to a programming error or oversight, the system stores the data in an inconsistent format that is not compatible with existing systems or data models - The system completes the storage operation, resulting in the inconsistent format data being saved in the database - When attempting to retrieve the stored data, other systems or processes encounter issues due to the inconsistent format - The inconsistent format data cannot be properly interpreted or utilized by existing systems or processes, leading to compatibility problems - Other systems or processes may misinterpret the inconsistent format data, leading to errors, inaccuracies, or disruptions in data processing	The stored data is in inconsistent formats that are not compatible with existing systems or data models, leading to potential data misinterpretation or processing errors	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	The endpoints of the device follow HL7's FHIR interoperability standard, information about data formats is provided in the IFU, provide technical support for troubleshooting
UR-026	Build report	ITP	Critical	User might misunderstand the data format or the meaning of specific data points generated by the device	Incorrect representation of data in the UI, leading to misinformed clinical decisions	- The IT professional retrieves data from the device to build a report - The IT professional may misunderstand certain aspects of the data, such as its structure, semantics, or context - Based on the interpreted data, the IT professional proceeds to build the report - The misinterpretation of data format or meaning may lead to inaccuracies, errors, or omissions in the report content - The report is completed and may be shared with healthcare practitioners	The IT professional misunderstands the data format or the meaning of specific data points, leading to inaccuracies or errors in the generated report	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden	3	4	12	The endpoints of the device follow HL7's FHIR interoperability standard, comprehensive information about values returned by the device provided in the IFU, Swagger documentation
UR-027	Build report	ITP	Critical	Data might be displayed inconsistently, such as varying units of measurement, inconsistent decimal places	Confusion or misinterpretation by IT professionals, potentially resulting in clinical errors when data are shown to HCP	- The IT professional retrieves data from the device to build a report - Due to oversight or programming error, the data is formatted inconsistently, such as using varying units of measurement or inconsistent decimal places - The inconsistent data is used to generate the report - The report displays the data inconsistently, with varying units of measurement or inconsistent decimal places - Healthcare practitioners may find it challenging to interpret the inconsistently displayed data accurately	The report displays data inconsistently, leading to potential misinterpretation or misapplication by healthcare practitioners	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden	3	3	9	The endpoints of the device follow HL7's FHIR interoperability standard, comprehensive information about values returned by the device provided in the IFU, Swagger documentation
UR-028	Read IFU	ITP	Critical	User may misunderstand the technical language or specific instructions provided in the IFU	Incorrect use of the device, leading to inaccurate data being generated, which could result in improper clinical decisions	- The user reads through the IFU document to understand the technical specifications, operating instructions, and safety precautions associated with the software - Due to complexity, ambiguity, or lack of clarity in the technical language or specific instructions, the user may misunderstand certain aspects of the IFU content - The user may inadvertently apply incorrect procedures, settings, or precautions based on the misunderstood information from the IFU	Misunderstanding the technical language or specific instructions provided in the IFU may lead to potential harm or errors in the use of the software	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	4	12	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), provide technical support
UR-029	Read IFU	ITP	Critical	User might skip through the IFU or skip sections, missing critical information about the device's operation or limitations	Incomplete understanding of device functionality, which may cause incorrect usage and potentially unreliable clinical data	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to time constraints, lack of attention, or overconfidence, the user may skim through or skip sections of the IFU without thoroughly reviewing all content - The user may miss critical information about the device's operation, maintenance requirements, safety precautions, or limitations - The user proceeds to execute procedures or interact with the software based on incomplete or misunderstood information from the IFU	Incomplete understanding of device operation or limitations, posing risks during device use	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	3	9	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, provide technical support
UR-030	Read IFU	ITP	Critical	User may not notice or fully comprehend safety warnings and precautions	Increased risk of harm to patients or practitioners due to ignored safety protocols, possibly leading to injury or incorrect clinical decisions	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to distractions, time constraints, or other factors, the user may not fully concentrate on reading the IFU content - As a result of lack of attention, the user may overlook or skim over safety warnings and precautions outlined in the IFU - The user proceeds to execute procedures or interact with the software or device without fully considering or understanding the safety warnings and precautions	User may not notice or fully comprehend safety warnings and precautions provided in the IFU, leading to potential harm or errors in device use	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	3	9	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, provide technical support
UR-031	Read IFU	ITP	Critical	User might not fully grasp the instructions if the IFU is not available in their preferred language or if the language used is too complex	Miscommunication of device usage steps, leading to errors in operation and data interpretation	- The user attempts to access the Instructions for Use (IFU) document provided with the software - If the IFU is not available in the user's preferred language, or if the language used is too complex for their comprehension level, the user may struggle to understand the instructions - As a result of language barriers or complexity, the user may only partially understand the instructions provided in the IFU - Despite incomplete understanding, the user proceeds to execute procedures or interact with the software based on their interpretation of the IFU instructions	Inadequate understanding of IFU instructions due to language barriers or complexity may lead to potential harm or errors in device use	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	4	12	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, translation of the IFU according to internal procedure (GP-022 Document translation), provide technical support
UR-032	Read IFU	ITP	Critical	User might misinterpret the device's intended use or its limitations, thinking it is a final diagnosis report instead of a support to diagnosis for the HCP	The IT staff builds a user interface in which the device results do not reach the HCP but are returned to the patient as a definitive diagnosis	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to lack of attention, incomplete comprehension, or preconceived notions, the user may misinterpret the device's intended use or its limitations - The user may mistakenly believe that the device provides a final diagnosis rather than serving as a support tool for healthcare practitioners (HCPs) during the diagnostic proces - Based on the misinterpretation, the user may assume that the device's outputs are conclusive diagnostic reports without further validation or confirmation by HCPs - Diagnosti report sent to patient instead of HCP	Misinterpreting the device's intended use or its limitations as final diagnostic reports may lead to potential harm or errors in clinical decision-making	Misdiagnosis, inappropriate treatments, legal concerns	3	4	12	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, information about intended use and users provided in the first section of the IFU, provide technical support
UR-033	Read IFU	ITP	Critical	User may not adhere to the recommended sequence of operations provided in the IFU	Incorrect use of the device, leading to potential errors in image analysis and data generation	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to lack of attention, oversight, or misunderstanding, the user may misinterpret or overlook the recommended sequence of operations outlined in the IFU - By deviating from the recommended sequence of operations, the user may fail to properly integrate the device	Non-adherence to the recommended sequence of operations provided in the IFU may lead to potential harm or errors in device use	Misdiagnosis, compromised performance, user frustration	3	3	9	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, provide technical support

Summative evaluation testing

Method

The study was conducted remotely during the month of June 2024 and first week of July 2024, and consisted of the simulated use of device by the selected IT professionals.

As mentioned above, the software version used during the summative evaluation test was 1.0.

The version of eIFU provided to participants is 17edb45, that is the latest version of the IFU for the device.

The study was conducted during multiple sessions, meaning that the usability test was not conducted simultaneously among all the selected participants.

Each participant performed the test in a single section lasting in average one hour and a half during which each participant performed the tasks listed below, after reading the IFU:

• Authentication endpoint tests
  • Credential validation
  • Token expiry and refresh
• Diagnosis support endpoint tests
  • The simplest workflow
  • Beyond the boundaries of the standard operating flow
  • Missing or extraneous field in the request
  • Submission of multiple images for a more reliable diagnosis
  • Handling corrupted images
  • Excessive number of images
• Severity assessment endpoint tests
  • Evaluating the severity of skin conditions of different natures
  • Scoring system requiring a supporting questionnaire
  • Inappropriate scoring system for the condition
  • Scoring system unspecified or misspelled
  • Sending more than one image.

Before executing the tests, all the participants received the instructions to perfom the tasks listed above (documented in the R-TF-012-016 Software usability test guide), the link to the eIFU and an evaluation form to be filled in after the execution of the test to gather qualitative data.

Participants or testers

The study was conducted with 5 participants who are representative of the intended users.

As explained in the R-TF-012-014 Summative evaluation plan, we considered this number of participants a representative sample to achieve meaningful results.

The 5 participants were carefully selected to match the user profile we defined in the R-TF-012-014 Summative evaluation plan and to represent the diversity of the intended user population in terms of level of experience.

All the participants matched the user profile and more details about the participants are provided below:

• Francisco Javier Linde: software developer with 10 years of experience
• Pedro Gómez: data scientist with more than 20 years of experience
• Ignacio Ramos: data scientist with 6 years of experience
• Miguel Sánchez: IT consultant with 5 years of experience
• Félix Blanco: software engineer with more than 10 years of experience.

Test environment

The summative evaluation test was perfomed in the offices where the intended users (IT professionals) usually execute their work.

The participants interacted directly with the API, engaging with the device in the same manner as they would in the real-world scenario.

Test results

Data analysis

After the execution of the summative evaluation test, we analysed objective and subjective data, as described in the R-TF-012-014 Summative evaluation plan.

Objective data analysis

Objective data are participants' results of the implementation of the tasks defined for the test.

For this first type of data, we defined in the R-TF-012-014 Summative evaluation plan success metrics for each task, and also listed below:

Use scenario	Task	Success criteria
Scenario 1: Authentication	Credential validation	Successful login when authenticating with correct credentials
Scenario 1: Authentication	Credential validation	Failed login attempts return a 401 Unauthorized status code with error message Invalid credentials when attempting authentication with incorrect credentials
Scenario 1: Authentication	Token expiry and refresh	API identifies expired tokens and returns a 401 Unauthorized status code with error message Invalid credentials
Scenario 1: Authentication	Token expiry and refresh	Successful refresh of token
Scenario 2: Diagnosis support	The simplest workflow (user sends an image to check patient's skin abnormality and stores the results)	API responds with a 200 OK status code for the request sent
Scenario 2: Diagnosis support	The simplest workflow (user sends an image to check patient's skin abnormality and stores the results)	API returns accurate diagnosis for the sent image following FHIR standards
Scenario 2: Diagnosis support	The simplest workflow (user sends an image to check patient's skin abnormality and stores the results)	JSON successfully stored to disk
Scenario 2: Diagnosis support	Beyond the boundaries of the standard operating flow (user sends low quality image(s) and/or image(s) outside the dermatological domain to receive a diagnosis)	API responds with a 200 OK status code for all the requests sent
Scenario 2: Diagnosis support	Beyond the boundaries of the standard operating flow (user sends low quality image(s) and/or image(s) outside the dermatological domain to receive a diagnosis)	API indicates the low-quality image has not passed the quality check
Scenario 2: Diagnosis support	Beyond the boundaries of the standard operating flow (user sends low quality image(s) and/or image(s) outside the dermatological domain to receive a diagnosis)	Non-dermatological image does not result in false positive, the API returns the category Non dermatological
Scenario 2: Diagnosis support	Missing or extraneous field in the request	API fails in processing request with a 422 Unprocessable content status code
Scenario 2: Diagnosis support	Missing or extraneous field in the request	API returns clear error messages for each of the 3 body requests: for body request 1, API returns Field required; for body request 2, API returns Extra inputs are not permitted; for body request 3, API returns Extra inputs are not permitted
Scenario 2: Diagnosis support	Submission of multiple images for a more reliable diagnosis	API responds with a 200 OK status code for the request sent
Scenario 2: Diagnosis support	Submission of multiple images for a more reliable diagnosis	The imagingStudySeries key of the JSON contains 2 JSON object, one for each image sent. Each object contains the results of the analysis following the FHIR interoperability standard
Scenario 2: Diagnosis support	Submission of multiple images for a more reliable diagnosis	The conclusions key of the JSON contains a differential diagnosis resulting from aggregating the results of the diagnosis for each image (endpoint follows the FHIR interoperability standard)
Scenario 2: Diagnosis support	Handling corrupted images	API responds with a 422 Unprocessable content staus code with error message Value error, only base64 data is allowed
Scenario 2: Diagnosis support	Excessive number of images	API responds with a 422 Unprocessable content staus code with error message List should have at most 5 items after validation, not 6
Scenario 3: Severity assessment	Evaluating the severity of skin conditions of different natures	API responds with a 200 OK status code for the request sent
Scenario 3: Severity assessment	Evaluating the severity of skin conditions of different natures	Calculated score for each scoring system specified in the request is returned
Scenario 3: Severity assessment	Evaluating the severity of skin conditions of different natures	For skin conditions where lesions can be counted, JSON contained in the response includes the processed image
Scenario 3: Severity assessment	Scoring system requiring a supporting questionnaire	API responds with a 200 OK status code for the request sent
Scenario 3: Severity assessment	Scoring system requiring a supporting questionnaire	If mandatory questionnaire is not included in the request, API returns 500 Internal server error
Scenario 3: Severity assessment	Inappropriate scoring system for the condition	API responds with a 200 OK status code for the request sent
Scenario 3: Severity assessment	Inappropriate scoring system for the condition	JSON contained in the response includes the processed image with the wrong scoring system applied
Scenario 3: Severity assessment	Scoring system unspecified or misspelled	When sending an empty list, API responds with 500 Internal server error
Scenario 3: Severity assessment	Scoring system unspecified or misspelled	When sending a misspelled scoring system, API returns 422 Unprocessable content status code with error message including a list of exact names of all available scoring systems in the device
Scenario 3: Severity assessment	Sending more than one image	API returns 422 Unprocessable content status code with error message Input should be a valid dictionary or object to extract fields from

For each task, we evaluated the participants' results against the established success metrics and we assigned the following codes:

• OK (Success): when the participants correctly performed a task according to the steps in the R-TF-012-016 Software usability test guide and in the IFU.
• UE (Use error): when the participants performed a task incorrectly and/or did not complete a necessary task, therefore the actual results of the tasks did not match the established success metrics.
• UD (Use difficulty): when the participants reported that they struggled to some extent while completing a task (e.g., confusion, taking longer than expected, hesitation, uncertainty on what to do), but eventually could complete the task.

Each task must be completed successfully (code OK) by at least 80% of the participants.

The objective data are summarized in the Findings section of this report.

Subjective data analysis

Subjective data are participants' subjective feedback collected by means of an evaluation form.

For this second type of data, we analysed the answers of the participants to the questions contained in the evaluation form and we summarised the results in the User feedback section of this report.

We assigned scores to the answers as shown below:

• When authenticating with the API, how easy was it to understand and use the authentication process?
  • Very difficult (1)
  • Difficult (2)
  • Neutral (3)
  • Easy (4)
  • Very easy (5)
• Were the error messages clear and helpful when login attempts failed?
  • Not clear at all (1)
  • Somewhat unclear (2)
  • Neutral (3)
  • Clear (4)
  • Very clear (5)
• How secure do you consider the authentication process to be?
  • Not secure (1)
  • Somewhat insecure (2)
  • Neutral (3)
  • Secure (4)
  • Very secure (5)
• How effective was the token management, including expiration and refresh?
  • Very ineffective (1)
  • Ineffective (2)
  • Neutral (3)
  • Effective (4)
  • Very effective (5)
• When you uploaded images labelled with the corresponding skin disease, did the API return the correct preliminary differential diagnosis (e.g.Melanoma for images labelled as such)?
  • Strongly disagree (1)
  • Disagree (2)
  • Neutral (3)
  • Agree (4)
  • Strongly agree (5)
• How well did the API handle images of different quality?
  • Very poorly (1)
  • Poorly (2)
  • Neutral (3)
  • Well (4)
  • Very well (5)
• How would you rate the response time of the "/diagnosis-support" endpoint?
  • Very slow (1)
  • Slow (2)
  • Average (3)
  • Fast (4)
  • Very fast (5)
• How well did the API manage multiple concurrent requests?
  • Very poorly (1)
  • Poorly (2)
  • Neutral (3)
  • Well (4)
  • Very well (5)
• How secure do you consider the data transmission process to be?
  • Not secure (1)
  • Somewhat insecure (2)
  • Neutral (3)
  • Secure (4)
  • Very secure (5)
• Do you agree the Instructions for Use are clear, comprehensive, and easy to understand?
  • Strongly disagree (1)
  • Disagree (2)
  • Neutral (3)
  • Agree (4)
  • Strongly agree (5)
• Overall, how satisfied are you with the medical device's API connectivity?
  • Very unsatisfied (1)
  • Unsatisfied (2)
  • Neutral (3)
  • Satisfied (4)
  • Very satisfied (5)

In the R-TF-012-014 Summative evaluation plan we established specific success metrics for the completeness and understandability of the IFU. We analysed the answers of the participants to the questions contained in the evaluation form related to the IFU and we evaluated the answers against the success metrics to draw conclusions.

The success metrics defined for IFU completeness and understandability are:

• At least 80% of participants are able to successfully complete the tasks using the IFU
• At least 80% of participants answered with Agree or Strongly agree to the question about rating the IFU as in terms of completeness and understandability in the evaluation form.

Conclusions about subjective data, including conclusion about IFU completeness and understandability, are summarized in the User feedback section of this report.

We also evaluated answers of the participants to questions of the evaluation form related to the safety information provided in the IFU and we summarised the results in the Information about safety section of this report.

Use related risk analysis

Finally, we reviewed and completed the use related risk analysis to evaluate each risk, in terms of probability and severity, after the execution of the summative evaluation test with the aim of verifying whether all use related risks have been addressed.

Conclusions about use related risks are provided in the Use related risks evaluation section of this report.

Findings

We summarised the results of each task performed during the summative evaluation test in the table below:

Use scenario	Task	Success criteria	Results: # of success (OK)	Results: % of success (OK)	Results: # of use error (UE)	Results: % of use error (UE)	Results: # of use difficulty (UD)	Results: % of use difficulty (UD)	Test passed
Scenario 1: Authentication	Credential validation	Successful login when authenticating with correct credentials	4	80%	0	0%	1	20%	TRUE
Scenario 1: Authentication	Credential validation	Failed login attempts return a 401 Unauthorized status code with error message Invalid credentials when attempting authentication with incorrect credentials	5	100%	0	0%	0	0%	TRUE
Scenario 1: Authentication	Token expiry and refresh	API identifies expired tokens and returns a 401 Unauthorized status code with error message Invalid credentials	5	100%	0	0%	0	0%	TRUE
Scenario 1: Authentication	Token expiry and refresh	Successful refresh of token	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	The simplest workflow (user sends an image to check patient's skin abnormality and stores the results)	API responds with a 200 OK status code for the request sent	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	The simplest workflow (user sends an image to check patient's skin abnormality and stores the results)	API returns accurate diagnosis for the sent image following FHIR standards	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	The simplest workflow (user sends an image to check patient's skin abnormality and stores the results)	JSON successfully stored to disk	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	Beyond the boundaries of the standard operating flow (user sends low quality image(s) and/or image(s) outside the dermatological domain to receive a diagnosis)	API responds with a 200 OK status code for all the requests sent	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	Beyond the boundaries of the standard operating flow (user sends low quality image(s) and/or image(s) outside the dermatological domain to receive a diagnosis)	API indicates the low-quality image has not passed the quality check	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	Beyond the boundaries of the standard operating flow (user sends low quality image(s) and/or image(s) outside the dermatological domain to receive a diagnosis)	Non-dermatological image does not result in false positive, the API returns the category Non dermatological	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	Missing or extraneous field in the request	API fails in processing request with a 422 Unprocessable content status code	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	Missing or extraneous field in the request	API returns clear error messages for each of the 3 body requests: for body request 1, API returns Field required; for body request 2, API returns Extra inputs are not permitted; for body request 3, API returns Extra inputs are not permitted	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	Submission of multiple images for a more reliable diagnosis	API responds with a 200 OK status code for the request sent	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	Submission of multiple images for a more reliable diagnosis	The imagingStudySeries key of the JSON contains 2 JSON object, one for each image sent. Each object contains the results of the analysis following the FHIR interoperability standard	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	Submission of multiple images for a more reliable diagnosis	The conclusions key of the JSON contains a differential diagnosis resulting from aggregating the results of the diagnosis for each image (endpoint follows the FHIR interoperability standard)	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	Handling corrupted images	API responds with a 422 Unprocessable content staus code with error message Value error, only base64 data is allowed	5	100%	0	0%	0	0%	TRUE
Scenario 2: Diagnosis support	Excessive number of images	API responds with a 422 Unprocessable content staus code with error message List should have at most 5 items after validation, not 6	5	100%	0	0%	0	0%	TRUE
Scenario 3: Severity assessment	Evaluating the severity of skin conditions of different natures	API responds with a 200 OK status code for the request sent	5	100%	0	0%	0	0%	TRUE
Scenario 3: Severity assessment	Evaluating the severity of skin conditions of different natures	Calculated score for each scoring system specified in the request is returned	5	100%	0	0%	0	0%	TRUE
Scenario 3: Severity assessment	Evaluating the severity of skin conditions of different natures	For skin conditions where lesions can be counted, JSON contained in the response includes the processed image	5	100%	0	0%	0	0%	TRUE
Scenario 3: Severity assessment	Scoring system requiring a supporting questionnaire	API responds with a 200 OK status code for the request sent	5	100%	0	0%	0	0%	TRUE
Scenario 3: Severity assessment	Scoring system requiring a supporting questionnaire	If mandatory questionnaire is not included in the request, API returns 500 Internal server error	5	100%	0	0%	0	0%	TRUE
Scenario 3: Severity assessment	Inappropriate scoring system for the condition	API responds with a 200 OK status code for the request sent	5	100%	0	0%	0	0%	TRUE
Scenario 3: Severity assessment	Inappropriate scoring system for the condition	JSON contained in the response includes the processed image with the wrong scoring system applied	5	100%	0	0%	0	0%	TRUE
Scenario 3: Severity assessment	Scoring system unspecified or misspelled	When sending an empty list, API responds with 500 Internal server error	5	100%	0	0%	0	0%	TRUE
Scenario 3: Severity assessment	Scoring system unspecified or misspelled	When sending a misspelled scoring system, API returns 422 Unprocessable content status code with error message including a list of exact names of all available scoring systems in the device	5	100%	0	0%	0	0%	TRUE
Scenario 3: Severity assessment	Sending more than one image	API returns 422 Unprocessable content status code with error message Input should be a valid dictionary or object to extract fields from	5	100%	0	0%	0	0%	TRUE

According to these results, more than 80% of participants could successfully execute and complete the test cases created for the summative evaluation test. All users could successfully execute all the test cases related to use scenario 2 (diagnosis support) and to use scenario 3 (severity assessment). One participant reported a use difficulty related to use scenario 1 (authentication), test case 1 (Successful login when authenticating with correct credentials). Eventually the participant could finalize the test case successfully but with some delay due to the difficulty encountered. We opened a nonconformity according to our internal procedure (GP-006) to immediately investigate the use difficulty. After the evaluation of JD-005 (Technical manager), we concluded that the issue arose because the participant sent the request body as JSON instead of Form-Data. To promptly resolve this issue, the instruction guide was updated to clearly state that user credentials should be submitted as Form-Data rather than JSON. The revised guide was then redistributed to all participants to prevent any further confusion.

Use errors

During the summative evaluation test performed in June and first week of July 2024, no use errors have been reported. All the participants could complete all the assigned test cases with results matching the success criteria defined for each test case. As mentioned in the previous section (Findings), we encountered only one use difficulty related to the authentication process. The participant reporting the difficulty could eventually finalise the test case.

User feedback

We evaluated user feedback by means of the questions contained in the evaluation form.

We summarised the analysis of the answers in the table below together with the established success criteria (minimum average score):

Question	Minimum average score	Average result
When authenticating with the API, how easy was it to understand and use the authentication process?	3	3.20
Were the error messages clear and helpful when login attempts failed?	3	3.40
How secure do you consider the authentication process to be?	3	4.20
How effective was the token management, including expiration and refresh?	3	4.20
When you uploaded images labelled with the corresponding skin disease, did the API return the correct preliminary differential diagnosis (e.g.Melanoma for images labelled as such)?	3	3.60
How well did the API handle images of different quality?	3	3.80
How would you rate the response time of the "/diagnosis-support" endpoint?	3	3.60
How well did the API manage multiple concurrent requests?	3	3.80
How secure do you consider the data transmission process to be?	3	3.80
Overall, how would you rate the clarity of error messages?	3	3.40
Overall, how satisfied are you with the medical device's API connectivity?	3	3.20

Plus, we asked the participants to answer the following questions:

• Please describe any difficulties encountered during the execution of the test (if any)
• Please provide any recommendations for improvements or other comments.

For the first question, the user feedback received is coming from one participant that reported a use difficulty during the authentication process. More details provided in the Findings section above.

For the second question, the recommendations for improvements received are related to:

• IFU Table of content: one participant reported that it would be better to have a navigable (clickable) table of contents to be able to move to the required section in a faster way
• JSON file: one participant reported that there were a few extra commas in the JSON file.

According to the success metrics established for the completeness and understandability of the IFU (see section Data analysis of this report), we can conclude that the results obtained are satisfactory since they met the success metrics: 80% of participants rated the IFU as complete and understandable, including the information about safety.

Overall, the participants found the IFU useful to complete the test cases and all the participants reported that the IFU were always available during the execution of the summative evaluation test.

Information about safety

Information about safety available in the IFU were evaluated by means of the following questions contained in the evaluation form sent to each participants:

• Do you agree that the safety information included in the Instructions for Use are clear, comprehensive, and easy to understand?
• Do you agree that the provided documentation was sufficient for using the API effectively?

According to the user's feedback, the safety information included in the IFU are clear, comprehensive and easy to understand; plus, as commented in the section above, overall the participants found the IFU useful to complete the test cases.

Use related risks evaluation

The test cases described in the R-TF-012-016 Software usability test guide have been designed to allow the evaluation of each risks identified and described in the use related risk analysis.

The table below represents the complete use related risk analysis with evaluation of severity and probability after risk control meausures implementation and execution of the test cases, and evaluation of residual risks.

Risk ID	Use Step/ Task Statement	User Group	Task Classification	Use Error (Potential Failure Modes)	Potential Result of Use Error or Hazard (Potential Failure Modes)	Foreseeable Sequence of Events (Potential Failure Modes)	Hazardous Situation (Potential Failure Modes)	Potential Harm (Potential Failure Modes)	Severity (Risk Pre-Mitigation)	Probability (Risk Pre-Mitigation)	Risk Priority Number (Risk Pre-Mitigation)	Risk control measures (Risk control)	Verification of implementation of risk control measures (Risk control)	Validation method (Risk control)	Verification of effectiveness of risk control measures (Risk control)	Severity (Residual risk evaluation)	Probability (Residual risk evaluation)	Risk Priority Number (Residual risk evaluation)	Residual risk evaluation (Risk/Benefit Analysis)	Additional mitigation measures (Risk/Benefit Analysis)
UR-001	Get access token	ITP	Critical	User may input incorrect username, password, or other required authentication details	Repeated failed login attempts could lead to account lockout or delays in data access, impacting timely patient care	- The IT professional attempts to access the API to integrate it into the healthcare organization's system - The IT professional inputs the required authentication details - The IT professional inputs incorrect authentication details (e.g., mistyped username or password)	The user is unable to obtain an access token due to the incorrect input of authentication details, leading to a failure to authenticate and access the API	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device	3	4	12	Clear error messages, provide technical support to assist users in troubleshooting and resolving authentication issues	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system	Summative evaluation test: Scenario 1 - Authentication	R-TF-012-015 Summative evaluation report_2024_001	3	2	6	As far as possible	Not applicable
UR-002	Get access token	ITP	Critical	User may enter an incorrect URL for the login endpoint	The device may fail to authenticate, preventing access to the protected endpoints and delaying clinical decision-making	- The IT professional attempts to access the API to integrate it into the healthcare organization's system - The IT professional configures the system with the URL for the login endpoint to obtain an access token - The IT professional enters an incorrect URL for the login endpoint - The system attempts to contact the incorrect URL, resulting in a failed request	The system fails to obtain an access token due to the incorrect URL, preventing authentication and access to the API	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden	3	3	9	Clear error messages, provide technical support to assist users in troubleshooting, provide information in the IFU	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-003	Get access token	ITP	Critical	User may use poor or unstable internet connection	Inability to retrieve or validate the access token, causing delays in accessing the device’s features	- The IT professional attempts to access the API to integrate it into the healthcare organization's system - The system sends a request to the login endpoint to obtain an access token - The internet connection is poor or unstable, leading to intermittent connectivity or a complete loss of connection - The request to the login endpoint fails or times out due to the unstable connection - The system or the user may attempt to retry the request multiple times, experiencing repeated failures	The system fails to obtain an access token due to the unstable connection, preventing authentication and access to the API	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Robust error handlings, state-of-the-art techniques of security and software availability, provide information in the IFU about minimum IT requirements, provide technical support to assist users in troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication	R-TF-012-015 Summative evaluation report_2024_001	3	2	6	As far as possible	Not applicable
UR-004	Build JSON with data	ITP	Critical	User might format the data incorrectly, such as wrong JSON structure or incorrect field names	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data. - The IT professional enters the data into the JSON object - The IT professional makes an error, resulting in incorrect JSON structure (e.g., missing brackets, commas) or incorrect field names (e.g., misspelled or mismatched with the API specification) - The incorrectly formatted JSON is submitted to the API endpoint - The API detects the formatting error and responds with an error message	The system fails to process the incorrectly formatted JSON, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Clear error messages, comprehensive information in the IFU that details the required JSON structure, field names, examples of correctly formatted data, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-005	Build JSON with data	ITP	Critical	User might omit mandatory fields required for the request	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data - The IT professional enters the data into the JSON object - The IT professional omits one or more mandatory fields required by the API specification - The incomplete JSON is submitted to the API endpoint - The API detects the missing mandatory fields and responds with an error message	The system fails to process the incomplete JSON, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Clear error messages, comprehensive information in the IFU that details the required JSON structure, field names, examples of correctly formatted data and mandatory fields, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-006	Build JSON with data	ITP	Critical	User may enter data in incorrect types (e.g., string instead of integer)	If the device is unable to parse the incorrect data type to the data type defined by the schema of the request, then the user's request will be rejected. If the error message is not clear enough, the user may not be able to send a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data - The IT professional enters the data into the JSON object - The IT professional enters data in incorrect types, such as a string instead of an integer or vice versa - The incorrectly typed JSON is submitted to the API endpoint - The API detects the type error and responds with an error message	The system fails to process the JSON due to incorrect data types, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	3	9	Clear error messages, comprehensive information in the IFU that details the required JSON structure, field names, examples of correctly formatted data, correct data types and mandatory fields, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-007	Build JSON with data	ITP	Critical	User may send images in an unsupported format (other than Base64) or incorrectly encoded in Base64	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data, including images - The IT professional encodes the images to be included in the JSON object - The IT professional either uses an unsupported format (other than Base64) or incorrectly encodes the images in Base64 - The incorrectly formatted JSON, containing the improperly encoded images, is submitted to the API endpoint - The API detects the unsupported or incorrect image encoding and responds with an error message	The system fails to process the JSON due to unsupported image formats or incorrect Base64 encoding, leading to a rejection of the data submission and a halt in the intended workflow	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Validation checks, clear error messages, information in the IFU, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-008	Build JSON with data	ITP	Critical	User may include extraneous data not required for the request	The device rejects the request and returns a cryptic error message that the user can't understand, making it difficult to determine what went wrong and potentially causing the user to fail in sending a successful request.	- The IT professional attempts to integrate the API by building a JSON object containing the required data - The IT professional enters the data into the JSON object - The IT professional includes extraneous data that is not required or specified by the API documentation - The JSON object, containing the extraneous data, is submitted to the API endpoint - The API processes the request and responds with an error message	The system may fail to process the JSON correctly	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	3	9	Clear error messages, comprehensive information in the IFU, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-009	Send JSON to device	ITP	Critical	User might send the request to the wrong endpoint or to a non-existent URL	Failed to connect to the device, delaying the decision-making process and potentially impacting patient care	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional incorrectly enters the URL or endpoint - The JSON request is sent - The device responds with an error	The system fails to process the JSON due to sending the request to an incorrect or non-existent endpoint, leading to a failure in data transmission	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	4	12	Clear error messages, comprehensive information in the IFU that details the correct endpoints, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-010	Send JSON to device	ITP	Critical	User may use the wrong HTTP method (e.g. GET instead of POST)	Failed to connect to the device, delaying the decision-making process and potentially impacting patient care	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional selects the HTTP method to be used for the request (e.g., GET, POST) - The IT professional mistakenly chooses the wrong HTTP method, such as selecting GET instead of the required POST - The JSON request is sent to the device using the incorrectly chosen HTTP method - The device responds with an error message	The system fails to process the JSON request due to using an incorrect HTTP method, leading to a rejection of the request by the device	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts	3	3	9	Clear error messages, comprehensive information in the IFU that details the supported HTTP methods, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-011	Send JSON to device	ITP	Critical	User may not include the access token in the request	Authentication will fail, leading to interruptions in workflow and potential delays in the clinical assessment process	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional obtains the access token required for authentication - The IT professional constructs the JSON request, intending to include the required access token - The IT professional mistakenly omits the access token from the request - The JSON request lacking the access token is sent to the device - The device detects the missing access token and responds with an error message	The system fails to authenticate the user due to the absence of the access token, leading to a rejection of the request by the device	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data security risk	3	4	12	Clear error messages, inclusion of the requirement for including access token in all requests in the IFU, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication, Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-012	Send JSON to device	ITP	Critical	User may include the access token in the body of the request instead of headers	Authentication will fail, leading to interruptions in workflow and potential delays in the clinical assessment process	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional obtains the access token required for authentication - The IT professional constructs the JSON request and mistakenly includes the access token in the body of the request payload instead of the request headers - The JSON request with the access token in the body is sent to the device - The device receives the request but does not recognize the access token in the body as it is expecting it in the headers	The device may fail to authenticate the request due to the access token being included in the wrong part of the request, leading to potential unauthorized access or data security risks.	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data security risk	3	4	12	Clear error messages, inclusion of correct method of including access token in all requests in the IFU, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication, Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-013	Send JSON to device	ITP	Critical	User might attempt to send an expired token	Authentication will fail, leading to interruptions in workflow and potential delays in the clinical assessment process	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional obtains the access token required for authentication - The access token obtained by the IT professional expires due to reaching its predefined expiration time - The IT professional attempts to use the expired access token to authenticate the JSON request - The JSON request with the expired access token is sent - The device receives the request with the expired token and responds with an error message	The device may fail to authenticate the request due to the access token being included in the wrong part of the request, leading to potential unauthorized access or data security risks.	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data security risk	3	3	9	Clear error messages, instructions on how to obtaining new token in the IFU, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication, Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-014	Send JSON to device	ITP	Critical	User might send the exact same request multiple times	Redundant processing, wasting resources and possibly causing confusion in clinical assessments	- The IT professional prepares to send the JSON object containing the required data to the API - The IT professional sends the JSON request - Due to network instability, or misunderstanding, the IT professional resends the same JSON request multiple times - The device receives multiple identical requests from the same user within a short period - The device processes each request as a new submission	The device processes multiple identical requests, potentially leading to duplicate actions or data entries	Misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data duplication	3	3	9	Clear error messages, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-015	Receive JSON from device	ITP	Critical	Network issues, device downtime, or incorrect handling of the request	Delays in obtaining clinical data, which can impact the timeliness and accuracy of clinical assessments	- The IT professional or system initiates a request to receive JSON data from the device - During the communication, network issues (such as instability, high latency, or disconnection) or device downtime occurs or the IT professional incorrectly handles the request, leading to improper processing or failure to receive the expected data - The device either fails to respond, sends a corrupted response, or the client misinterprets the response due to handling errors - The client system either waits indefinitely, retries the request, or processes an incomplete/corrupt JSON response	The system does not receive the necessary JSON data correctly or in a timely manner, leading to a failure in processing the required information	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, incomplete data processing	3	4	12	State-of-the-art techniques of security and software availability, clear error messages, implementation of recovery strategy, monitoring of device downtime, provide technical support for troubleshooting, information on how to handle request provided in the IFU	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	2	6	As far as possible	Not applicable
UR-016	Receive JSON from device	ITP	Critical	The response might be incomplete or corrupted due to data transmission issues	Incomplete or corrupted data can lead to incorrect clinical assessments	- The IT professional or system initiates a request to receive JSON data from the device - The device processes the request and sends the JSON data back to the client system - During the data transmission, network instability, interference, or other issues cause the data to be incomplete or corrupted - The client system receives the JSON response, but the data is either incomplete, or corrupted - The client system attempts to process the received JSON data, leading to potential errors or failures	The system processes incomplete or corrupted data, leading to potential errors in the application's functioning and data interpretation	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	State-of-the-art techniques of security and software availability, clear error messages, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	2	6	As far as possible	Not applicable
UR-017	Receive JSON from device	ITP	Critical	Misinterpretation of the HTTP status code of the response	If a user misinterprets a successful status code (e.g., 200 OK) as an error, they might not process the received data. Conversely, interpreting an error status code (e.g., 404 Not Found, 500 Internal device Error) as successful could lead to the belief that the data has been processed correctly, when it hasn't. This delay or failure in data processing can impact timely clinical decisions.	- The IT professional initiates a request to receive JSON data from the device - The device processes the request and sends back a response with an HTTP status code and JSON data - The client system receives the response from the device, including the HTTP status code and JSON payload - The client system or IT professional misinterprets the HTTP status code (e.g., interpreting a 500 Internal Server Error as a 200 OK, or vice versa) - Based on the misinterpreted status code, the client system or IT professional takes incorrect actions, such as processing incomplete data or failing to retry a failed request	The system proceeds with incorrect assumptions about the success or failure of the request, leading to potential errors in data handling and application behavior	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	Information provided in the IFU, clear error messages, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-018	Receive JSON from device	ITP	Critical	User may accidentally delete or modify the request during the process	Clinical information generated by the device is lost. The user would have to send the same request, leading to a delay in the decision-making process	- The IT professional or system initiates a request to receive JSON data from the device - The client system prepares the request for data transmission - The IT professional accidentally deletes or modifies the request - The client system attempts to handle the response, which may be an error message or an unexpected result due to the incorrect request	The system does not receive the expected JSON data due to the accidental modification or deletion of the request, leading to potential errors in the application's functioning and data interpretation	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	Clear error messages, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-019	Receive JSON from device	ITP	Critical	User might incorrectly map the response data to the clinical workflow or patient records	Incorrect data mapping can result in misidentification or misassociation of clinical results with the wrong patient, leading to potential harm	- The IT professional or system initiates a request to receive JSON data from the device - The device processes the request and sends back a JSON response containing the clinical data - The client system receives the JSON response from the device - The IT professional processes the JSON response and maps the data to the clinical workflow or patient records - During this process, the user incorrectly maps the data (e.g., assigns data to the wrong patient record, uses incorrect data fields, or mismatches data types) - The mapped data is used in the clinical workflow or patient records, impacting clinical evaluations or decisions	Incorrect mapping of response data leads to erroneous data being integrated into the clinical workflow or patient records	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	Information provided in the IFU, error handling	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-020	Process and store JSON	ITP	Critical	User may misinterpret the format or structure of the response data, leading to incorrect data extraction	Inaccurate or incomplete clinical data being stored, potentially leading to incorrect clinical decisions	- The client system receives the JSON response from the device - The IT professional begins processing the JSON response to extract relevant data - The user misinterprets the format or structure of the JSON data (e.g., misunderstanding the hierarchy, incorrect field names, or data types) - Due to the misinterpretation, the system extracts data incorrectly (e.g., extracting wrong values, missing important data, or misinterpreting data fields) - The incorrectly extracted data is stored in the database or integrated into the clinical workflow	Incorrectly extracted and stored data leads to errors in the clinical workflow or patient records	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	Detailed information on the JSON response format and structure to guide users in correctly interpreting the data provided in the IFU, clear error messages, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-021	Process and store JSON	ITP	Critical	Issues with the storage system, such as database failures or insufficient storage capacity	Loss of critical clinical data or inability to retrieve stored data when needed can delay patient care	- The client system receives the JSON response from the device - The client system processes the JSON data to prepare it for storage - The system attempts to store the processed data in the database - The storage system encounters issues such as database failures, insufficient storage capacity, or connection problems - The system may fail to store the data correctly, partially store the data, or not store the data at all - Based on the system's handling of the error, the user may take incorrect actions, such as assuming the data is stored correctly	Data loss or corruption due to issues with the storage system, leading to incomplete or missing clinical data	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	The deployment of the medical devices uses elastic demand design. The medical device makes constant backups. State-of-the-art techniques of security and software availability. Due to the inherent features of the REST protocol, when a user send a request and the device is down, the device returns a specific code informing of the state of the device, including downtime. This means that the user will be automatically aware of downtime, as well as any other states. Clear error messages, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system	Summative evaluation test: Scenario 2 - Diagnosis support	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-022	Process and store JSON	ITP	Critical	A decoder other than JSON is used to extract the information from the response	A non-JSON decoder might misinterpret the structure and content of the JSON response, leading to corrupted data. This could result in incorrect values being extracted, or the data might be entirely unusable. Misinterpreted data can lead to inaccurate clinical assessments and inappropriate clinical decisions	- The client system receives the JSON response from the device - The IT professional initiates the process to decode and extract data from the response - Instead of using a JSON decoder, a different type of decoder (e.g., XML, YAML, or binary decoder) is incorrectly selected to process the response - The incorrect decoder fails to properly interpret the JSON structure, leading to errors such as corrupted data, incomplete data extraction, or extraction of incorrect values - The incorrectly decoded data is stored in the database or integrated into the clinical workflow	Incorrectly decoded and stored data leads to errors in the clinical workflow or patient records	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	3	9	Information in the IFU, error handling	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-023	Process and store JSON	ITP	Critical	New data overwriting existing data due to incorrect handling of unique identifiers or version control	Loss of historical data, which can be crucial for longitudinal patient studies and follow-up care	- The client system receives the JSON response from the device - The IT professional initiates the process to handle and store the received data - Due to a programming error or oversight, the system incorrectly interprets or mishandles the unique identifiers or version control information - The system erroneously concludes that the new data should overwrite existing data, based on the mishandled identifiers or version control - The system proceeds to store the new data, potentially overwriting or modifying existing data in the database - The storage operation completes, with the new data potentially replacing or modifying existing records	Existing data may be inadvertently overwritten or modified, leading to data loss or corruption	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data loss, corruption	3	4	12	Implementation of REST protocol to inherently avoid bad practices in programming such as data re-writing (every request is independent and cannot be edited), error handling, provide technical support for troubleshooting	TEST_007_If something does not work, the API returns meaningful information about the error, TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner	Summative evaluation test: Scenario 2 - Diagnosis support	R-TF-012-015 Summative evaluation report_2024_001	3	0	0	The risk is now impossible	Not applicable
UR-024	Process and store JSON	ITP	Critical	User failing to implement adequate security measures to protect stored data	Unauthorized access to or breaches of sensitive patient data can lead to privacy violations and potential legal consequences	- The client system receives JSON data from the device and proceeds to process and store it in the database - The user fails to implement appropriate security measures to safeguard the stored data - Due to the lack of security measures, the stored data becomes vulnerable to various security threats such as unauthorized access, data breaches, or malicious attacks - The compromised data may be stolen, tampered with, or deleted by unauthorized parties	The stored data is exposed to risks of unauthorized access, manipulation, or theft due to inadequate security measures	Data breaches, security violations, data manipulation, loss of trust, regulatory and legal consequences	3	4	12	Implementation of data encryption, implementation of robust authentication mechanisms such as OAuth or JWT to ensure that only authorized users can access the device, monitoring of security threats, information about cybersecurity included in the IFU	TEST_011_We facilitate the integration of the device into the users' system, TEST_012_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-025	Process and store JSON	ITP	Critical	Storing data in inconsistent formats that are not compatible with existing systems	Difficulties in data integration, retrieval, and analysis, compromising clinical workflow efficiency	- The client system receives JSON data from the device and initiates the process to store it in the database - Due to a programming error or oversight, the system stores the data in an inconsistent format that is not compatible with existing systems or data models - The system completes the storage operation, resulting in the inconsistent format data being saved in the database - When attempting to retrieve the stored data, other systems or processes encounter issues due to the inconsistent format - The inconsistent format data cannot be properly interpreted or utilized by existing systems or processes, leading to compatibility problems - Other systems or processes may misinterpret the inconsistent format data, leading to errors, inaccuracies, or disruptions in data processing	The stored data is in inconsistent formats that are not compatible with existing systems or data models, leading to potential data misinterpretation or processing errors	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, IT professionals may experience frustration and stress due to repeated failed attempts, data inconsistency	3	4	12	The endpoints of the device follow HL7's FHIR interoperability standard, information about data formats is provided in the IFU, provide technical support for troubleshooting	TEST_011_We facilitate the integration of the device into the users' system, TEST_013_The data that users send and receive follows the FHIR healthcare interoperability standard, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-026	Build report	ITP	Critical	User might misunderstand the data format or the meaning of specific data points generated by the device	Incorrect representation of data in the UI, leading to misinformed clinical decisions	- The IT professional retrieves data from the device to build a report - The IT professional may misunderstand certain aspects of the data, such as its structure, semantics, or context - Based on the interpreted data, the IT professional proceeds to build the report - The misinterpretation of data format or meaning may lead to inaccuracies, errors, or omissions in the report content - The report is completed and may be shared with healthcare practitioners	The IT professional misunderstands the data format or the meaning of specific data points, leading to inaccuracies or errors in the generated report	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden	3	4	12	The endpoints of the device follow HL7's FHIR interoperability standard, comprehensive information about values returned by the device provided in the IFU, Swagger documentation	TEST_011_We facilitate the integration of the device into the users' system, TEST_013_The data that users send and receive follows the FHIR healthcare interoperability standard, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-027	Build report	ITP	Critical	Data might be displayed inconsistently, such as varying units of measurement, inconsistent decimal places	Confusion or misinterpretation by IT professionals, potentially resulting in clinical errors when data are shown to HCP	- The IT professional retrieves data from the device to build a report - Due to oversight or programming error, the data is formatted inconsistently, such as using varying units of measurement or inconsistent decimal places - The inconsistent data is used to generate the report - The report displays the data inconsistently, with varying units of measurement or inconsistent decimal places - Healthcare practitioners may find it challenging to interpret the inconsistently displayed data accurately	The report displays data inconsistently, leading to potential misinterpretation or misapplication by healthcare practitioners	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden	3	3	9	The endpoints of the device follow HL7's FHIR interoperability standard, comprehensive information about values returned by the device provided in the IFU, Swagger documentation	TEST_011_We facilitate the integration of the device into the users' system, TEST_013_The data that users send and receive follows the FHIR healthcare interoperability standard, R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-028	Read IFU	ITP	Critical	User may misunderstand the technical language or specific instructions provided in the IFU	Incorrect use of the device, leading to inaccurate data being generated, which could result in improper clinical decisions	- The user reads through the IFU document to understand the technical specifications, operating instructions, and safety precautions associated with the software - Due to complexity, ambiguity, or lack of clarity in the technical language or specific instructions, the user may misunderstand certain aspects of the IFU content - The user may inadvertently apply incorrect procedures, settings, or precautions based on the misunderstood information from the IFU	Misunderstanding the technical language or specific instructions provided in the IFU may lead to potential harm or errors in the use of the software	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	4	12	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), provide technical support	R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication, Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-029	Read IFU	ITP	Critical	User might skip through the IFU or skip sections, missing critical information about the device's operation or limitations	Incomplete understanding of device functionality, which may cause incorrect usage and potentially unreliable clinical data	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to time constraints, lack of attention, or overconfidence, the user may skim through or skip sections of the IFU without thoroughly reviewing all content - The user may miss critical information about the device's operation, maintenance requirements, safety precautions, or limitations - The user proceeds to execute procedures or interact with the software based on incomplete or misunderstood information from the IFU	Incomplete understanding of device operation or limitations, posing risks during device use	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	3	9	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, provide technical support	R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication, Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-030	Read IFU	ITP	Critical	User may not notice or fully comprehend safety warnings and precautions	Increased risk of harm to patients or practitioners due to ignored safety protocols, possibly leading to injury or incorrect clinical decisions	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to distractions, time constraints, or other factors, the user may not fully concentrate on reading the IFU content - As a result of lack of attention, the user may overlook or skim over safety warnings and precautions outlined in the IFU - The user proceeds to execute procedures or interact with the software or device without fully considering or understanding the safety warnings and precautions	User may not notice or fully comprehend safety warnings and precautions provided in the IFU, leading to potential harm or errors in device use	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	3	9	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, provide technical support	R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication, Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-031	Read IFU	ITP	Critical	User might not fully grasp the instructions if the IFU is not available in their preferred language or if the language used is too complex	Miscommunication of device usage steps, leading to errors in operation and data interpretation	- The user attempts to access the Instructions for Use (IFU) document provided with the software - If the IFU is not available in the user's preferred language, or if the language used is too complex for their comprehension level, the user may struggle to understand the instructions - As a result of language barriers or complexity, the user may only partially understand the instructions provided in the IFU - Despite incomplete understanding, the user proceeds to execute procedures or interact with the software based on their interpretation of the IFU instructions	Inadequate understanding of IFU instructions due to language barriers or complexity may lead to potential harm or errors in device use	Delay in device integration leading to misdiagnosis or delays in the decision-making process, loss of trust in the device, increase technical support burden, user frustration	3	4	12	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, translation of the IFU according to internal procedure (GP-022 Document translation), provide technical support	R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication, Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-032	Read IFU	ITP	Critical	User might misinterpret the device's intended use or its limitations, thinking it is a final diagnosis report instead of a support to diagnosis for the HCP	The IT staff builds a user interface in which the device results do not reach the HCP but are returned to the patient as a definitive diagnosis	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to lack of attention, incomplete comprehension, or preconceived notions, the user may misinterpret the device's intended use or its limitations - The user may mistakenly believe that the device provides a final diagnosis rather than serving as a support tool for healthcare practitioners (HCPs) during the diagnostic proces - Based on the misinterpretation, the user may assume that the device's outputs are conclusive diagnostic reports without further validation or confirmation by HCPs - Diagnosti report sent to patient instead of HCP	Misinterpreting the device's intended use or its limitations as final diagnostic reports may lead to potential harm or errors in clinical decision-making	Misdiagnosis, inappropriate treatments, legal concerns	3	4	12	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, information about intended use and users provided in the first section of the IFU, provide technical support	R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication, Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable
UR-033	Read IFU	ITP	Critical	User may not adhere to the recommended sequence of operations provided in the IFU	Incorrect use of the device, leading to potential errors in image analysis and data generation	- The user accesses the Instructions for Use (IFU) document provided with the software - Due to lack of attention, oversight, or misunderstanding, the user may misinterpret or overlook the recommended sequence of operations outlined in the IFU - By deviating from the recommended sequence of operations, the user may fail to properly integrate the device	Non-adherence to the recommended sequence of operations provided in the IFU may lead to potential harm or errors in device use	Misdiagnosis, compromised performance, user frustration	3	3	9	IFU has been written according to the applicable regulations (MDR 2017/745 and ISO 15223-1) taking into account language requirements for the intended users, provide IFU in accessible format (electronic IFU), IFU structure optimization to facilitate easy navigation, implementation of a comprehensive table of contents as first section of the IFU, provide technical support	R-TF-001-006 IFU and label validation 2023_001	Summative evaluation test: Scenario 1 - Authentication, Scenario 2 - Diagnosis support, Scenario 3 - Severity assessment	R-TF-012-015 Summative evaluation report_2024_001	3	1	3	Acceptable	Not applicable

Conclusions

According to the results coming from the execution of the summative evaluation test (objective data) and the results coming from the evaluation form (subjective data), we can conclude that the objective of the summative evaluation test (validate that the Software as a Medical Device (SaMD) can be used safely and effectively by the intended user population in the intended use environment) has been successfully met.

All the test cases were successfully passed and only one user reported a use difficulty for one test case, the users' feedback is positive especially for the following aspects: security and effectiveness of the authentication process, security of data transmission, clarity of error messages, API's capability of managing multiple concurrent requests, response time of endpoints, API's capability of handling images with different quality.

All the risks related to usability have been evaluated and addressed and, after the execution of the summative evaluation test, all risks are acceptable and/or reduced as far as possible, therefore no further mitigation measures are required for the safe use of the device.

Although the overall results of the summative evaluation test are satisfactory, we aim to follow up on usability aspects during the post-market surveillance phase by gathering data about user's experience on integration of our API.

Record signature meaning

• Author: JD-004
• Review: JD-003
• Approval: JD-005