R-TF-025-002 Identification of characteristics for safety and possible use errors
Purpose
This document is intended to identify the main characteristics related to safety of the medical device as well as potential user errors.
These errors of use help to identify hazardous situations that could arise for the patient.
Identification of user interface characteristics related to safety
The characteristics of the user interface that could affect safety are identified by conducting a task analysis from the high-level description of our medical device. A list of tasks and sub-tasks, that can also be divided into a sequence of steps, is established to describe all the primary operating functions of the medical device.
These primary operating functions are recorded as use scenarios, identified by a unique identifier (ID). These scenarios will be the common thread of the usability process.
ITP Use Scenario
Phase | Task | Classification | Success Criteria |
---|---|---|---|
Build JSON with data | Enter data | Critical | User does not enter data in incorrect types (e.g., string instead of integer) |
Build JSON with data | Enter mandatory fields | Critical | User enters mandatory fields required for the request |
Build JSON with data | Format the data | Critical | User formats the data correctly, does not include the wrong JSON structure or incorrect field names |
Build report | Generate report | Critical | User understands the data format, and the data are displayed consistently. |
Build report | Send images to the API | Critical | User codes for images to be sent in a supported format, or correctly encoded in Base64 |
Build report | Send required data to the API | Critical | User does not include extraneous data not required for the request |
Get access token | Input authorization details | Critical | User inputs correct username, password, or other required authentication details |
Get access token | Input correct URL for login endpoint | Critical | User enters the correct URL for the login endpoint |
Get access token | Use a stable internet connection | Critical | User uses a stable internet connection |
Process and store JSON | Extract the data | Critical | User extracts the data with the JSON decoder. The extracted data are compatible with existing systems and do not overwrite existing data. |
Receive JSON from device | Process the request | Critical | The user completes the response without corrupting it, misinterpreting the HTTP status, deleting or modifying it, or incorrectly mapping the response data to the clinical workflow or patient records. |
Send JSON to device | Input correct HTTP method | Critical | User uses the correct HTTP method |
Send JSON to device | Send access token with the request | Critical | User includes the access token in the header. The access token is not expired, and the request is only sent once |
Send JSON to device | Send request to the endpoint | Critical | User sends the request to the correct endpoint |
HCP Use Scenario
Use Scenario | Phase | Task | Classification | Success Criteria |
---|---|---|---|---|
HCP Use Scenario 1: Simulated Use: No Lesion | Authenticate in the system | User logs into the system. | Critical | The user enters the correct username and password and logs in successfully |
HCP Use Scenario 1: Simulated Use: No Lesion | Authenticate in the system | User logs into the system. | Critical | The user enters the correct username and password and logs in successfully |
HCP Use Scenario 1: Simulated Use: No Lesion | Take pictures | User takes a photo of the patient's lesion. | Critical | The photo has good quality, captures the relevant skin structure, is well lit, focused, with adequate distance |
HCP Use Scenario 1: Simulated Use: No Lesion | Upload pictures | User uploads photos to the Legit.Health plus client. | Critical | The user uploads a high-quality image |
HCP Use Scenario 2: Simulated Use: Lesion | Authenticate in the system | User logs into the system. | Critical | The user enters the correct username and password and logs in successfully |
HCP Use Scenario 2: Simulated Use: Lesion | Upload pictures | User uploads photos to the Legit.Health plus client. | Critical | The user uploads a high-quality image |
HCP Use Scenario 2: Simulated Use: Lesion | Upload pictures | User uploads photos to the Legit.Health plus client. | Critical | The user uploads a high-quality image |
HCP Use Scenario 3: Knowledge Assessment | Read and interpret the report | The user understands that the report is not a certain diagnosis. | Critical | The user acknowledges that the output report is not a standalone diagnosis. |
HCP Use Scenario 3: Knowledge Assessment | Read and interpret the report | User correctly interprets the report output. | Critical | The user correctly identifies the quantification of disease intensity, extent, or count of clinical signs. |
HCP Use Scenario 3: Knowledge Assessment | Read and interpret the report | User correctly reports the probability of certain conditions based on the ICD categories. | Critical | The user identifies the probability of conditions listed in the report |
HCP Use Scenario 3: Knowledge Assessment | Read and interpret the report | User understands the format of the device's output | Critical | The user identifies the information in the report |
Identification of possible use errors and hazard-related scenarios
Based on the use scenarios and the Use specification already established, can identify multiple potential use errors that could affect the normal use of the software. We find potential use errors of the device by answering the following questions for each use scenario:
- What do users need to perceive?
- What do users need to know?
- What do users need to decide?
- What do users need to do?
These use errors are then treated to identify a potential risk linked to them. If a risk is identified, it will be treated in the global risk matrix according to GP-013 Risk Management
. The corresponding scenario is identified as related to the hazard and the tasks composing it are considered as critical tasks.
ID? | Hazard or Use Error? | Type? | Hazardous Situation or Vulnerability? | Foreseeable sequence of events? | Harm? | Risk or Threat? | Security (CIAA)? | User group? | User task? | Cause Requirement(s)? | Affected Asset, Part or People? | Likelihood (Initial)? | Severity (Initial)? | RPN (Initial)? | Control Opt (ABC)? | Implemented mitigation measures? | Mitigation or Control Requirement(s)? | Responsible? | Verification of implementation of risk control measures? | Severity (Controlled)? | Likelihood (Controlled)? | RPN (Controlled)? | Residual risk evaluation? | Verification of effectiveness of risk control measures? | Benefit-risk analysis? | Risks arise from risk control measures?? | Is risk control complete?? | Overall residual risk acceptability? | Threat Model Ref(s)? | Post-Market Plan Ref(s)? |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
R-HBD | Misrepresentation of magnitude returned by the device | Usability Product | The care provider's system represent a value as if was representing a different magnitude. |
| Misdiagnosis; delay in diagnosis/patient's follow up/treatment | The name of the endpoints of the device do not follow a standard | ITP | Use a stable internet connection | PRS-1XUPRS-5LJ | Managing Organisation | 3 | 3 | 9 | AC | The endpoints of the device follow HL7's FHIR interoperability standard and we developed a integration manual within the IFU that explains the values, as well as a Swagger documentation | PRS-1XUPRS-5LJ | Technical director | IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information. HL7's FHIR standard compliance is verified at the TEST_013_The data that users send and receive follows the FHIR healthcare interoperability standard. | 3 | 1 | 3 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | |||
R-DAG | Incorrect diagnosis or follow up | Usability Regulatory | The medical device outputs a wrong result |
| This could lead to misdiagnosis; delays in treatment and worsening of the patient's health status. | The interpretive distribution assigns a low probability to the correct ICD class among the potential ICD classes. | HCP | User logs into the system. | PRS-1V6PRS-1XUPRS-5LJPRS-8QJPRS-9J5 | Patient | 4 | 3 | 12 | AC | Information about device outputs are detailed in the IFU. The medical device returns metadata about the output that helps supervising it, such as explainability media and other metrics. The device returns an interpretative distribution representation of possible ICD categories, not just one single condition. AI models undergo retraining using expanded dataset of images. | PRS-1V6PRS-1XUPRS-5LJPRS-8QJPRS-9J5 | Technical director | Process for verification is defined in GP-012 Design, redesign and development. Implementation of device output information in the IFU verified in TEST_011. Verification of the implementation of metadata about the output of the device to help supervising the output: TEST_001, TEST_002, TEST_003. Verification of the implementation of interpretative distribution representation of possible ICD categories verified in TEST_004. | 3 | 2 | 6 | As far as possible | R-TF-012-015 Summative evaluation report_2024_001 R-TF-015-003 Clinical Evaluation Report (sections: Instructions for Use, Associated Design Product Requirement, Valid clinical association of the International Classification of Diseases (ICD) classes, Clinical performance) | Benefits outweigh the risks | FALSE | TRUE | Acceptable | |||
R-SKK | Incorrect results shown to patient | Usability Cybersecurity Regulatory Artificial Intelligence | The patient see erroneous results. |
| The patient is affected and may suffer anxiety or delays visiting the HCP and their consequent treatment; worsening their health status. | The interpretive distribution assigns a low probability to the correct ICD class among the potential ICD classes.,Patient is using the device without the HCP monitoring | IntegrityAvailability | HCP | User takes a photo of the patient's lesion. | PRS-1V6PRS-1XUPRS-4QWPRS-5LJPRS-8QJPRS-9J5 | Patient | 4 | 3 | 12 | AC | Information about device outputs are detailed in the IFU. The medical device returns metadata about the output that helps supervising it, such as explainability media and other metrics. The device returns an interpretative distribution representation of possible ICD categories, not just one single condition. AI models undergo retarining using expanded dataset of images. | PRS-1V6PRS-1XUPRS-4QWPRS-5LJPRS-8QJPRS-9J5 | Technical director | Process for verification is defined in GP-012 Design, redesign and development. Implementation of device output information in the IFU verified in TEST_011. Verification of the implementation of metadata about the output of the device to help supervising the output: TEST_001, TEST_002, TEST_003. Verification of the implementation of interpretative distribution representation of possible ICD categories verified in TEST_004. | 3 | 1 | 3 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 R-TF-015-003 Clinical Evaluation Report (sections: Instructions for Use, Associated Design Product Requirement, Valid clinical association of the International Classification of Diseases (ICD) classes, Clinical performance) | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | T-024-006-AML-001T-024-006-DAT-002 | T-024-007-AUD-001T-024-007-CVE-002 |
R-E7Z | Inaccessible skin areas | Usability | The device cannot analyse certain skin areas |
| Misdiagnosis; delays in treatment and worsening of the patient's health status. | Inability to access or take a picture of the skin structure due to its location in an unreachable body site and lack of aid in the process | HCP | User takes a photo of the patient's lesion. | PRS-1V6PRS-1XUPRS-5LJPRS-7XK | Patient | 3 | 3 | 9 | AC | A requirement of the device defines the creation of a processor whose purpose is to ensure that the image have enough quality. In other words, an algorithm, similar to the ones used to classify diseases, is used to check the validity of the image and provides an image quality score. The device returns meaningful messages to the users about the quality score of the images. This allows care providers to re-take a photo. The IFU contain the `How to take pictures` section with recommendation on how to take pictures with high quality plus in the Contraindications section of the IFU we state the following: We advise the user not to use the device if skin structures are not accessible by a camera, such as being located in a skin fold or is otherwise covered. We also offer training to the users to optimize the imaging process so that it is optimal for the device's operation. | PRS-1V6PRS-1XUPRS-5LJPRS-7XKPRS-9F2 | Technical director | TEST_009_Notify the user if the quality of the image is insufficient TEST_007_If something does not work, the API returns meaningful information about the error TEST_011_We facilitate the integration of the device into the users' system R-TF-001-006 IFU and label validation | 3 | 1 | 3 | Acceptable | R-TF-015-003 Clinical Evaluation Report (sections: Associated Design Product Requirement, Associated Design Verification Test, Clinical performance) R-TF-012-015 Summative evaluation report_2024_001 | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | |||
R-TA9 | Inadequate camera usage or settings | Product Usability | Poor image quality due to inadequate resolution, lighting, focus or camera settings |
| Misdiagnosis; delays in proper treatment and worsening of the patient's health status. | Inadequate image processing algorithms | ITP | Input authorization details | PRS-1V6PRS-1XUPRS-2ZBPRS-3YHPRS-5LJPRS-7XK | Managing Organisation | 4 | 3 | 12 | AC | A requirement of the device defines the creation of a processor whose purpose is to ensure that the image have enough quality. In other words, an algorithm, similar to the ones used to classify diseases, is used to check the validity of the image and provides an image quality score. The device returns meaningful messages about the quality score of the images. This allows care providers to re-take a photo. The IFU contain a dedicated section on how to take pictures (section name: `How to take pictures`) and technical specifications of the camera (section: `Technical specifications`) We also offer training to the users to optimize the imaging process so that it is optimal for the device's operation. | PRS-1V6PRS-1XUPRS-2ZBPRS-3YHPRS-5LJPRS-7XK | Technical director | Verification is defined in TEST_009_Notify the user if the quality of the image is insufficient. TEST_007 verifies REQ_007. TEST_011 and R-TF-001-006 IFU and label validation verify REQ_012 | 3 | 1 | 3 | Acceptable | R-TF-015-003 Clinical Evaluation Report (sections: Validation and testing of machine learning models, Post-market Clinical investigations (with focus on clinical studies whose primary endpoint is improving diagnostic accuracy)) R-TF-012-015 Summative evaluation report_2024_001 | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | |||
R-UI5 | Inadequate instructions for use: product information for cybersecurity is not included in the IFU | Usability Cybersecurity Regulatory | Presence of vulnerabilities that may compromise the integrity of the system and patient data |
| Unauthorized access to sensitive patient information; incorrect diagnosis; loss of trust | Inadequate information provided by the manufacturer | ConfidentialityIntegrityAvailabilityAuthenticity | ITP | Input authorization details | PRS-1V6PRS-1XUPRS-2KQPRS-7Z8PRS-9F2 | Manufacturer | 4 | 3 | 12 | C | We specify in the IFU the product information for cybersecurity in the section `Security requirements and recommendations` | PRS-1V6PRS-1XUPRS-2KQPRS-7Z8PRS-9F2 | Technical director | IFU verification documented in TEST_011 and in R-TF-001-006 IFU and label validation 2023_001 | 3 | 1 | 3 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | T-024-006-DOC-001T-024-006-AUT-004 | T-024-007-EDU-001T-024-007-VUL-004 |
R-5L4 | Inadequate lighting conditions during image capture | Usability Product | The medical device receives an input that does not have sufficient quality |
| Misdiagnosis; delays in proper treatment and worsening of the patient's health status. | Inadequate image processing algorithms | HCP | User takes a photo of the patient's lesion. | PRS-1V6PRS-1XUPRS-2ZBPRS-5LJPRS-7XK | Patient | 4 | 3 | 12 | AC | A requirement of the device defines the creation of a processor whose purpose is to ensure that the image have enough quality. In other words, an algorithm, similar to the ones used to classify diseases, is used to check the validity of the image and provides an image quality score. The device returns meaningful messages about the quality score of the images. This allows care providers to re-take a photo. The IFU contain a dedicated section on how to take pictures (section name: `How to take pictures`) and technical specifications of the camera (section: `Technical specifications`) We also offer training to the users to optimize the imaging process so that it is optimal for the device's operation. | PRS-1V6PRS-1XUPRS-2ZBPRS-5LJPRS-7XK | Technical director | Verification is defined in TEST_009_Notify the user if the quality of the image is insufficient. TEST_007 verifies REQ_007. TEST_011 and R-TF-001-006 IFU and label validation verify REQ_012 | 3 | 2 | 6 | As far as possible | R-TF-015-003 Clinical Evaluation Report (sections: Validation and testing of machine learning models, Post-market Clinical investigations (with focus on clinical studies whose primary endpoint is improving diagnostic accuracy)) R-TF-012-015 Summative evaluation report_2024_001 | Benefits outweigh the risks | FALSE | TRUE | Acceptable | |||
R-2S3 | Integration failure or errors | Usability Product | Failure to communicate with other systems |
| Misdiagnosis; delayed treatment; loss of trust in the device | Inadequate information provided by the manufacturer | ITP | Input authorization details | PRS-1V6PRS-1XUPRS-5LJ | Managing Organisation | 4 | 3 | 12 | C | We specify the intended user and the required qualification in the IFU Additionally, we include at the IFU the instructions and information required by the ITPs to perform the integration of the device within their system | PRS-1V6PRS-1XUPRS-5LJPRS-9F2 | Technical director | IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information. | 3 | 1 | 3 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | |||
R-HAX | Incorrect interpretation of device outputs | Usability Regulatory | The HCP validates the wrong skin condition, even if the device outputs the correct result |
| Incorrect or delayed diagnosis; inappropriate treatment or follow-up; loss of confidence in the device | Inadequate information provided by the manufacturer | HCP | User takes a photo of the patient's lesion. | PRS-1XU | Patient | 3 | 3 | 9 | C | The IFU explains the medical device's intended purpose The IFU explain the device's outputs The IFU contain a specific section (`User interface`) in which we explain the minimum requirements for the user interface that the ITP will implement | PRS-1XU | Technical director | IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information. | 3 | 1 | 3 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | |||
R-O5Y | Complicated instructions for use: the instructions for use are too complicated and more intricate than they need to be | Usability Regulatory | Misinterpretation of IFU |
| User discomfort and dissatisfaction; misdiagnosis; delay in diagnosis/patient's follow up/treatment | Inadequate information provided by the manufacturer, lack of clear regulatory guidance or failure to meet labeling standards | HCP | User takes a photo of the patient's lesion. | PRS-0MCPRS-1XUPRS-3YHPRS-5LJ | HCP | 4 | 3 | 12 | C | IFU has been written according to the applicable regulations: MDR 2017/745 and ISO 15223-1 | PRS-0MCPRS-1XUPRS-3YHPRS-5LJ | Technical director | IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information. | 3 | 1 | 3 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 Internal/external audits | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | |||
R-UK2 | Inadequate warnings in the IFU | Usability Regulatory | Lack of critical safety information required for the correct use of the device |
| Misdiagnosis; delay in diagnosis/patient's follow up/treatment | Inadequate information provided by the manufacturer, lack of clear regulatory guidance or failure to meet labeling standards | HCP | User takes a photo of the patient's lesion. | PRS-0MCPRS-1XUPRS-3YH | Patient | 3 | 3 | 9 | C | IFU has been written according to the applicable regulations: MDR 2017/745 and ISO 15223-1 | PRS-0MCPRS-1XUPRS-3YH | Technical director | IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information. | 3 | 1 | 3 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 Internal/external audits | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | |||
R-GTY | Instructions for use are not available at the time of use due to downtime | Usability Regulatory | User cannot consult the IFU |
| User discomfort and dissatisfaction; delays in diagnosis; treatment and follow up | IFU are only electronically available, connectivity issue, server issues | ITP | Input authorization details | PRS-1V6PRS-1XUPRS-4QWPRS-5LJ | Managing Organisation | 3 | 2 | 6 | A | If the issue is access to the internet, the use would also not be able to use the device, so there is no risk of using the device without access to the instructions. Furthermore, the IFU can be downloaded by PDF. Moreover, the IFU is hosted on a independent instance to improve the resiliency of the information system, this means that downtime in the device does not imply downtime in the IFU. The device sends messages to the user when there is any problem with the communication between the device and the user end, so the user always receives basic instructions when something is wrong. Furthermore, the procedure SP-001-001 - eIFU management explains the process to fulfil customer's request for paper IFU | PRS-1V6PRS-1XUPRS-4QWPRS-5LJPRS-9F2 | Technical director | The selection of independent instances is performed according to GP-012 Design, redesign and development IFU are verified as registered at the TEST_011_We facilitate the integration of the device into the users' system. Process to provide customers with IFU in paper format is explained in SP-001-001 - eIFU management | 2 | 1 | 2 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | |||
R-109 | Electronic instructions for use are not compatible with different devices | Usability Regulatory | Intended user cannot consult IFU |
| User discomfort and dissatisfaction; delays in diagnosis; treatment and follow up | Electronic IFU are developed in a non-universal platform or technology. | ITP | Input authorization details | PRS-0MCPRS-1V6PRS-1XUPRS-5LJPRS-7Z8 | Managing organisation | 3 | 2 | 6 | A | The electronic instructions for use are accessible via a web app that is accessible via any browser with any operating system. The instructions do not contain features, graphics or materials that are not universally accessible. It is also relevant to mention that the electronic access to the IFU is actually our recommended method of interacting with them, due to the intrinsic nature of the device Users can request IFU in paper format | PRS-0MCPRS-1V6PRS-1XUPRS-5LJPRS-7Z8 | Technical director | The selection of independent instances is performed according to GP-012 Design, redesign and development IFU are verified as registered at the TEST_011_We facilitate the integration of the device into the users' system | 2 | 1 | 2 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | |||
R-4Z5 | Lack of version control or traceability | Usability Regulatory | The ITP cannot identify the version of the device being used |
| User discomfort and dissatisfaction; delays in diagnosis; treatment and follow up | Inadequate information provided by the manufacturer | ITP | Input authorization details | PRS-1V6PRS-1XUPRS-5LJPRS-9F2 | Managing organisation | 3 | 2 | 6 | AC | We include within one of the requirements defined during the design stage that one of the outputs of the device must be the version being used and this information is included in the IFU | PRS-1V6PRS-1XUPRS-5LJPRS-9F2 | Technical director | IFU are verified as registered at the TEST_011_We facilitate the integration of the device into the users' system | 2 | 1 | 2 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable | |||
R-BXD | Insufficient knowledge to display electronic IFU | Usability Regulatory | Fail to properly display the instructions for use |
| User discomfort; dissatisfaction. Misdiagnosis; delays in diagnosis/proper treatment and worsening of the patient's health status. | Lack of information/requirement on how to access eIFU | ITP | Input authorization details | PRS-1V6PRS-1XU | Managing organisation | 3 | 3 | 9 | AC | IFU is designed in such a way that it is accessible via a dedicated and secure URL and it is also available in the website. The only requirement for accessing the eIFU is having internet connection. The users can access the IFU via any web browsers with any operations system. Upon user's request, we provide the user with IFU in paper format according to the internal procedure SP-001-001 eIFU management | PRS-1V6PRS-1XUPRS-9F2 | Technical director | IFU verification is recorded at R-TF-001-006 IFU and label validation 2023_001 and in TEST_011 to ensure that they include the information. SP-001-001 eIFU management | 3 | 1 | 3 | Acceptable | R-TF-012-015 Summative evaluation report_2024_001 | Not applicable (acceptable risk) | FALSE | TRUE | Acceptable |
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix
of the GP-001
, are:
- Author: Team members involved
- Reviewer: JD-003, JD-004
- Approver: JD-001