T-012-014 Summative evaluation plan_YYYY_nnn
Objective
This study aims at validating through objective and empirical evidence that the intended users of a medical device can use the device easily and safely.
Abbreviations and definitions
- Instruction for use: in accordance with the Medical Device Regulation, the term `instructions for use refers to the information provided by the manufacturer to inform the user of a device's intended purpose and proper use and of any precautions to be taken
- Intended user: Users are defined in
IEC 62366-1:2015as person interacting with (i.e. operating or handling) the medical device. - Summative user testing: It (quantitative testing) provides an indirect assessment of the usability of the design. A group of testers are given a task, and based on their performance, the usability quotients of the design elements are measured.
- Tester: person who has been asked to perform the tasks described in the present protocol.
- eIFU : Electronic instruction for use
- IFU: Instruction for use
Introduction
Include a brief description of the company and the device object of the usability testing. Include the number of participants to evaluate usability and a brief summary of the rationale behind the number
Description of intended users, uses, environments and training
Device Identification
| Information | |
|---|---|
| Device name | Legit.Health Plus (hereinafter, the device) |
| Model and type | NA |
| Version | 1.0.0.0 |
| Basic UDI-DI | 8437025550LegitCADx6X |
| Certificate number (if available) | MDR 792790 |
| EMDN code(s) | Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software) |
| GMDN code | 65975 |
| Class | Class IIb |
| Classification rule | Rule 11 |
| Novel product (True/False) | FALSE |
| Novel related clinical procedure (True/False) | FALSE |
| SRN | ES-MF-000025345 |
Intended use
The device is a computational software-only medical device intended to support health care providers in the assessment of skin structures, enhancing efficiency and accuracy of care delivery, by providing:
- quantification of intensity, count, extent of visible clinical signs
- interpretative distribution representation of possible International Classification of Diseases (ICD) classes.
Quantification of intensity, count and extent of visible clinical signs
The device provides quantifiable data on the intensity, count and extent of clinical signs such as erythema, desquamation, and induration, among others; including, but not limited to:
- erythema,
- desquamation,
- induration,
- crusting,
- dryness,
- oedema,
- oozing,
- excoriation,
- swelling,
- lichenification,
- exudation,
- depth,
- edges,
- undermining,
- pustulation,
- hair loss,
- type of necrotic tissue,
- amount of necrotic tissue,
- type of exudate,
- peripheral tissue edema,
- peripheral tissue induration,
- granulation tissue,
- epithelialization,
- nodule count,
- papule count,
- pustule count,
- cyst count,
- comedone count,
- abscess count,
- draining tunnel count,
- lesion count
Image-based recognition of visible ICD classes
The device is intended to provide an interpretative distribution representation of possible International Classification of Diseases (ICD) classes that might be represented in the pixels content of the image.
Device description
The device is computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organisations to gather data and improve their workflows.
The generated data is intended to aid healthcare practitioners and organizations in their clinical decision-making process, thus enhancing the efficiency and accuracy of care delivery.
The device should never be used to confirm a clinical diagnosis. On the contrary, its result is one element of the overall clinical assessment. Indeed, the device is designed to be used when a healthcare practitioner chooses to obtain additional information to consider a decision.
Intended medical indication
The device is indicated for use on images of visible skin structure abnormalities to support the assessment of all diseases of the skin incorporating conditions affecting the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral and genital), the dermis, the cutaneous vasculature and the subcutaneous tissue (subcutis).
Intended patient population
The device is intended for use on images of skin from patients presenting visible skin structure abnormalities, across all age groups, skin types, and demographics.
Intended user
The medical device is intended for use by healthcare providers to aid in the assessment of skin structures.
User qualification and competencies
In this section we specificy the specific qualifications and competencies needed for users of the device, to properly use the device, provided that they already belong to their professional category. In other words, when describing the qualifications of HCPs, it is assumed that healthcare professionals (HCPs) already have the qualifications and competencies native to their profession.
Healthcare professionals
No official qualifications are needes, but it is advisable if HCPs have some competencies:
- Knowledge on how to take images with smartphones.
IT professionals
IT professionals are responsible for the integration of the medical device into the healthcare organisation's system.
No specific official qualifications are needed, but it is advisable that IT professionals using the device have the following competencies:
- Basic knowledge of FHIR
- Understanding of the output of the device.
Use environment
The device is intended to be used in the setting of healthcare organisations and their IT departments, which commonly are situated inside hospitals or other clinical facilities.
The device is intended to be integrated into the healthcare organisation's system by IT professionals.
Operating principle
The device is computational medical tool leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures.
Body structures
The device is intended to use on the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral and genital), the dermis, the cutaneous vasculature and the subcutaneous tissue (subcutis).
In fact, the device is intended to use on visible skin structures. As such, it can only quantify clinical signs that are visible, and distribute the probabilities across ICD classes that are visible.
Intended use environment
Description of the intended use environment during actual use. The intended use environments might vary in ways that impact users' ability to use the device safely and effectively. Environmental variables that may affect device use safety and effectiveness are the following:
- Distractions
- Lighting conditions
- Background noise
Training
Define whether the manufacturer will perform training to the users before the use of the device.
Summary of known use problems
We conducted research on known or expected use problems for similar products and product types and previous models of the device. Based on this research, the following use problems have been identified to pertain to the device:
- complete the list
Analysis of hazards associated with the use of the device
We conducted a task analysis on the use of the device which led to a comprehensive use related risk analysis and list of potential use errors. This process involved analyzing known use problems with similar devices, identifying the user interface characteristics related to safety, identifying potential use errors, and identifying known and foreseeable hazards and hazardous situations.
Include a table coming from the risk management record with the hazards related to usability.
Details of summative evaluation testing
Overall study approach
The study will be conducted on {Insert date}, at {Insert location} and consists of the simulated use of device by representative users.
Insert a general description of the study design: timing, methods, description of the tasks to be performed by the testers.
Participants or testers
The study will include insert number of participants participants who are representative of the intended users.
The table below describes the user's groups information.
| User Group | Number of participants | Characteristics |
|---|---|---|
Study device and materials
The device used in this study will be a production equivalent device (including the electronic IFU). The device is commercially representative and ensures that the performance replicates actual device performance.
Electronic IFU version
Insert the version of eIFU that will be used for the summative evaluation testing
Use scenarios and tasks
All participants will complete all use scenarios listed and described in the table below.
| Use scenario | Task | Success criteria |
|---|---|---|
As the manufacturer expects that users will not receive training on the device prior to use, this study will only include untrained participants to reflect the conditions of real use of the device.
Data coding
The data to be collected are as follows:
- Observational Data: Participants will be given an opportunity to use the devices independently and in as realistic a manner as possible, without guidance, coaching, praise, or critique from the study moderator.
- Interview Data: The study moderator will ask participants open-ended questions to understand any difficulties they may have with the test devices and tasks.
- Subjective assessment of use: This data consists of participants' subjective feedback on the root cause of potentially safety-related use errors and use difficulties, and opinions on all tasks for any residual risk, as well as the adequacy of instructions.
The following coding system will be used for critical simulated use and knowledge task data collection and analysis:
- Success (OK): This coding describes a situation in which the participant correctly performs a task, or answers a knowledge task question, according to the steps in the IFU or an acceptable alternative.
- Use error (UE): This coding describes a situation in which the participant performs a task incorrectly (i.e., not in accordance with the intended manner of use), does not complete a necessary task, or answers a knowledge task question incorrectly or incompletely.
- Close call (CC): This coding describes a situation in which the participant performs or nearly performs a task incorrectly (i.e., not in accordance with the intended manner of use), or answers a knowledge task question incorrectly or incompletely, but then resolves the issue prior to any harm being done without assistance.
- Use difficulty (UD): This coding describes a situation in which the participant struggles to some extent while completing a task or answering a knowledge task question (e.g., confusion, taking longer than expected, difficulty manipulating a device's components), but eventually can completes the task or answers the question successfully without assistance.
- Not applicable (NA): This coding is used if a question or task do not apply to a scenario or specific situation during the testing session.
Recording equipment
Insert information of recording equipment in the case the test will be recorded.
Data analysis
The collected data will be analyzed using the definitions of success criteria of each task presented in the table above, to determine if the study objectives have been met.
During this analysis, all observations of the tasks coded as anything other than “OK”, including moderator observations and participant comments, as well as feedback from interviews with participants regarding device use, critical tasks, use errors, and problems (as applicable), will be evaluated.
Description and analysis of all use errors, close calls, and use difficulties that could cause serious harm, root causes of the problems, and implications for additional risk elimination or reduction will be used to determine if there are patterns of use events that pose unacceptable risk to the patient or user.
Evaluation form
Describe the evaluation form's content and attached it as annex of this protocol.
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:
- Author: Team members involved
- Reviewer: JD-003, JD-004
- Approver: JD-001
Template signature meaning
Delete this section when you create a new record from this template.
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:
- Author: Team members involved
- Reviewer: JD-003, JD-004
- Approver: JD-001