R-TF-012-008 Formative evaluation report_2023_001
Characterization of the medical device
Sofware version
The software version that was tested is version 2.1 of the legacy device.
As explained in the R-TF-012-007 Formative evaluation plan, we conducted the formative evaluation test with the legacy device (API only) because it is the prototype of the current version of the API, being the core functionality of the API in the legacy device identical to that in the current version.
Intended medical indication
The device is a software-only medical device leveraging computer vision to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organisations to gather data and improve their workflows.
Intended patient population
The device is intended for use on images of skin from patients presenting visible skin structure abnormalities, across all age groups, skin types, and demographics.
Regarding assessing facial palsy, the device is intended for use on patients presenting facial nerve injury.
Intended part of the body or type of tissue applied to or interacted with
The device is intended to be used on the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral and genital), the dermis, the cutaneous vasculature and the subcutaneous tissue (subcutis).
Regarding assessing facial palsy, the device is intended to be used specifically on the face.
Intended user
The intended users are healthcare organisations and their stakeholders, including Health Care Practitioners (HCP).
Regarding usability, the Information Technology professionals who integrate the device are considered a relevant stakeholder of the managing organisation.
Intended use environment
The device is intended to be used in the setting of healthcare organisations and their IT departments, which commonly are situated inside hospitals or other clinical facilities, although not always.
The device should never be used to confirm a clinical diagnosis. On the contrary, its result is one element of the overall clinical assessment. Indeed, the device is designed to be used when a healthcare practitioner chooses to obtain additional information to make a decision.
Operating principle
The device is computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures.
The device is accessed via API programmatically, and integrated into software such as Electronic Health Records. As such, the "interface" comprises the endpoints, documentation, error messages, and data formats of the API. In other words: there are no visual user interfaces with buttons and images: the device is an API that receives JSON files, and outputs JSON files.
Presence of user interface of unknown provenance
There is no software of unknown provenance in the user interface.
User interface characteristics related to safety and potential use errors
The user interface of the device comprises 3 elements:
- API Endpoint: URL structures, expected HTTP methods, and status codes.
- API Documentation: IFU, descriptive guides, use cases, sample requests, and responses.
- Data Structure: JSON payload formats, including field names and FHIR nomenclature.
1. API Endpoint
Regarding safety and potential use errors, API Endpoint has the following aspects to consider:
- URL structures
- Expected HTTP methods
- Status codes.
2. API documentation
Regarding safety and potential use errors, API documentation has the following aspects to consider:
- Descriptive guides, use cases
- Sample requests and responses.
Our medical device includes an OpenAPI Specification.
OpenAPI Specification (formerly known as Swagger Specification) is an API description format for REST APIs. An OpenAPI file allows you to describe an entire API, including:
- Available endpoints and operations on each endpoint (GET, POST)
- Operation parameters Input and output for each operation
- Authentication methods
- Contact information, license, terms of use and other information.
This means that our API itself has embedded specifications that help the user understand the type of values that are transmitted by the API.
3. Data Structure
Regarding safety and potential use errors, Data Structure has the following aspects to consider:
- JSON payload formats
- Field names (FHIR nomenclature).
FHIR is a standard for healthcare data exchange, published by HL7. FHIR is suitable for use in a wide variety of contexts: mobile phone apps, cloud communications, EHR-based data sharing, server communication in large institutional healthcare providers, and much more.
FHIR solves many challenges of data interoperability by defining a simple framework for sharing data between systems.
Method for formative evaluation
The method applied for the formative evaluation consists of making the testers perform an integration of the device and answer a questionnaire at the same time.
Questionnaires
Questionnaires are considered one of the most effective tools for gathering information about usability because they provide a structured format for collecting feedback. This ensures consistency in the type of information gathered across different users or sessions. Furthermore, questionnaires provide direct insight into the user's perspective, capturing their experiences, difficulties, and satisfaction.
The questionnaire is designed with the user in mind, focusing on their experience, challenges, and satisfaction. This user-centric approach ensures that the device is not just technically sound but also user-friendly.
Each section of the questionnaire is dedicated to one of the three key usability scenarios. This ensures that feedback is collected systematically, allowing for targeted improvements in each area. By focusing on specific scenarios, participants can provide concrete examples and detailed feedback, which is more valuable than general impressions.
User feedback
User feedback is the most direct and reliable source of information about usability, ensuring that the device meets the needs and expectations of its users.
The questionnaire collects both quantitative (ratings) and qualitative (comments) data. Quantitative data provides a clear and objective measure of the user's experience, while qualitative data provides context and insight into the reasons behind their ratings. The combination of these two types of data ensures a balanced view of the device's usability, capturing both the severity of any issues and the nuances of the user's experience.
Rating scale
The use of a 1-5 rating scale allows for a nuanced assessment of each aspect of usability. It provides a standardized way of measuring user satisfaction and perceived ease of use, making comparisons and trend analysis possible over time or across different user groups.
The scale provides a quantitative measure of usability, which can be easily analyzed and used to track improvements or identify areas needing attention.
Prompting errors
By including a section on error handling, the questionnaire specifically addresses how well the system communicates and allows recovery from errors, which is a crucial aspect of usability, especially for a medical device.
Feedback on error messages and recovery mechanisms provides direct insight into the resilience and user-friendliness of the system under adverse conditions.
Testing environment
By replicating the real-world integration environment, simplifying the test execution, focusing on direct API interaction, encouraging troubleshooting, and mimicking real-world application scenarios, the test conditions and environment are well-aligned with the actual conditions of use. This ensures that the testing yields accurate, relevant, and actionable insights, directly contributing to the improvement of the device's usability in real-world scenarios.
By directly interacting with the API, participants are engaging with the device in the same manner as they would in a real-world scenario. This direct interaction provides a clear and straightforward evaluation of the device's usability, focusing specifically on the API's functionality, reliability, and ease of use.
Furthermore, users are instructed to create an environment similar to the one in which they would integrate the device, using tools commonly employed for such tasks. This approach ensures that users are testing the device in an environment that closely mirrors their real-world usage scenarios, accounting for any potential system-specific issues or integration challenges that might arise.
About the testers
The participants for the formative evaluation test were selected according to the user group and profile described in the R-TF-012-007 Formative evaluation plan_2023_001. They are professionals with technical expertise in integrating APIs into their systems, and they occupy positions in care provider organizations.
The profile of these participants aligns closely with the intended user profile for a cloud medical device accessed via API. Healthcare organizations looking to integrate such a device would rely on their Information Technology (IT) professionals to handle the integration and ensure seamless communication between the device and their existing systems, such as Electronic Health Records (EHR) and Electronic Medical Records (EMR). Thus, selecting participants who are familiar with API integration and work within care provider organizations ensures relevance and accuracy in the usability evaluation.
Their positions within care provider organizations ensure that they understand the unique requirements and challenges of integrating medical devices within healthcare settings. This is crucial for ensuring that the device meets the specific needs of healthcare organizations.
In conclusion, the selected participants are highly representative of the intended user profile, possessing the necessary technical expertise and healthcare context to provide valuable feedback on the device's usability. Their familiarity with API integration and their roles within care provider organizations ensure that the test conditions align closely with the real-world use environment. By grouping them based on their technical roles and healthcare contexts, the test captures a diverse and relevant set of perspectives, ensuring comprehensive and actionable insights for improving the device's usability.
Safety considerations
We determined that delivering the device via an HTTPS API that follows the REST protocol was the most minimal and safest way of allowing healthcare organisations access to the device.
The REST protocol separates the user interface from the server and the data storage, and generates the minimal exposure of data. Thanks to this, REST API always adapts to the type of syntax or platforms that the user may use, which gives considerable freedom and autonomy to the user, while at the same time reducing security risks. With a REST API, the user can use either PHP, Java, Python or Node.js servers, and the connection happens server-to-server, fully encrypted. As a result, the safety of the device increased by design.
The only indispensable thing is that the requests and the responses must always take place in the language used for the information exchange: JSON. This also means that instead of a visual user interface, users integrate the API into their systems. This carries some dedicated usability concerns, which we cover in this report.
Requirements of the Design and development process that relate to usability
These are the requirements of the Design and development process that relate to usability:
- REQ_005_The user can send requests and get back the output of the device as a response in a secure, efficient and versatile manner
- User Requirement 5.1: Users shall be able to send secure and efficient requests to the device and receive outputs promptly.
- Software Requirement Specification 5.2: The device shall be designed to handle multiple concurrent requests and respond with minimal latency, ensuring data privacy and integrity during transmission.
- Design Requirement 5.3: Design the device endpoints in a RESTful manner, ensuring straightforward access and response management, while implementing HTTPS for secure communication.
- REQ_006_The data that users send and receive follows the FHIR healthcare interoperability standard
- User Requirement 6.1: Users should be able to send and receive data adhering to FHIR healthcare interoperability standards.
- Software Requirement Specification 6.2: Implement FHIR data standards in device's data processing, ensuring all inputs and outputs comply with this standard.
- Design Requirement 6.3: Device responses and requests shall be structured according to the FHIR standard, utilizing standardized healthcare data objects and attributes.
- REQ_007_If something does not work, the API returns meaningful information about the error
- User Requirement 7.1: Users shall be notified with meaningful error messages when an issue arises.
- Software Requirement Specification 7.2: Implement error-handling mechanisms to generate user-friendly error messages for all potential issues during device interactions.
- Design Requirement 7.3: Design the error response format to be consistent and self-explanatory, using clear key naming and utilising HTTP status codes effectively.
- REQ_012_We facilitate the integration of the into the users' system
- User Requirement 12.1: Users should find it straightforward to integrate the device into their existing systems.
- Software Requirement Specification 12.2: The device should be designed and documented in such a way that facilitates straightforward integration into various systems, with comprehensive documentation.
- Design Requirement 12.3: Ensure the device design adheres to RESTful principles and industry best practices, providing clear endpoint documentation and example requests/responses.
Formative evaluation results
Context
To contextualize the results of the evaluation and allow the reproducibility of the process, the setup, elements and considerations involved in the test are presented below:
- Period of evaluation: The device was tested between the 8th of may 2023, and 29th of September 2023.
- Participants: The users who performed the tests are:
- Josetxo Susperregui
- Eñaut Rojo
- Victor Li
- Maxime Lagreze
- Danielle Beneduci
- Ramón Chalavera
- Device version: 2.1
- IFU version: The version of the IFU used for this evaluation is available in our version control repository: 58be642026638197ce60a600c6dd0f4d4e01befa. To allow external reviewers to see it, we have deployed a live view of this commit in the URL https://ifu-58be642.legit.health/.
The users were selected by their technical expertise, and their positions in care provider organisations, according to the user group and profile described in R-TF-012-007 Formative evaluation plan_2023_001.
There was no difference between the test environment and the real-world environment. Users chose their preferred method to replicate the real-world environment.
Summary of results and insights
The following table shows the results of the first formative evaluation.
| # | Scenario | Avg. Score | Test passed |
|---|---|---|---|
| 1 | Success integrating | 1 | TRUE |
| 2 | Ease of integration | 4.83 | TRUE |
| 3 | Success sending data | 5 | TRUE |
| 4 | Success receiving data | 5 | TRUE |
| 5 | Full integrity of the data | 5 | TRUE |
| 6 | 'Data Missing' error shows relevant message | 1 | TRUE |
| 7 | 'Access denied' error shows relevant message | 1 | TRUE |
| 8 | 'Quality Image' error shows relevant message | 1 | TRUE |
According to what we established in the R-TF-012-007 Formative evaluation plan_2023_001 since the tests performed in the first formative evaluation are all successfully passed, no further iteration of formative evaluation is needed.
Conclusions
The device seems to be easy to use by all users. This is not surprising due to the simplicity of the device and its level of standardisation. The good thing with an API is that all APIs work pretty much in the same way.
Record signature meaning
- Author: JD-004
- Review: JD-003
- Approver: JD-005