REQ_008 Notify the user if the image does not represent a skin structure

Category

Minor

Source

• Taig Mac Carthy, Design & Development Manager

INTERNAL SYSTEM INPUTS AND OUTPUTS ALARMS, WARNINGS AND MESSAGES USER DATABASE AND DATA DEFINITION ARCHITECTURE

Activities generated

• MDS-451

Causes failure modes

• The AI model mistakenly identifies non-skin structures as skin structures.
• The AI model is too sensitive, flagging images incorrectly as non-skin structures.
• The AI model is not sensitive enough, missing non-skin structures in images.
• Notifications are not clear or specific enough for the user to understand why the image was flagged, causing confusion about what is wrong with the image.
• The algorithm has been trained on a limited dataset, making it less effective at accurately identifying non-skin structures.

Related risks

• 9. Image artefacts/resolution: the medical device receives an input that does not have sufficient quality in a way that affects it performance
• 30. The user is unable to provide adequate lighting conditions
• 37. Inaccurate training data: image datasets used in the development of the device are not properly labeled.
• 38. Biased or incomplete training data: image datasets used in the development of the device are not properly selected
• 54. The device inputs images that do not represent skin structure
• 61. Stagnation of model performance: the AI/ML models of the device do not benefit from the potential improvement in performance that comes from re-training
• 62. Degradation of model performance: automatic re-training of models decreases the performance of the device

User Requirement, Software Requirement Specification and Design requirement

User Requirement 8.1: Users should be notified when uploaded images do not represent a skin structure.

Software Requirement Specification 8.2: Implement image analysis algorithms to determine whether an uploaded image represents a skin structure.

Design Requirement 8.3: Device responses should include a key-value pair indicating whether the uploaded image represents a skin structure, utilizing standardized terminology and coding.

Description

Many deep learning algorithms, particularly convolutional neural networks, excel at performing specific tasks. For instance, they can proficiently find the distribution of the potential ICD categories within the pixels of an image or identify visual indicators like erythema. However, it's important to note that they are restricted to these predefined tasks. Consequently, when presented with an image of a non-skin-related subject, these algorithms may attempt to generate an ICD distribution or detect erythema in an inappropriate context. While this behavior aligns with their standard operation, it is preferred to limit their application to images containing skin structures, as users may inadvertently submit images unrelated to this context.

To enhance the user experience and prevent such errors, we are in the process of developing a computer vision algorithm designed specifically for detecting skin structures. This algorithm will evaluate an image and provide a likelihood score, ranging from 0 to 100, indicating the probability of a skin structure being present within the image.

Dataset

To achieve our objective, we employ an extensive dataset comprising skin pathology images, recognized for their inclusion of skin structures. Additionally, we incorporate other non-dermatology datasets such as the ImageNet and MSCOCO datasets, which encompass a diverse array of everyday objects, ranging from vehicles to animals. Furthermore, we introduce texture data, as plain colors resembling skin tones can potentially be mistaken for close-up images of skin.

Methodology

Given the inherently manageable nature of this task, which typically experiences minimal failure rates, our approach focuses on optimizing speed. To accommodate this, we have opted for compact Convolutional Neural Network (CNN) architectures, prioritizing efficiency. It's worth noting that our model will operate continuously, independently of any additional functionalities requested by the user. In alignment with current industry standards, we will also employ a streamlined transformer model that delivers high performance.

Success metrics

The first objective is crucial for integrating the output seamlessly into the medical device. The second objective focuses on performance of the computer vision algorithm designed for detecting skin structures. Given the task's complexity, high performance is anticipated, as the process is considered relatively straightforward and executable by non-specialists. An AUC of 0.8 or higher is regarded as excellent.

Goal	Metric
Users know if the image contains a skin structure	Users receive data of the presence of a skin structure
Skin structures are detected with proficiency	AUC > 0.8

Previous related requirements

• REQ_005
• REQ_007

Signature meaning

The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:

• Tester: JD-017, JD-009, JD-005, JD-004
• Approver: JD-003