R-TF-028-004 Data Annotation Instructions - Non-clinical data
Table of contents
Context
This document provides the instructions on how to annotate dermatological images for non-clinical data. The annotation process is distinct from the ICD-11, Binary Indicator, and Visual Sign labeling, and involves several different tasks, including visual image quality scoring, image type categorization, and drawing bounding boxes for specific body parts. This process is fundamental for training the supervised learning models that contribute to the overall functioning of the device.
Objectives
- To create a comprehensive, multi-modal ground-truth dataset for the quantification and description of non-clinical attributes of the images.
- To provide clear, unambiguous instructions to medical annotators to ensure consistency and accuracy across all annotation tasks.
- To establish a robust foundation for training, validating, and testing the AI/ML models responsible for non-clinical data analysis.
Annotation Personnel
Role
Data annotation professional.
Qualifications
- Required: Normal visual aquity and average computer skills.
- Experience: A minimum of three years of experience in data annotation services.
Responsibilities
- To review and understand these annotation instructions thoroughly.
- To perform annotations on the provided images using the designated annotation platform.
- To complete each annotation task accurately and consistently according to the specific guidelines below.
General Annotation Workflow
All annotations will be performed using a web-based medical image annotation platform. For each image assigned, the annotator will:
- Examine the image to identify all relevant features related to the task at hand.
- Perform the annotation as per the specific instructions for that task type.
- Repeat the process until all non-clinical labels have been annotated.
Specific Annotation Instructions by Task Type
This section details the distinct annotation methodologies for each category of visual sign.
Task: Image Quality Assessment (Ordinal)
- Instruction: Select the visual quality category from the provided list that best represents the perceived visual quality of the image.
| Rating | Visual quality |
|---|---|
| 1 | Bad |
| 2 | Poor |
| 3 | Fair |
| 4 | Good |
| 5 | Excellent |
Task: Image domain (Categorical)
- Instruction: Label the image as one of the following categories:
- Clinical image
- Dermoscopy image
- Histology image
- Skin image but non-dermatology
- Out-of-domain image
Task: Head localization (Bounding Boxes)
- Instruction: If a head is visible in the image, use the bounding box tool to carefully draw a bounding box that encloses the person's head.
- Guidelines:
- The outline should be as precise as possible, tightly enclosing the skull.
- Place polygon points close enough together to accurately capture curves.
- For conditions with multiple heads (e.g. the image is a mosaic of different head views), draw a separate bounding box for each head.
Task: Extent Annotation (Polygons)
This task is for signs that cover a surface area.
- Signs to Annotate: skin vs. non-skin pixels.
- Instruction: Using the polygon tool, carefully trace the complete boundary of the target area.
- Guidelines:
- The outline should be as precise as possible, closely following the visible edge of the target area.
- Place polygon points close enough together to accurately capture curves.
Quality Control
To ensure the creation of a high-fidelity reference standard and to measure inter-observer variability, a rigorous quality control process will be followed.
- Multi-Annotator Process: To ensure the highest quality, images will be annotated by one or more qualified annotators. In the case of image quality assessment (IQA), at least 15 observers will be used for each annotation project, following the ITU-T P.910 recommendation from the International Telecommunications Union (ITU).
- Consensus Reference Standard: When multiple experts annotate the same image, the final Reference Standard label will be established by pooling their independent assessments. The method for establishing consensus will depend on the task (e.g., mean for image quality scores, voting for categories, or algorithmic fusion for bounding boxes/polygons).
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:
- Author: JD-009
- Reviewer: JD-009
- Approver: JD-005