R-TF-028-004 Data Annotation Instructions - Visual Signs
Table of contents
Context
The Legit.Health Plus device is intended to provide quantifiable data on the intensity, count, and extent of a wide range of visible clinical signs[cite: 551]. To develop and validate the AI/ML algorithms that perform this quantification, a high-quality, ground-truth dataset is required.
This document provides the instructions for medical experts on how to annotate dermatological images for these specific visual signs. The annotation process is distinct from the ICD-11 and Binary Indicator labeling and involves several different tasks, including assigning intensity scores, drawing bounding boxes for counting, and creating polygons to define extent. This process is fundamental for training the supervised learning models.
Objectives
- To create a comprehensive, multi-modal ground-truth dataset for the quantification of visual clinical signs.
- To provide clear, unambiguous instructions to medical annotators to ensure consistency and accuracy across all annotation tasks.
- To establish a robust foundation for training, validating, and testing the AI/ML models responsible for quantifying intensity, count, and extent.
Annotation Personnel
Role
Medical expert (e.g., Dermatologist, Wound Care Specialist) responsible for providing ground-truth annotations for visual sign intensity, lesion count, and lesion extent.
Qualifications
- Required: Board-certified specialist relevant to the pathology being assessed (e.g., Dermatology, Plastic Surgery).
- Experience: A minimum of three years of clinical experience in assessing and managing relevant skin conditions is required to ensure expertise.
Responsibilities
- To review and understand these annotation instructions thoroughly.
- To perform annotations on the provided images using the designated annotation platform.
- To apply clinical expertise to complete each annotation task accurately and consistently according to the specific guidelines below.
General Annotation Workflow
All annotations will be performed using a web-based medical image annotation platform. For each image assigned, the annotator will:
- Examine the image to identify all relevant clinical signs present.
- For each sign identified, select the corresponding annotation task from the tool's interface.
- Perform the annotation as per the specific instructions for that task type (Intensity, Count, or Extent).
- Repeat the process until all visible and relevant signs in the image have been annotated.
Specific Annotation Instructions by Task Type
This section details the distinct annotation methodologies for each category of visual sign.
Task: Intensity Annotation (Ordinal & Categorical)
This task is for signs where severity is assessed on a scale or by category.
- Signs to Annotate:
- Ordinal Scale (e.g., 0-4): Erythema, Desquamation, Induration, Crusting, Xerosis (dryness), Swelling (oedema), Oozing, Excoriation, Lichenification, Wound stage.
- Categorical (Non-Ordinal): Edge characteristics (5 types), Tissue types (5 types), Exudate types (4 types), Wound bed tissue (5 types), Perif. features and Biofilm-Comp. (3 types).
- Instruction: For each sign present in the image, select the numerical value or descriptive category from the provided list that best represents its intensity or type.
- Guidelines & Example (Ordinal - Erythema): The platform will present a scale. Detailed clinical scoring guides based on established literature will be provided as a reference.
| Score | Description |
|---|---|
| 0 | None |
| 1 | Mild, faint redness |
| 2 | Moderate, clear redness |
| 3 | Severe, deep redness |
| 4 | Very severe, fiery redness |
Task: Count Annotation (Bounding Boxes)
This task is for discrete, countable lesions.
- Signs to Annotate: Nodule, Papule, Pustule, Cyst, Comedone, Abscess, Draining Tunnel, Inflammatory Lesion.
- Instruction: Using the bounding box tool, draw a tight rectangle around each individual instance of the specified sign.
- Guidelines:
- The box must contain the entire visible lesion with as little background as possible.
- Overlapping lesions should be boxed individually if they are clinically distinguishable as separate entities.
- Ensure every instance of the target sign within the image is annotated.
Task: Extent Annotation (Polygons)
This task is for signs that cover a surface area.
- Signs to Annotate: Hair Loss, Scalp, Wound Bed, Granulation Tissue, Biofilm/Slough, Necrosis, Maceration, Orthopedic Material, Bone/Cartilage/Tendon, Erythema extent, Hypopigmentation, Hyperpigmentation, Nail Plate, Nail Lesion.
- (Note: Some signs like Erythema may also be annotated for extent in specific contexts, which will be clearly indicated in the task instructions).
- Instruction: Using the polygon tool, carefully trace the complete boundary of the area affected by the specified sign.
- Guidelines:
- The outline should be as precise as possible, closely following the visible edge of the sign.
- Place polygon points close enough together to accurately capture curves.
- For conditions with multiple, non-contiguous patches, draw a separate polygon for each distinct area.
Task: Image Categorization (Categorical)
This task is for assigning a single categorical label to an entire image based on its overall characteristics or clinical presentation.
-
Signs to Annotate: Hurley Staging, inflammatory pattern, folicular pattern.
-
Instruction: For each image, carefully examine the overall clinical presentation and select the single classification label that best represents the image as a whole.
-
Guidelines:
- Evaluate the complete image to determine the appropriate classification.
- Refer to established clinical criteria and diagnostic guidelines to differentiate between adjacent severity levels.
- When uncertain between two categories, select the label that best captures the predominant clinical feature visible in the image.
Quality Control
To ensure the creation of a high-fidelity reference standard and to measure inter-observer variability, a rigorous quality control process will be followed.
- Multi-Annotator Process: To ensure the highest quality, images will be annotated by one or more qualified medical experts. Based on expert opinion regarding the complexity and subjectivity of each specific annotation task (e.g., intensity of erythema vs. counting nodules), a lead medical expert will determine the required number of annotators to establish a reliable consensus.
- Consensus Reference Standard: When multiple experts annotate the same image, the final Reference Standard label will be established by pooling their independent assessments. The method for establishing consensus will depend on the task (e.g., mean/median for intensity scores, voting for categories, or algorithmic fusion for bounding boxes/polygons).
- Senior Review: A designated senior specialist will review cases with high inter-annotator disagreement to resolve discrepancies and ensure the final label is clinically sound.
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:
- Author: JD-009
- Reviewer: JD-009
- Approver: JD-005