R-TF-028-004 Data Annotation Instructions - Binary Indicator Mapping

Table of contents

• Context
  • Reference Standard for ICD Category Distribution
  • Derivation of Binary Indicators
  • Purpose of this Document
• Objectives
• Annotation Personnel
  • Primary Annotator
  • Secondary Reviewer
• Annotation Protocol
  • ICD-11 Category Extraction and Scope Definition
  • Annotation Worksheet Preparation
  • Primary Clinical Annotation
    • Clinical Decision Criteria for Each Binary Indicator:
  • Independent Secondary Review
  • Consensus Resolution and Finalization
  • Technical Validation and Version Control
    • Data Schema Specification
• Quality Control and Validation
  • Clinical Quality Assurance
  • Regulatory and Documentation Standards
  • Technical Validation
  • Post-Deployment Monitoring
  • Approval and Sign-Off

Context

The Legit.Health Plus device incorporates multiple AI/ML algorithms to support clinical workflows, including diagnostic support and severity assessment. For clinical decision support specifically, the device provides two complementary types of outputs:

1. ICD Category Distribution: Multi-class probability distribution across dermatological conditions.
2. Binary Indicators: Clinical decision support flags for malignancy, premalignancy, association to malignancy, pigmented lesion status, urgent referral, and high-priority referral.

Reference Standard for ICD Category Distribution

The reference standard labels for the ICD Category Distribution algorithm originate directly from source datasets (retrospective archive data and prospectively collected custom data) where clinical labels were established by qualified dermatologists. The processing and quality control of these clinical labels is detailed in R-TF-028-004 Data Annotation Instructions - ICD-11 Mapping. No additional clinical annotation is required for these primary clinical categories.

Derivation of Binary Indicators

The Binary Indicators are not directly predicted by the AI model. Instead, they are deterministically derived from the ICD Category Distribution output through a post-processing step. Specifically, the probability for each binary indicator is computed by summing the predicted probabilities of all relevant ICD-11 categories, as defined by a clinically validated mapping matrix. More specifically, for $N$ categories and 6 indicators, the mapping matrix has a size of $N \times 6$, and the computation of each indicator $j \in [1, 2, \ldots, 6]$ is defined as:

$$\text{Binary Indicator}_j = \sum_{i=1}^{n} \big(M_{ij} \times p_i\big)$$

where $p_i$ is the probability for the $i$-th ICD-11 category, and $M_{ij}$ is the binary weight coefficient ($M_{ij} \in [0, 1]$) that indicates whether category $i$ contributes to indicator $j$.

This approach ensures that binary indicators inherit the probabilistic outputs of the primary clinical model while maintaining clinical interpretability and traceability.

Purpose of this Document

This document defines the formal protocol for creating and validating the ICD-to-Binary-Indicator Mapping Matrix, which serves as the deterministic logic governing the derivation of binary indicators from ICD-11 predictions. The mapping will be established by qualified dermatologists based on established medical literature, clinical guidelines, and expert consensus.

Objectives

The primary objectives of this mapping matrix creation procedure are:

• Clinical Validity: To establish a definitive, evidence-based mapping matrix that formally links every unique ICD-11 category in the device's clinical scope to each of the six binary clinical indicators, ensuring alignment with current dermatological knowledge and clinical practice guidelines.

• Regulatory Traceability: To produce a version-controlled, formally approved artifact that defines the deterministic post-processing logic for deriving binary indicators from ICD-11 probability distributions, thereby establishing the reference standard for all subsequent validation and testing activities.

• Data Science Implementation: To provide the data science team with a clinically validated, machine-readable specification that can be directly implemented in the device software and used to generate reference standard labels for model validation datasets.

• Quality Assurance: To ensure the mapping is internally consistent, clinically justifiable, and supported by appropriate medical literature references, with formal consensus review by independent dermatological experts.

This mapping matrix is a critical component of the device's algorithm specification and is referenced in R-TF-028-001 AI/ML Description as the definitional basis for binary indicator outputs.

Annotation Personnel

Primary Annotator

Role: Lead Medical Annotator (Board-Certified Dermatologist)

Minimum Qualifications:

• Board certification in Dermatology from a recognized medical authority (e.g., European Board of Dermatology and Venereology, American Board of Dermatology, or equivalent).
• Minimum 5 years of post-certification clinical experience in general dermatology.
• Demonstrated expertise across multiple dermatological domains: inflammatory dermatoses, infectious diseases, benign and malignant neoplasms, and pigmented lesion assessment.

Preferred Qualifications:

• Experience in dermoscopy and skin cancer assessment.
• Academic or clinical research experience in dermatological diagnosis or classification.
• Familiarity with ICD-11 coding and medical device regulatory requirements (desirable but not mandatory).

Responsibilities:

• Conduct a comprehensive clinical review of all unique ICD-11 categories present in the device's clinical scope.
• Assign binary (TRUE/FALSE) values for each of the six clinical indicators for every ICD-11 category based on established medical knowledge, clinical guidelines, and dermatological literature.
• Provide written clinical justification with literature references for all classifications, particularly for ambiguous or borderline cases.
• Participate in consensus discussions with the secondary reviewer to resolve any discrepancies.

Secondary Reviewer

Role: Independent Quality Reviewer (Board-Certified Dermatologist)

Minimum Qualifications:

• Same minimum qualifications as the Primary Annotator.
• Must be independent (not involved in the initial annotation process).

Responsibilities:

• Conduct an independent review of the completed mapping matrix.
• Identify any clinically questionable classifications or internal inconsistencies.
• Participate in consensus discussions with the primary annotator to resolve discrepancies.
• Provide formal approval of the final consensus matrix.

Annotation Protocol

The creation and validation of the mapping matrix follows a structured, quality-controlled process with clear roles, responsibilities, and deliverables at each stage.

ICD-11 Category Extraction and Scope Definition

Responsible Party: Medical Data Science (MDS) Team

Activities:

1. Process the final, curated training dataset (LegitHealth-DX) to extract the complete set of unique ICD-11 categories present across all clinical labels.
2. Include both primary conditions and differential conditions to ensure comprehensive coverage.
3. Verify that all extracted ICD-11 codes are valid according to the official ICD-11 classification (WHO).
4. Document the frequency distribution of each category to provide context for the clinical reviewers.

Deliverable: A validated list of unique ICD-11 categories with codes and prevalence statistics, ready for clinical annotation.

Annotation Worksheet Preparation

Responsible Party: MDS Team

Activities:

1. Prepare a structured annotation worksheet (spreadsheet format: CSV or Excel) containing:

   • ICD-11 Code (e.g., "2F20")
   • ICD-11 Category Name (e.g., "Malignant melanoma of skin")
   • Frequency in Dataset (number of cases)
   • Binary Indicator Columns (6 columns): malignancy, premalignancy, association_to_malignancy, pigmented_lesion, urgent_referral, high_priority_referral
   • Clinical Justification (text field for annotator comments and literature references)
   • Reviewer Notes (text field for secondary reviewer comments)

2. Provide a comprehensive annotation instruction document with clinical definitions and decision criteria (see Section 4.3).

3. Conduct a briefing session with the Primary Annotator to clarify objectives, methodology, and expected deliverables.

Deliverable: Completed annotation worksheet template and instruction document, approved by both MDS team and Primary Annotator.

Primary Clinical Annotation

Responsible Party: Primary Annotator (Lead Dermatologist)

Activities:

The Primary Annotator conducts a systematic, row-by-row review of the annotation worksheet. For each ICD-11 category, a binary classification (TRUE/FALSE or 1/0) is assigned for each of the six clinical indicators according to the decision criteria defined below.

General Annotation Principles:

• All classifications must be based on the inherent clinical characteristics of the condition as defined by the ICD-11 category, not on specific case presentations or individual patient factors.
• When clinical evidence is ambiguous or controversial, the annotator should apply the principle of clinical caution (favoring sensitivity over specificity for safety-critical indicators such as malignancy or urgent referral).
• All borderline or clinically complex classifications require written justification with references to clinical guidelines or peer-reviewed literature.

Clinical Decision Criteria for Each Binary Indicator:

In this section, we describe each binary indicator and provide some example of ICD categories with their corresponding binary indicator annotations.

Malignancy

Definition: Conditions representing invasive malignant neoplasms with capacity for local tissue destruction, metastasis, and mortality if untreated.

Assign TRUE for:

• All invasive malignant skin cancers: melanoma (ICD-11: 2F20), squamous cell carcinoma (2F22), basal cell carcinoma (2F21), Merkel cell carcinoma (2F26), etc.
• Malignant skin manifestations of systemic cancers (e.g., cutaneous metastases).

Assign FALSE for:

• All benign neoplasms (e.g., seborrheic keratosis, dermal nevus).
• Premalignant lesions (classified separately; see below).
• Inflammatory, infectious, or other non-neoplastic conditions.

Clinical Reference: WHO Classification of Skin Tumours, AJCC Cancer Staging Manual.

Premalignancy

Definition: Conditions with established potential for malignant transformation but not currently invasive, including in situ carcinomas and dysplastic precursor lesions.

Assign TRUE for:

• Actinic keratosis (EK90.0)
• Bowen's disease / squamous cell carcinoma in situ (2F22.Z)
• Lentigo maligna (melanoma in situ, 2F20.0)
• Dysplastic (atypical) nevi with documented atypia (not all melanocytic nevi)

Assign FALSE for:

• Benign lesions with no malignant potential (e.g., common acquired nevus, seborrheic keratosis).
• Already invasive malignancies (classified under malignancy).

Clinical Reference: European consensus on actinic keratosis, NCCN Guidelines for melanoma risk.

Association to Malignancy

Definition: Benign or inflammatory conditions that may clinically mimic malignancy, frequently appear in differential diagnoses of malignant lesions, or present diagnostic ambiguity requiring careful clinical or histopathological exclusion of malignancy.

Assign TRUE for:

• Atypical or changing melanocytic nevi (common melanoma mimics).
• Pigmented seborrheic keratoses (melanoma mimics).
• Pigmented basal cell carcinoma (diagnostic overlap).
• Spitz nevus (histologic mimicry of melanoma).
• Inflamed or irritated benign lesions that may appear atypical clinically.

Assign FALSE for:

• Conditions with no clinical or dermoscopic overlap with malignancy (e.g., milia, viral warts, classic psoriasis).

Clinical Rationale: This indicator supports clinical decision-making by flagging conditions that may require closer follow-up or biopsy to definitively exclude malignancy, even if the condition itself is benign.

Pigmented Lesion

Definition: Lesions characterized by abnormal melanin deposition or melanocyte proliferation, forming the clinical subset relevant for melanoma risk stratification and pigmented lesion-specific diagnostic pathways.

Assign TRUE for:

• All melanocytic lesions (benign nevi, dysplastic nevi, melanoma).
• Pigmented non-melanocytic lesions commonly in differential diagnoses (pigmented basal cell carcinoma, seborrheic keratosis, dermatofibroma with pigmentation, pigmented actinic keratosis).
• Lentigines, ephelides (freckles), and other pigmented macules.

Assign FALSE for:

• Non-pigmented inflammatory conditions (e.g., eczema, psoriasis).
• Non-pigmented neoplasms (e.g., non-pigmented basal cell carcinoma, squamous cell carcinoma).
• Vascular lesions (e.g., hemangiomas, unless pigmented).

Clinical Reference: Consensus guidelines on pigmented lesion management, International Dermoscopy Society criteria.

Urgent Referral

Definition: Conditions requiring immediate or urgent dermatological assessment within 48 hours due to high risk of rapid progression, severe morbidity, or life-threatening complications if delayed.

Assign TRUE for:

• Suspected or confirmed melanoma (especially nodular or ulcerated presentations).
• Rapidly growing or bleeding malignancies (e.g., advanced SCC, BCC with extensive tissue destruction).
• Acute severe skin infections suggesting systemic involvement (e.g., necrotizing fasciitis, severe cellulitis).
• Suspected cutaneous vasculitis with systemic symptoms.
• Acute blistering disorders suggesting pemphigus or Stevens-Johnson syndrome / toxic epidermal necrolysis.

Assign FALSE for:

• Stable, non-urgent malignancies (e.g., small, asymptomatic BCC).
• Premalignant lesions without signs of progression (e.g., stable actinic keratosis).
• Chronic inflammatory conditions without acute exacerbation.

Clinical Reference: NICE Guidelines (UK), National Comprehensive Cancer Network (NCCN) urgent referral criteria.

High-Priority Referral

Definition: Conditions that should be evaluated by a dermatologist within 2 weeks (14 days) according to standard cancer referral pathways and dermatology triage guidelines, typically for suspected malignancy or high-risk lesions not meeting urgent criteria.

Assign TRUE for:

• All non-urgent confirmed or suspected skin cancers (BCC, SCC, melanoma).
• High-risk premalignant lesions (e.g., large or symptomatic actinic keratosis, Bowen's disease).
• Atypical pigmented lesions with dermoscopic features of concern (e.g., atypical nevus requiring biopsy).
• Rapidly changing or symptomatic lesions of unclear etiology.

Assign FALSE for:

• Benign, stable lesions with no malignant features (e.g., seborrheic keratosis, dermal nevus).
• Chronic inflammatory conditions managed in routine care (e.g., stable psoriasis, eczema).

Clinical Reference: UK 2-Week Wait cancer referral pathway, European guidelines for skin cancer referral.

Documentation Requirements:

• For each TRUE classification, the annotator should briefly note the clinical rationale.
• For ambiguous or borderline cases, the annotator must provide a written justification with at least one literature reference (guideline, consensus statement, or peer-reviewed publication).

Deliverable: Fully annotated mapping matrix with clinical justifications, submitted for secondary review.

Independent Secondary Review

Responsible Party: Secondary Reviewer (Independent Dermatologist)

Activities:

1. Conduct a comprehensive, independent review of the completed annotation matrix without prior discussion with the Primary Annotator.
2. Assess clinical accuracy, internal consistency, and appropriateness of classifications.
3. Identify any disagreements, questionable classifications, or internal inconsistencies.
4. Document all discrepancies and proposed alternative classifications with clinical justification.

Deliverable: Secondary review report documenting agreement/disagreement for each classification, with detailed notes on all discrepancies.

Consensus Resolution and Finalization

Responsible Parties: Primary Annotator, Secondary Reviewer, MDS Team (facilitator)

Activities:

1. Consensus Meeting: The Primary Annotator and Secondary Reviewer meet to discuss all identified discrepancies.
2. Evidence-Based Resolution: Each disagreement is resolved through:
   • Review of relevant clinical guidelines and literature.
   • Discussion of clinical rationale and intended use context.
   • Application of the principle of clinical caution where evidence is ambiguous.
3. Final Consensus Matrix: All classifications are finalized with documented consensus rationale for previously disputed cases.
4. Formal Approval: Both dermatologists provide written approval of the final matrix.

Deliverable: Final, consensus-approved mapping matrix with full documentation of the resolution process for all discrepancies.

Technical Validation and Version Control

Responsible Party: MDS Team

Activities:

1. Format Validation: Verify the matrix structure, completeness (no missing values), and machine readability against the defined data schema (see below).
2. Consistency Checks: Perform automated logical consistency checks to ensure clinical validity:
   • Malignancy Rule: If malignancy is TRUE, then either urgent_referral OR high_priority_referral MUST be TRUE.
   • Premalignancy Rule: If premalignancy is TRUE, then high_priority_referral SHOULD be TRUE (unless explicitly justified as low-risk).
   • Referral Hierarchy: A condition cannot be both urgent_referral and high_priority_referral (they are mutually exclusive priority levels, though both imply referral). Note: If the system design allows overlapping flags, this check ensures the highest priority is acted upon.
3. Version Control: Assign a formal version number (e.g., v1.0) and register the matrix as a controlled artifact (e.g., DATA-001-BinaryMappingMatrix.csv).
4. Integration: Implement the mapping logic in the device software post-processing module and validate correct implementation through unit testing.

Deliverable: Version-controlled, technically validated mapping matrix ready for integration into the device and use in validation testing.

Data Schema Specification

To ensure seamless integration with the AI/ML pipeline, the final mapping matrix must adhere to the following schema:

Column Name	Data Type	Description
icd11_code	String	Unique ICD-11 identifier (Primary Key)
icd11_title	String	Official ICD-11 Category Name
malignancy	Boolean (0/1)	Flag for invasive malignancy
premalignancy	Boolean (0/1)	Flag for premalignant potential
association_to_malignancy	Boolean (0/1)	Flag for diagnostic mimics/risk association
pigmented_lesion	Boolean (0/1)	Flag for pigmented status
urgent_referral	Boolean (0/1)	Flag for ≤48h referral priority
high_priority_referral	Boolean (0/1)	Flag for ≤2 week referral priority
justification	String	Clinical rationale for classification
references	String	Citations supporting the classification (DOI, URL, or Citation)

Quality Control and Validation

To ensure the highest level of clinical accuracy, regulatory compliance, and technical robustness, the following quality control and validation measures are implemented throughout the mapping matrix lifecycle:

Clinical Quality Assurance

• Dual Expert Review: Mandatory independent review by two board-certified dermatologists with documented consensus resolution ensures clinical validity and inter-rater reliability.
• Evidence-Based Classification: All classifications, particularly for ambiguous cases, are supported by references to clinical guidelines, consensus statements, or peer-reviewed literature.
• Internal Consistency Validation: Automated checks verify logical relationships between indicators (e.g., if malignancy = TRUE, then typically urgent or high-priority referral should also be TRUE).

Regulatory and Documentation Standards

• Traceability: Complete documentation trail from initial annotation through consensus resolution to final approval, maintained as part of the device technical file.
• Version Control: The mapping matrix is maintained as a controlled document with formal versioning, change history, and approval signatures.
• Justification Records: All clinical justifications and literature references are retained to support regulatory review and potential audits.

Technical Validation

• Format Verification: Automated validation of matrix structure, completeness (no null values), and machine readability.
• Implementation Testing: Unit tests verify correct software implementation of the mapping logic and probability summation calculations.
• Reference Standard Generation: The matrix is used to generate reference standard binary indicator labels for validation datasets, enabling quantitative performance assessment.

Post-Deployment Monitoring

• Periodic Review: The mapping matrix is subject to periodic clinical review (recommended every three years or when significant new clinical evidence emerges) to ensure continued alignment with current medical knowledge and clinical practice.
• External Trigger Review: Updates to the WHO ICD-11 classification that affect dermatological categories within the device scope will trigger an immediate review of this mapping matrix.
• Change Management: Any revisions to the mapping matrix follow formal change control procedures, including clinical re-review, regulatory impact assessment, and version update.

Approval and Sign-Off

The final mapping matrix requires formal written approval from:

• Primary Annotator (Lead Dermatologist)
• Secondary Reviewer (Independent Dermatologist)
• Medical Device Responsible Person or designee

This approved matrix serves as the definitive specification for binary indicator derivation and the reference standard reference standard for all subsequent validation and testing of binary indicator performance.

Signature meaning

The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:

• Author: JD-009
• Reviewer: JD-009
• Approver: JD-005