Skip to main content
QMSQMS
QMS
  • Welcome to your QMS
  • Quality Manual
  • Procedures
  • Records
  • Legit.Health Plus Version 1.1.0.0
  • Legit.Health Plus Version 1.1.0.1
  • Legit.Health Utilities
  • Licenses and accreditations
  • Applicable Standards and Regulations
  • BSI Non-Conformities
    • Technical Review
      • Round 1
        • M1: Diagnostic Function
        • M2: Software V&V
        • N1: Information Supplied
        • N2: Usability
        • N3: Risk Management
          • Question
          • Research and planning
          • Response
    • Clinical Review
    • BSI Non-Conformities
  • Pricing
  • Public tenders
  • BSI Non-Conformities
  • Technical Review
  • Round 1
  • N3: Risk Management
  • Response

Response

We acknowledge that BSI correctly identified a traceability gap in R-TF-013-002 for risk R-DAG ("The medical device outputs a wrong result"), per ISO 14971:2019 clause 7.2 (verification of implementation of risk control measures). The mitigationRequirements field contained infrastructure SRS codes identical to causeRequirements, and the verificationOfImplementation test cases verified only the transport layer (API port, HTTP status codes, JSON format), not the clinical output mitigations BSI flagged.

The implementations exist and have been verified — the gap was in the traceability documentation, not in the actual risk controls. We have corrected R-TF-013-002 to establish complete traceability from mitigation → requirement → test case → result for each implemented mitigation.

Mitigation-by-mitigation traceability (R-DAG)​

Mitigation 1: "Information about device outputs are detailed in the IFU"​

ElementReference
Mitigation requirementLR-4XK (Read IFU before use), LR-9WR (Device outputs interpretation guidance), LR-4RZ (Warnings and precautions), LR-8YN (Device supervision requirement)
IFU locationapps/eu-ifu-mdr/versioned_docs/version-1.1.0.0/installation-manual/user-interface.mdx — Full JSON output structure including probability distributions, entropy scores, explainability media, clinical indicators, severity scores
IFU locationapps/eu-ifu-mdr/versioned_docs/version-1.1.0.0/troubleshooting/clinical.mdx — Interpretation of distributions, entropy as uncertainty measure, top-5 accuracy approach
VerificationLabeling requirements verification documented in R-TF-012-037; complete evidence provided in M2 Q2 response

Mitigation 2: "The medical device returns metadata about the output that helps supervising it, such as explainability media and other metrics"​

ElementReference
Mitigation requirementSRS-0AB (Generate per-image ICD analysis with explainability heat map), SRS-K7M (Orchestrate diagnosis support workflow with pixel-level attention indicators)
Test caseC256 (T123): Verify response includes per-image ICD probabilities and heat maps for top five categories — validates explanation.attentionMap objects, colour model data, Base64-encoded image data
Test caseC265 (T132): Verify diagnosis workflow returns ranked ICD-11 codes, binary indicators, and explainability maps — validates entropy, pixel-level attention indicators (heat maps/saliency masks)
Test executionR-TF-012-033 Software Tests Plan — all tests passed
Model-level verificationAI Models Integration Tests (T307-T379, C466-C539) verify each AI model produces correct explainability outputs

Mitigation 3: "The device returns an interpretative distribution representation of possible ICD categories, not just one single condition"​

ElementReference
Mitigation requirementSRS-Q3Q (Generate aggregated ICD probability distribution from a set of images), SRS-K7M (Compute normalized probability vector across all supported ICD-11 categories)
Test caseC255 (T122): Verify API returns aggregated ICD probability distribution with structured code details — validates hypotheses array with numeric probability fields, valid ICD-11 code structures
Test caseC265 (T132): Verify diagnosis workflow returns ranked ICD-11 codes — validates top-5 ranked ICD-11 categories, probability sum = 100% across full distribution, entropy, five binary indicators
Test executionR-TF-012-033 Software Tests Plan — all tests passed
Model-level verificationAI Models Integration Tests (T307-T379, C466-C539) verify each AI model produces correct probability_distribution and icd_distribution data

Mitigation 4: "AI models are subject to retraining under expanded datasets"​

This is a prospective lifecycle control, not a runtime software feature. It has no software-level test cases because it is verified through QMS process adherence, not through runtime tests.

ElementReference
Process definitionGP-028 AI Development, § AI Updates → Retraining: "Retraining is performed when an algorithm's core logic or data foundation is modified"
Change governanceGP-023 Change Control: classifies retraining as minor or major AI model version change
Verification mechanismR-TF-028-010 AI V&V Checks: mandatory verification before any retrained model is released
DocumentationR-TF-028-007 AI Retraining Report: mandatory output of any retraining activity

Note: No retraining has been performed for v1.1.0.0 (no completed R-TF-028-007 record exists). Retraining is a prospective control triggered by PCCP criteria (e.g., post-market data indicating performance drift, new training data available). The mitigation statement in R-TF-013-002 has been reworded to accurately reflect this prospective nature, per ISO 14971:2019 clause 7.2 note on risk control measures that may include "inherent safety by design, protective measures, or information for safety."

Systematic audit of all risks​

BSI noted: "It is unclear if other risks are similarly impacted." A systematic audit of all 62 risks in R-TF-013-002 was performed. 29 risks were identified with traceability gaps analogous to R-DAG. They fall into three categories:

Category A: Infrastructure/API risks with process-level mitigations (21 risks)​

These risks had mitigationRequirements identical to causeRequirements with infrastructure-only verification. The mitigations are process-level controls (security best practices, SOUP analysis, QMS procedures) that require process-level verification references, not just software test cases.

Risk IDRisk nameProcess-level verification added
R-T8QData transmission failure from HCP systemSecurity and availability techniques per R-TF-012-006; error handling in API documentation
R-3N5Data input failureSecurity and availability techniques per R-TF-012-006; error handling in API documentation
R-YF4Data accessibility failureSecurity and availability techniques per R-TF-012-006; error handling in API documentation
R-LRPData transmission failureFHIR interoperability per IFU; added LR-5TG, LR-7XP to mitigationRequirements
R-MWDInterruption of serviceElastic scaling and backup infrastructure per R-TF-012-006; REST protocol error handling
R-OM1Data overwriteREST protocol architecture per R-TF-012-006; request immutability is inherent design feature
R-B63Inconsistent or unreliable outputAlgorithm V&V per GP-012; representative dataset validation per R-TF-028-010
R-VL1Device failure or performance degradationElastic scaling infrastructure per R-TF-012-006; error handling in API documentation
R-72DSOUP anomaly/incompatibilitySOUP analysis per R-TF-012-023; compatibility evaluation per GP-012; design review records in DHF
R-MQ1SOUP not maintained nor patchedSOUP monitoring and patching process per R-TF-012-006; SOUP records in R-TF-012-023
R-QLFNon-compliance with GSPRGSPR compliance per R-TF-001-006 GSPR Checklist; design per harmonized standards per R-TF-012-006
R-ES8Absence of risk management processISO 14971 implementation per GP-013; risk management records in R-TF-013-001 and R-TF-013-002
R-C6QAbsence of PMS & PMCF processPMS plan per R-TF-018-001; PMCF plan per R-TF-015-002
R-27MInadequate maintenanceMaintenance activities per GP-012; SOUP maintenance per R-TF-012-006
R-9SSSOUP cybersecurity vulnerabilitiesSOUP analysis per R-TF-012-023; cybersecurity evaluation per SP-012-002; design review records in DHF
R-33BElectronic IFU tamperedGit workflow with GPG-signed commits per GP-012; RBAC and branch protections in repository configuration
R-GY6Inaccurate training dataImage selection and HCP labeling process per GP-028; dataset quality records in R-TF-028-001
R-7USBiased or incomplete training dataImage selection and HCP labeling process per GP-028; dataset diversity records in R-TF-028-001

Category B: Security risk with missing SRS codes (1 risk)​

Risk IDRisk nameSRS codes addedTest cases added
R-HH0Electronic data and content tamperedSRS-1KW, SRS-SDZ, SRS-WER, SRS-WGFC332, C333, C343, C344, C345, C351, C352, C353, C354

Category C: Retraining mitigation with no traceability (5 risks)​

These risks include AI model retraining as an implemented mitigation but had no process-level verification reference:

Risk IDRisk nameCorrective action
R-DAGWrong result (ICD distribution)The original BSI finding — added SRS-Q3Q, SRS-0AB, SRS-K7M; C255, C256, C265; retraining process refs (GP-028, GP-023, R-TF-028-010)
R-75HIncorrect clinical informationAdded SRS-0AB, SRS-K7M; C256, C265; retraining process refs; reworded mitigation to prospective form
R-SKKIncorrect results shown to patientFixed typo "retarining" → "retraining"; added SRS-Q3Q, SRS-0AB, SRS-K7M; C255, C256, C265; retraining process refs
R-75LStagnation of model performanceReworded mitigation to prospective form; added retraining process refs (GP-028, GP-023, R-TF-028-010)
R-PWKDegradation of model performanceReworded mitigation to prospective form; added manual retraining process refs (GP-028, GP-023, R-TF-028-010)

All affected risks have been corrected in the updated R-TF-013-002.

Summary of changes to R-TF-013-002​

R-DAG (Risk: "The medical device outputs a wrong result")​

FieldChange
implementedMitigations[3]Reworded from "AI models undergo retraining using expanded dataset of images" to "AI models are subject to retraining under expanded datasets as governed by GP-028 (§ AI Updates → Retraining) and GP-023 (Change Control), with verification through R-TF-028-010 (AI V&V Checks) before any retrained model is released."
mitigationRequirementsAdded SRS-Q3Q, SRS-0AB, SRS-K7M (kept existing infrastructure codes alongside)
verificationOfImplementationAdded test cases C255 (T122), C256 (T123), C265 (T132); added reference to AI Models Integration Tests (T307-T379, C466-C539); added process-level references for retraining (GP-028, GP-023, R-TF-028-010)

R-75H (Risk: "Incorrect clinical information")​

FieldChange
implementedMitigations[2]Reworded retraining statement to prospective form
mitigationRequirementsAdded SRS-0AB, SRS-K7M
verificationOfImplementationAdded test cases C256 (T123), C265 (T132); added AI Models Integration Tests reference; added process-level retraining references

R-SKK (Risk: "Incorrect results shown to patient")​

FieldChange
implementedMitigations[3]Corrected typo "retarining" → "retraining"; reworded to prospective form
mitigationRequirementsAdded SRS-Q3Q, SRS-0AB, SRS-K7M
verificationOfImplementationAdded test cases C255 (T122), C256 (T123), C265 (T132); added AI Models Integration Tests reference; added process-level retraining references

R-HH0 (Risk: "Electronic data and content are tampered")​

FieldChange
mitigationRequirementsAdded SRS-1KW (TLS), SRS-SDZ (hashed passwords), SRS-WER (OAuth), SRS-WGF (AES-256 encryption)
verificationOfImplementationAdded test cases C332, C333 (TLS), C343, C344, C345 (auth), C351, C352, C353 (OAuth), C354 (encryption)

R-75L (Risk: "Stagnation of model performance")​

FieldChange
implementedMitigationsReworded to prospective form with explicit governance references (GP-028, GP-023, R-TF-028-010)
verificationOfImplementationAdded process-level retraining references

R-PWK (Risk: "Degradation of model performance")​

FieldChange
implementedMitigationsReworded to clarify manual-only retraining with explicit governance references (GP-028, GP-023, R-TF-028-010)
verificationOfImplementationAdded process-level retraining references

Category A risks (21 risks with process-level mitigations)​

For all Category A risks (R-T8Q, R-3N5, R-YF4, R-LRP, R-MWD, R-OM1, R-B63, R-VL1, R-72D, R-MQ1, R-QLF, R-ES8, R-C6Q, R-27M, R-9SS, R-33B, R-GY6, R-7US):

FieldChange
verificationOfImplementationAdded process-level verification references to QMS procedures and records where mitigations are documented
mitigationRequirementsFor R-LRP: Added LR-5TG, LR-7XP (FHIR IFU documentation)

Regulatory compliance​

The corrective actions address:

  • ISO 14971:2019 clause 7.2: Verification of implementation of risk control measures — complete traceability chain now documented
  • ISO 14971:2019 clause 7.6: Completeness of risk control — systematic audit confirmed all analogous risks have been corrected
  • ISO 14971:2019 clause 7.4: Benefit-risk analysis — conclusions unchanged by traceability corrections
  • GSPR 1: Intended performance — mitigations (explainability, distributions, IFU) ensure device output supports HCP decision-making as intended
  • GSPR 4: Risk management system per Annex I §3 — traceability chain is now complete
  • GSPR 17.2: Diagnostic accuracy — ICD probability distribution and explainability media are the mechanisms by which accuracy/precision are communicated to HCP
  • Annex II 5(b): Description and justification of residual risks — R-TF-013-002 demonstrates residual risks acceptable after verified controls
  • Annex II 6.1(a)/(b): Evidence of GSPR compliance — verification test cases now clearly map to mitigations
  • Annex II 6.2(f): Risk analysis including risk control measures — complete traceability chain fulfils this requirement

Note on traceability matrix (R-TF-012-001)​

The Software Requirements Specification (SRS) codes and test cases added to R-TF-013-002 already existed in the traceability matrix R-TF-012-001 (apps/qms/docs/legit-health-plus-version-1-1-0-0/design-and-development/R-TF-012-001.json). Specifically:

  • SRS-Q3Q, SRS-0AB, SRS-K7M (clinical output mitigations) → already mapped to test cases C255, C256, C265
  • SRS-1KW, SRS-WER, SRS-SDZ, SRS-WGF (security mitigations) → already mapped to test cases C332, C333, C343, C344, C345, C351, C352, C353, C354

The gap identified by BSI was a referencing error in the Risk Management Record (R-TF-013-002), not a missing implementation or verification gap. The requirements were implemented, the tests were executed, and the traceability was documented in R-TF-012-001 — the Risk Record simply failed to reference the correct codes.

R-TF-012-001 has not been modified as part of this corrective action because no changes to the requirement-to-test traceability were necessary.

Documents modified​

DocumentPathChanges
R-TF-013-002 Risk Management Recordapps/qms/docs/legit-health-plus-version-1-1-0-0/risk-management/R-TF-013-002.jsonCorrected traceability for 29 risks: (1) R-DAG, R-75H, R-SKK — added clinical mitigation SRS codes and test cases; (2) R-HH0 — added security SRS codes and test cases; (3) R-75L, R-PWK — reworded retraining mitigations to prospective form; (4) 21 Category A risks — added process-level verification references; (5) R-LRP — added LR codes for FHIR documentation; (6) R-GY6, R-7US — fixed incomplete mitigation text; (7) Fixed R-SKK typo "retarining" → "retraining"

Documents not impacted​

DocumentPathReason
R-TF-012-001 Traceability Matrixapps/qms/docs/legit-health-plus-version-1-1-0-0/design-and-development/R-TF-012-001.jsonSRS codes and test cases already existed; corrective action was referencing, not implementation
Previous
Research and planning
Next
Clinical Review: Round 1
  • Mitigation-by-mitigation traceability (R-DAG)
    • Mitigation 1: "Information about device outputs are detailed in the IFU"
    • Mitigation 2: "The medical device returns metadata about the output that helps supervising it, such as explainability media and other metrics"
    • Mitigation 3: "The device returns an interpretative distribution representation of possible ICD categories, not just one single condition"
    • Mitigation 4: "AI models are subject to retraining under expanded datasets"
  • Systematic audit of all risks
    • Category A: Infrastructure/API risks with process-level mitigations (21 risks)
    • Category B: Security risk with missing SRS codes (1 risk)
    • Category C: Retraining mitigation with no traceability (5 risks)
  • Summary of changes to R-TF-013-002
    • R-DAG (Risk: "The medical device outputs a wrong result")
    • R-75H (Risk: "Incorrect clinical information")
    • R-SKK (Risk: "Incorrect results shown to patient")
    • R-HH0 (Risk: "Electronic data and content are tampered")
    • R-75L (Risk: "Stagnation of model performance")
    • R-PWK (Risk: "Degradation of model performance")
    • Category A risks (21 risks with process-level mitigations)
  • Regulatory compliance
  • Note on traceability matrix (R-TF-012-001)
  • Documents modified
    • Documents not impacted
All the information contained in this QMS is confidential. The recipient agrees not to transmit or reproduce the information, neither by himself nor by third parties, through whichever means, without obtaining the prior written permission of Legit.Health (AI Labs Group S.L.)