Research and planning

Internal working document

This page is for internal planning only. It will not be included in the final response to BSI.

Analysis

BSI identified that the protocol (R-TF-025-004 §14.7 "Data Analysis") explicitly commits to:

• Root cause analysis: "examining the data to determine the underlying cause of each use problem, with a focus on investigating contributing factors to each problem and determining ways to prevent them in the future"
• Residual risk analysis: "All risks that remain after human factors validation testing will be analyzed to determine whether they can be reduced or eliminated"

The report (R-TF-025-007) listed the use problems in a "Detailed Scenario Notes" table but provided no RCA section and no residual risk assessment.

The standard/GSPR at stake is EN 62366-1 §5.9.

Relevant documents

Document	Path	Key section
R-TF-025-004 Summative Evaluation Protocol	legit-health-plus-version-1-1-0-0/product-verification-and-validation/usability/R-TF-025-004-Summative-Evaluation-Protocol.mdx	"Data Analysis" section: "Description of Root Cause Analysis" and "Description of Residual Risk Analysis"
R-TF-025-007 Summative Evaluation Report	R-TF-025-007-Summative-Evaluation-Report/R-TF-025-007-Summative-Evaluation-Report.mdx	"Detailed Scenario Notes" table lists problems; "Conclusion" section claims safe use — but no RCA or residual risk section bridges the two
HCP test data (raw)	R-TF-025-007-Summative-Evaluation-Report/2025-10-22-hcp-results/usability_test_responses.json	Verbatim answers for all 18 participants including the non-OK responses

Applicable standard references

IEC 62366-1:2015+AMD1:2020 §5.9

• Summative evaluation shall provide "objective evidence that the residual use-related risk is acceptable."
• AMD1:2020 removed the previous acceptance criteria subclause — there is no numerical pass/fail threshold. The assessment is qualitative and risk-based.
• When use problems are found, the manufacturer shall determine root cause, assess whether the UI is a contributing factor, and document the rationale for acceptability of residual risk.
• Manufacturers may provide "rationale on the acceptability of the residual risks individually, with the rationale potentially sought in the risk/benefit ratio of the device's use."

IEC 62366-1 definitions (Clause 3)

• Use error (3.21): User action or lack of action that led to a different result than intended. Requires potential for harm.
• Close call: A use difficulty where the user almost commits a use error but recovers in time. Defined as a NOTE under use difficulty — close calls are NOT use errors.
• Use difficulty: Occurs temporarily and is overcome by the user. Can become a use error if not overcome.

FDA "Applying Human Factors and Usability Engineering to Medical Devices" (2016)

• Close calls are distinct from use errors — they demonstrate the user's error detection and correction capability.
• Root cause analysis must go "far beyond blaming the user" and determine the specific UI aspect that caused problems.
• Residual risk is acceptable when: (1) further reduction is not practical, AND (2) benefits outweigh residual risks.
• "It is practically impossible to make any device error-proof or risk-free; some residual risk will remain."
• No standard defines an acceptable use error rate — the assessment is qualitative, not a numerical threshold.

Gap analysis

This is a genuine gap. The protocol promises RCA and residual risk analysis; the report did not deliver them.

What the protocol promises	What the report contained
"RCA will involve examining the data to determine the underlying cause of each use problem"	Listed use problems in a table but no RCA section
"All risks that remain after HF validation testing will be analyzed"	Conclusion stated testing demonstrates safe use but no formal residual risk assessment

Scoring reclassification

During root cause analysis, two observations were found to have been scored more conservatively than warranted:

1. HCP-013 (Laura Yuste), Q4: CC → OK. The handwritten response reads "PUEDE AYUDAR MUCHO AL DIAGNÓSTICO, INCLUSO SIN SER UN DIAGNÓSTICO POR SI MISMO, UNA ALTA SOSPECHA." The initial transcription misread "sin" (without) as "si" (if). Upon review, the participant correctly stated the device is not a diagnosis itself.

2. HCP-014 (Marta Borrás), Q2: UD → OK. The response "Muy baja, cerca del 0%" is qualitatively correct for the displayed malignancy probability (0.08%). The initial scoring treated lack of an exact number as a use difficulty, but the participant conveyed the correct clinical meaning.

Updated counts: 6 non-OK (1 UE, 3 CC, 2 UD), down from 8 (1 UE, 4 CC, 3 UD).

• Q1: 94.4% OK (1 UD) — unchanged
• Q2: 100% OK — changed from 94.4%
• Q3: 100% OK — unchanged
• Q4: 72.2% OK — changed from 66.7%
• Perfect score (all 4 Qs OK): 72.2% (13/18) — changed from 61.1% (11/18)

Response strategy

Acknowledge gap + provide the analysis in an updated R-TF-025-007. ✅ DONE.

Two new sections have been added to the report:

1. "Root Cause Analysis of Observed Use Problems" — for each of the 6 non-OK observations, documents:

   • What was observed (with verbatim participant response)
   • Root cause determination
   • Whether it indicates a user interface design issue (answer: No, for all 6)
   • Risk implication

2. "Residual Risk Assessment" — evaluates:

   • Close calls as positive evidence of error recognition and recovery (per IEC 62366-1 definition)
   • The single use error as isolated and participant-specific (not systematic)
   • Acceptability via two-part test: (1) further reduction not practical (device already has Option A inherent safety + Option C information for safety + voluntary IFU enhancement), (2) benefits outweigh residual risks (clinical benefit of skin assessment outweighs isolated knowledge-based misunderstanding, bounded by physician-in-the-loop architecture)
   • Specific assessment of R-CGQ and R-TBN mitigation effectiveness

Key arguments:

• No standard defines an acceptable use error rate (IEC 62366-1 AMD1:2020 removed acceptance criteria subclause)
• Close calls are NOT use errors — they are positive evidence of self-correction
• 1/18 (5.6%) true use error rate is isolated, not systematic
• Device architecture (clinical decision support requiring physician interpretation) provides inherent safety net
• 100% success in simulated use scenarios demonstrates correct operational use
• Voluntary IFU enhancement demonstrates commitment to continuous improvement

Priority: Highest. ✅ COMPLETED. This feeds into sub-items b and d.

Post-review corrections

During internal review (acting as BSI auditor), two issues were identified and resolved:

1. Missing "Effectiveness of Information for Safety" section in R-TF-025-007. The response to sub-item d claimed this section existed, but only three of the four planned sections had been added to the report. The section has now been added between "Usability of Instructions for Use" and "Conclusion," containing: a test-item-to-safety-topic mapping table, per-question aggregated results for both user groups, a voluntary enhancement subsection referencing the IFU safety callout, and an explicit conclusion on effectiveness.

2. Voluntary IFU safety callout not implemented. Responses to sub-items a, b, and d, as well as the residual risk assessment in R-TF-025-007, referenced a "dedicated safety information callout" in the Clinical User Manual as already implemented — but the IFU had not been modified. A new page has been created: apps/eu-ifu-mdr/versioned_docs/version-1.1.0.0/clinical-user-manual/important-safety-information.mdx (sidebar_position 0, first page in Clinical User Manual). Contains a :::warning admonition with the non-diagnostic statement and guidance on understanding the probabilistic output.

3. Spelling errors in precautions.mdx. Three spelling errors ("sympthoms", "recomended", "alognside") in the IFU Clinical User Manual precautions page were fixed. These are in the safety information section and their presence could undermine credibility during BSI review.