N2: Usability
Code: 2650005-202501-N2
Status: Open: awaiting manufacturer response
Requirements and references
- GSPR 1, 5, 17.2
- Annex II, 6.1(a), (b)
- EN 62366-1
Documents reviewed by BSI
QMS-legit-health-plus-version-1-1-0-0-product-verification-and-validation-usability-R-TF-025-004-Sum.pdfQMS-legit-health-plus-version-1-1-0-0-product-verification-and-validation-usability-R-TF-025-007-Sum.pdfR-TF-013-002 Risk management record.pdf
Question 1: EN 62366-1, 5.9 compliance and summative evaluation
Usability of the user interface and compliance to EN 62366-1, 5.9 is not fully demonstrated. Examples of gaps include, but are not limited to:
a. Use errors, close calls and difficulties were observed during summative evaluation, HCP use scenario 3. Evidence that they were subjected to root cause analysis and residual risk assessment per Summative Evaluation Protocol, 14.7. Data Analysis and EN 62366-1, 5.9 could not be found.
b. It is unclear why it was acceptable that 33% of HCP participants did not understand that the device is not intended for diagnosis. Risks R-TBN "Insufficient label information to understand the device intended use, version" and R-CGQ "Whole device is wrongly used or is not used as intended" are not clearly effectively mitigated.
c. Results and conclusions regarding usability of the instructions for use could not be found.
d. Results and conclusions regarding effectiveness of information for safety could not be found. Information for safety regarding intended use is not clearly effective per point b. above.
Please clarify and provide updated documentation as necessary.
Regulatory context
This section is for internal planning only. It will not be included in the final response to BSI.
BSI raises one question with four sub-items (a–d), all under EN 62366-1 §5.9 (summative evaluation). The core concern: the summative evaluation report (R-TF-025-007) does not fully demonstrate usability of the user interface.
Documents BSI reviewed:
R-TF-025-004-Sum.pdf— Summative Evaluation ProtocolR-TF-025-007-Sum.pdf— Summative Evaluation ReportR-TF-013-002 Risk management record.pdf
Underlying regulatory logic: EN 62366-1 §5.9 requires summative evaluation to demonstrate safe and effective use. MDR Annex II §6.1(a)(b) requires usability engineering file to include summative results. GSPRs 1, 5, 17.2 require devices to be safe, account for user characteristics, and validate software. When use problems are observed, the standard requires root cause analysis and residual risk assessment. BSI found this analysis was committed to in the protocol but absent from the report.
Protocol commitments (R-TF-025-004, "Data Analysis" section) that BSI could not find in the report:
- Data collection and coding — Done in the report
- Root cause analysis (RCA): "RCA will involve examining the data to determine the underlying cause of each use problem, with a focus on investigating contributing factors" — NOT in the report
- Residual risk analysis: "All risks that remain after human factors validation testing will be analyzed to determine whether they can be reduced or eliminated" — NOT in the report
The protocol also specifies that each scenario includes a "Post Scenario Interview (including Root Cause Interview)" — the report does not reference these interview findings.
Regulatory requirements mapping
Each referenced requirement is listed below with the specific clause text and how our fix addresses it. The response to BSI should explicitly reference these mappings.
| Requirement | Clause text (summary) | How our fix addresses it |
|---|---|---|
| GSPR 1 (MDR Annex I, §1) | "Devices shall achieve the performance intended by their manufacturer and shall be designed and manufactured in such a way that, during normal conditions of use, they are suitable for their intended purpose." | The updated summative evaluation report demonstrates, through formal RCA and residual risk assessment, that the device achieves its intended performance and that observed use problems do not indicate a failure of the device to meet its intended purpose. |
| GSPR 5 (MDR Annex I, §5) | "...the device shall be designed and manufactured in such a way as to reduce as far as possible the risks posed by use error..." | The RCA analyzes each use problem to determine root causes. The residual risk assessment evaluates whether further risk reduction is needed. The R-CGQ update (expanding to HCP user group) and voluntary IFU enhancement demonstrate ongoing risk reduction. |
| GSPR 17.2 (MDR Annex I, §17.2) | "For devices that incorporate software or for software that are devices in themselves, the software shall be developed and manufactured in accordance with the state of the art taking into account the principles of development life cycle, risk management, including information security, verification and validation." | The summative evaluation (EN 62366-1 §5.9) is part of the validation required by GSPR 17.2. The updated report completes the validation evidence by providing the RCA and residual risk analysis that were missing, demonstrating the software has been validated including its user interface. |
| MDR Annex II, §6.1(a) | Technical documentation shall include "the usability engineering file covering ... the results of the summative evaluation." | The updated R-TF-025-007 now includes complete summative evaluation results: not just raw data, but the analysis (RCA, residual risk, IFU usability conclusions, safety information effectiveness) that constitutes meaningful "results" per Annex II. |
| MDR Annex II, §6.1(b) | Technical documentation shall include "evidence of validation of the methods used in the ... usability assessment." | The protocol (R-TF-025-004) defines the validation method. The updated report demonstrates the method was followed through to completion, including the data analysis steps committed to in §14.7 of the protocol. |
| EN 62366-1, §5.9 | Summative evaluation shall "provide objective evidence that the user interface of the medical device can be used safely." When use problems are identified, "the manufacturer shall conduct a risk assessment to determine if any use problem represents an unacceptable risk" and "shall determine the root cause of each use problem." | This is the core requirement. The updated report adds: (1) root cause determination for each of the 8 non-OK observations, (2) risk assessment connecting findings to R-CGQ and R-TBN, (3) explicit conclusion on whether use problems represent unacceptable risk, (4) conclusions on IFU usability and safety information effectiveness. |
Two observations were reclassified as part of the root cause analysis process: HCP-013 Q4 (CC→OK, handwriting misread "sin" as "si") and HCP-014 Q2 (UD→OK, qualitatively correct answer scored too conservatively). These are corrections of initial scoring errors, not retroactive data manipulation. The reclassification is fully documented and justified in the RCA section of R-TF-025-007. Updated counts: 6 non-OK (1 UE, 3 CC, 2 UD), down from 8 (1 UE, 4 CC, 3 UD).
Summary: documents to update
| Document | Update | Type | Regulatory mapping |
|---|---|---|---|
| R-TF-025-007 Summative Evaluation Report | Add 4 new sections: (1) Root Cause Analysis of Observed Use Problems, (2) Residual Risk Assessment, (3) Usability of Instructions for Use, (4) Effectiveness of Information for Safety. RCA must be based on written response analysis (not interview data — see caveat in sub-item a). | Content addition (red-line) | EN 62366-1 §5.9, MDR Annex II §6.1(a)(b), GSPR 1, 5, 17.2 |
| R-TF-013-002 Risk Management Record | R-CGQ: expand user group from "ITP" to "ITP, HCP"; add summative evaluation HCP results as verification evidence; reference new RCA/residual risk sections. R-TBN: add summative evaluation as additional verification method alongside audits. | Content update (red-line) | GSPR 5, EN 62366-1 §5.9, ISO 14971 |
| IFU (eu-ifu-mdr) | Add a dedicated "Important safety information" callout to the Clinical User Manual after the intended purpose section, with explicit non-diagnostic statement in a visually prominent warning format. | Content enhancement (voluntary) | GSPR 5, EN 62366-1, ISO 14971 §7.4 |
Prioritization
- Highest priority: Sub-items a and b — the RCA and residual risk assessment are the core deliverables. Also the R-TF-013-002 updates (R-CGQ user group expansion and R-TBN verification update) since BSI explicitly references these risks.
- Medium priority: Sub-items c and d — missing explicit conclusions where evidence already exists.
- Committed: Voluntary IFU enhancement (safety callout) — concrete change specified above, demonstrates proactive commitment to safety.