Research and planning

Internal working document

This document is for internal use only. It contains analysis, gap identification, and response strategy for Item 4 of the BSI Clinical Review Round 1. It will not be included in the final response to BSI.

1. What BSI is asking

BSI's clinical reviewer read the CEP (R-TF-015-001, §20, lines 644–650) which states:

"The summative usability validation will be conducted as an integral part of the upcoming clinical investigation."

The CEP was written in future tense and provides no further details. BSI wants to know:

1. Which CI the summative evaluation refers to
2. Whether it has been completed
3. A summary of the methodology: users, sample size, critical tasks
4. Results

This is an observation/request, not a deficiency finding. The regulatory basis is GSPR 5 (eliminating/reducing use-related risks) and EN ISO 14971.

2. The answer is straightforward

The summative evaluation has been completed. It was conducted as a standalone HF validation study (not embedded in a specific CI), documented in:

• R-TF-025-004: Summative Evaluation Protocol
• R-TF-025-005: Summative Evaluation Observation Form
• R-TF-025-006: Summative Evaluation Questionnaires (HCP + ITP)
• R-TF-025-007: Summative Evaluation Report

Key facts

Parameter	Value
Study dates	October 14–25, 2025
Location	HCP: in-person, Valencia, Spain. ITP: remote via video conference
Total participants	36 (18 HCP + 18 ITP)
HCP professions	10 nurses (55.6%), 5 dermatologists (27.8%), 3 GPs (16.7%)
ITP professions	Software engineers, DevOps, backend developers, API integration specialists, systems integrators
Standards	IEC 62366-1:2015 §5.9, FDA HF guidance (Feb 2016), ISO 14971
Equipment	HCPs used their own personal smartphones (ecological validity per FDA guidance)

Critical tasks tested

HCP (3 scenarios):

1. Scenario 1 — Simulated use: no lesion (photograph submission, report interpretation)
2. Scenario 2 — Simulated use: lesion (photograph submission, report interpretation with clinical findings)
3. Scenario 3 — Knowledge assessment (4 questions: report contents, malignancy probability, detected conditions, diagnostic vs support tool distinction)

ITP (1 scenario + knowledge assessment):

1. Scenario 1 — Simulated use: 7 tasks (access IFU, authenticate, send/receive API requests, verify response fields, check API version)
2. Knowledge assessment — 6 questions on endpoint URLs, response handling, error handling

Results summary

Metric	HCP	ITP
Simulated use success	100% (18/18) for scenarios 1 and 2	100% (18/18) for all 7 tasks
Knowledge assessment	Variable — Q1: 94.4%, Q2: 100%, Q3: 100%, Q4: 72.2%	100% (18/18) for all 6 questions
Use errors	1 (HCP Scenario 3 Q4)	0
Close calls	3 (HCP Scenario 3 Q4)	0
Use difficulties	2 (HCP Scenario 3 Q1, Q4)	0
SUS score	82.5 (Excellent)	85.2 (Excellent)
SUS target	>70 (Good)	>70 (Good)

Conclusion from R-TF-025-007

Both HCP and ITP testing demonstrate safe and effective use for all intended user groups. SUS scores exceed the target threshold.

3. Gap analysis

#	Aspect	Status	Gap
1	Study completed?	Done — October 2025	None
2	Methodology documented?	Done — R-TF-025-004 protocol	None
3	Results documented?	Done — R-TF-025-007 report	None
4	CEP updated to reflect completion?	Gap — CEP still uses future tense	CEP line 644–650 should be updated to past tense with reference to the report

The only gap for Item 4 specifically is that the CEP still describes the summative evaluation in future tense. The evaluation has been completed but the CEP was not updated to reflect this.

4. Cross-NC connections

Technical Review N2 — Usability (same study, deeper issues)

Critical cross-reference

The Technical Review N2 non-conformity (N2 index) addresses the same summative evaluation but raises much deeper concerns:

• N2.a: Use errors, close calls, and difficulties in HCP Scenario 3 were not subjected to root cause analysis (RCA) or residual risk assessment — despite the protocol (R-TF-025-004 §14.7) committing to this analysis
• N2.b: 33% of HCPs did not understand the device is not diagnostic (Q4 results: 72.2% success)
• N2.c: No conclusions about IFU usability
• N2.d: No conclusions about effectiveness of safety information

The fixes for N2 (adding RCA, residual risk assessment, IFU conclusions, and safety information effectiveness analysis to R-TF-025-007) will also strengthen the response to Item 4. The response to Item 4 should reference N2's more detailed analysis rather than duplicating it.

Clinical Review Item 3a — Clinical data analysis

Item 3a's CER fix requirements include integrating the summative evaluation results into the CER's clinical analysis. The CER currently references usability at line 766 but delegates to the summative report without summarising the results.

5. Response strategy

This is one of the simpler items to respond to. The response should:

1. Confirm the summative evaluation has been completed — reference R-TF-025-007
2. Provide the methodology summary BSI requested (users, sample size, critical tasks) — directly from the data above
3. Provide the results summary — success rates, SUS scores, use problems identified
4. Note that the CEP has been updated to reflect the completed status
5. Reference Technical Review N2 for the detailed RCA and residual risk analysis of the use problems identified (the clinical reviewer's concern is simply "has it been done?" — the deeper analysis is N2's territory)

Fix required

Fix 1: Update CEP future tense (minor)

In R-TF-015-001, lines 644–650, change:

"The summative usability validation will be conducted..."

To past tense with reference to the completed report:

"The summative usability validation was conducted in October 2025 in accordance with R-TF-025-004. Results are documented in R-TF-025-007."

6. Risk assessment

Risk	Impact	Mitigation
BSI reads the response and follows up with deeper usability questions	Low — the clinical reviewer's question is surface-level; deeper issues are handled by the technical reviewer in N2	Keep the response concise; reference N2 for detailed analysis
HCP Q4 result (72.2% on "is it diagnostic?") could concern BSI from a clinical perspective	Medium — 33% misunderstanding rate on intended use is clinically significant	This is addressed head-on in N2.b; the response here should acknowledge the finding but defer analysis to N2

7. Open items

None — all information needed to respond to Item 4 is already available in the QMS. This item can proceed directly to response drafting after the CEP future-tense fix is applied.