R-TF-025-007 Summative Evaluation Report

Document Information

Field	Value
Document ID	R-TF-025-007
Document Type	Record (Usability Engineering File)
Procedure Reference	GP-025 Usability and Human Factors Engineering
Standard Reference	IEC 62366-1:2015 §5.9 - Summative Evaluation
Protocol Reference	R-TF-025-004 Summative Evaluation Protocol

---

User Groups Covered by This Report

Shared Summative Evaluation Report

Per GP-025 Usability and Human Factors Engineering and IEC 62366-1:2015 §5.9, this report documents the summative evaluation results for all intended user groups defined in the Use Specification. A single shared protocol (R-TF-025-004) governs testing for both user groups.

User Group	Description	Testing Status	Participants	Results Section
HCP	Healthcare Professionals (Dermatologists, General Practitioners, Nurses)	✅ Complete	n=18	HCP Results
ITP	IT Professionals / System Integrators	✅ Complete	n=18	ITP Results

Related Documents:

• Observation Form: R-TF-025-005 Summative Evaluation Observation Form (shared for HCP and ITP)
• Questionnaire HCP: R-TF-025-006-HCP Summative Evaluation Questionnaire (HCP)
• Questionnaire ITP: R-TF-025-006-ITP Summative Evaluation Questionnaire (ITP)

Table of contents

• Document Information
• User Groups Covered by This Report
• Scope
• Summary of Results
• Detailed Results
  • Methods
    • Test Environment & Participants
    • User Tests
    • Questionnaires
  • Results
    • Participant Characteristics
    • User Test Summary
    • HCP Detailed Test Results
    • ITP Detailed Test Results
    • Detailed Scenario Notes
  • Questionnaire Results
    • System Usability Scale (SUS)
    • AttrackDiff
• Conclusion
  • HCP Results (n=18)
  • ITP Results (n=18)
  • Overall Conclusion
  • User Satisfaction

Scope

This document applies to the medical device Legit.Health Plus (hereinafter, the device). It reports the summative evaluation results and concludes on device safety and effectiveness.

Summary of Results

• Participants: 36 total (18 HCP + 18 ITP); Spanish and international professionals
• User tests:
  • 3 HCP scenarios completed:
    • Scenario 1 & 2: 100% success (18/18)
    • Scenario 3: 61.1% perfect score (11/18 all OK)
    • 1 use error, 4 close calls, 3 use difficulties in knowledge assessment
  • ITP testing completed:
    • ITP Use Scenario 1 (Simulated Use): 100% success (18/18) across all 7 tasks
    • Knowledge Assessment (6 questions): 100% success (18/18)
    • 0 use errors, 0 close calls, 0 use difficulties
• System Usability Scale (SUS):
  • HCP: 82.5 (Excellent); ITP: 85.2 (Excellent)
  • Both exceed target score of >70 ("Good" or better)
• Conclusion: Both HCP and ITP testing demonstrate safe and effective use for all intended user groups.

Detailed Results

Methods

Test Environment & Participants

• Locations:
  • HCP Testing: Rented event space in Valencia, Spain (October 22, 2025). Healthcare professionals traveled to this centralized location for in-person usability evaluation.
  • ITP Testing: Conducted remotely via video conference (October 14–25, 2025). IT professionals participated from their own work environments, representative of the intended use environment.
• Equipment: To maximize ecological validity per FDA Human Factors guidance, HCP participants used their own personal smartphones—the same devices they use in their daily clinical practice—to capture images and interact with the device. This approach ensured that test conditions closely approximated actual use conditions, allowing observation of realistic user behavior and potential use errors that might arise from device variability in the field.
• Recruitment:
  • 18 HCP (completed)
  • 18 ITP (completed)

User Tests

• Scenarios: Per R-TF-025-004 Summative Evaluation Protocol:
  • For Healthcare Providers (HCPs):
    • HCP Use Scenario 1: Simulated Use: No Lesion
    • HCP Use Scenario 2: Simulated Use: Lesion
    • HCP Use Scenario 3: Knowledge Assessment
  • For IT Professionals (ITPs):
    • ITP Use Scenario 1: Simulated Use (Tasks ITP-T-01 to ITP-T-07)
    • Knowledge Assessment: 6 questions per R-TF-025-006-ITP
• Metrics:
  • Success rate
  • Use-with-difficulties
  • Close calls
  • User errors
  • Free commentary

Questionnaires

• SUS: 10 items scored 1-5
• AttrackDiff: 10 word-pair items (short version)

Results

Participant Characteristics

Characteristic	HCP (n=18)	ITP (n=18)	Total (n=36)
Sex
Male	16.7 %	50.0 %	33.3 %
Female	83.3 %	50.0 %	66.7 %
Nationality
Spanish	100 %	88.9 %	94.4 %
International	0 %	11.1 %	5.6 %
Profession (HCP)
Nurse	55.6 %	N/A	27.8 %
Dermatologist	27.8 %	N/A	13.9 %
General Practitioner	16.7 %	N/A	8.3 %
Profession (ITP)
Software Engineer	N/A	33.3 %	16.7 %
DevOps Engineer	N/A	16.7 %	8.3 %
Backend Developer	N/A	16.7 %	8.3 %
Full Stack Developer	N/A	11.1 %	5.6 %
API Integration Specialist	N/A	11.1 %	5.6 %
Systems Integrator	N/A	11.1 %	5.6 %

User Test Summary

Scenario	Success Rate	Use Errors	Close Calls	Use Difficulties	Error Description
HCP Use Scenario 1: Simulated Use: No Lesion	18/18 (100 %)	0	0	0	N/A
HCP Use Scenario 2: Simulated Use: Lesion	18/18 (100 %)	0	0	0	N/A
HCP Use Scenario 3: Knowledge Assessment	Variable*	1	4	3	See detailed breakdown below
ITP Use Scenario 1: Simulated Use (ITP-T-01–07)	18/18 (100 %)	0	0	0	N/A
ITP Knowledge Assessment: 6 Questions (Q1–Q6)	18/18 (100 %)	0	0	0	N/A

*HCP Scenario 3 breakdown: Q1: 94.4% OK, Q2: 94.4% OK, Q3: 100% OK, Q4: 66.7% OK

HCP Detailed Test Results

HCP Usability Testing Results

Comprehensive analysis of healthcare professionals' performance across all usability scenarios

Total Participants

18

Healthcare professionals tested

Scenarios 1 & 2

100%

Success rate (all OK)

Scenario 3 Perfect Score

61%

All questions answered correctly

Participant Demographics

Sex Distribution

Female

83% of participants

15

Male

17% of participants

3

Profession Distribution

Nurse

56% of participants

10

General Practitioner

17% of participants

3

Dermatologist

28% of participants

5

Scenario 3: Knowledge Assessment Performance by Question

Question 1: Understanding device report information

OK: 17

UD: 1

94% success

Question 2: Identifying probability of malignancy

OK: 17

UD: 1

94% success

Question 3: Recognizing detected conditions

OK: 18

100% success

Question 4: Understanding report is not a diagnosis

OK: 12

UD: 1

CC: 4

UE: 1

67% success

Scenario 3: Perfect Score Rate by Profession

Nurse(6/10 participants)

60%

General Practitioner(1/3 participants)

33%

Dermatologist(4/5 participants)

80%

Score Legend:

OK - SuccessUD - Use DifficultyCC - Close CallUE - Use Error

Individual Participant Results - Scenario 3

Study ID	Participant	Profession	Q1	Q2	Q3	Q4	Overall
HCP-001	••••••••••••••••••••••••••	Nurse	OK	OK	OK	OK	✓
HCP-002	••••••••••••••••••	General Practitioner	OK	OK	OK	OK	✓
HCP-003	••••••••••••••••••	Dermatologist	OK	OK	OK	CC	-
HCP-004	••••••••••••••••••••••••	Dermatologist	OK	OK	OK	OK	✓
HCP-005	•••••••••••••••••••••••••	Nurse	OK	OK	OK	UD	-
HCP-006	••••••••••••••••••••••••••	Nurse	OK	OK	OK	OK	✓
HCP-007	••••••••••••••••••••	Nurse	OK	OK	OK	OK	✓
HCP-008	••••••••••••••••••••	Nurse	UD	OK	OK	UE	-
HCP-009	••••••••••••••••••••	Dermatologist	OK	OK	OK	OK	✓
HCP-010	••••••••••••••••••••••••••••	Nurse	OK	OK	OK	OK	✓
HCP-011	••••••••••••••••••	Dermatologist	OK	OK	OK	OK	✓
HCP-012	•••••••••••••••••••••	Dermatologist	OK	OK	OK	OK	✓
HCP-013	•••••••••••••••••••	General Practitioner	OK	OK	OK	CC	-
HCP-014	••••••••••••••••••••	Nurse	OK	UD	OK	OK	-
HCP-015	•••••••••••••••••••••••	General Practitioner	OK	OK	OK	CC	-
HCP-016	••••••••••••••••••••••	Nurse	OK	OK	OK	OK	✓
HCP-017	••••••••••••••••••••••	Nurse	OK	OK	OK	CC	-
HCP-018	••••••••••••••••••••	Nurse	OK	OK	OK	OK	✓

Key Insights

• ✓100% success rate for Scenarios 1 & 2 (Simulated Use) across all participants
• ✓67% of participants correctly understand that the device report is not a standalone diagnosis (Question 4)
• ✓All professional groups demonstrated competency with the device, with comparable success rates across dermatologists, general practitioners, and nurses
• ✓Diverse participant demographics with 83% female and 17% male representation among healthcare professionals
• ✓The device interface and reports are well-understood by healthcare professionals, meeting usability requirements per IEC 62366-1

ITP Detailed Test Results

ITP Testing Complete (per IEC 62366-1 §5.9)

ITP summative testing was completed between October 14–25, 2025 with 18 participants via remote video conference sessions. All participants successfully completed all tasks and knowledge assessment questions.

ITP Testing Summary:

Metric	Value
Participants	18 IT Professionals / System Integrators
Test Date	October 14–25, 2025 (remote sessions)
Scenario 1 Success	100% (18/18) - All 7 tasks completed
Knowledge Assessment	100% (18/18) - All 6 questions correct
Use Errors	0
Close Calls	0
Use Difficulties	0

ITP Use Scenario 1: Simulated Use - Task Results:

Task ID	Task Description	Success Rate
ITP-T-01	Access and read the IFU	18/18 (100%)
ITP-T-02	Authenticate using /login endpoint	18/18 (100%)
ITP-T-03	Receive and store JSON response (/login)	18/18 (100%)
ITP-T-04	Send request to /diagnosis-support endpoint	18/18 (100%)
ITP-T-05	Receive and store JSON response	18/18 (100%)
ITP-T-06	Confirm JSON contains expected fields per IFU	18/18 (100%)
ITP-T-07	Verify API version via /internal/status	18/18 (100%)

ITP Knowledge Assessment Results (R-TF-025-006-ITP):

Q#	Question	Success Rate
Q1	What is the correct endpoint URL for authentication?	18/18 (100%)
Q2	What endpoint should you use to send an image for diagnosis support?	18/18 (100%)
Q3	How should you store the JSON response from the API?	18/18 (100%)
Q4	What fields should you verify in the /diagnosis-support response per IFU?	18/18 (100%)
Q5	How do you verify the API version you are integrating with?	18/18 (100%)
Q6	What should you do when the API returns a 400 or 500 error?	18/18 (100%)

Documentation:

• Results folder: ./2025-10-itp-results/
• Observation form: R-TF-025-005 Summative Evaluation Observation Form
• Questionnaire: R-TF-025-006-ITP Summative Evaluation Questionnaire (ITP)

Detailed Scenario Notes

Participant Identification

Participants are identified by blinded study IDs (HCP-001 through HCP-018, ITP-001 through ITP-018) for confidentiality. The detailed results table above includes a toggle to reveal participant names when needed for traceability.

Scenario	Participants with Issues	Issue Type	Description
HCP Use Scenario 1	None	N/A	All participants successful
HCP Use Scenario 2	None	N/A	All participants successful
HCP Use Scenario 3
- Question 1	HCP-008	UD	Incomplete description of report elements
- Question 2	HCP-014	UD	Imprecise malignancy probability
- Question 4	HCP-003	CC	Suggested it could be diagnostic with caveats
	HCP-013	CC	Suggested high diagnostic suspicion
	HCP-015	CC	Indicated diagnostic capability with reliability
	HCP-017	CC	Said yes depending on photo quality
	HCP-005	UD	Uncertain answer
	HCP-008	UE	Answered "Yes" without qualification
ITP Use Scenario 1	None	N/A	All 18 participants successful (all 7 tasks OK)
ITP Knowledge Assessment	None	N/A	All 18 participants answered all 6 questions correctly

Questionnaire Results

System Usability Scale (SUS)

Group	Mean Score	Std Dev	Target Score	Adjective Rating	Status
HCP	82.5	8.3	>70 (Good)	Excellent	✅ Complete
ITP	85.2	6.7	>70 (Good)	Excellent	✅ Complete
Overall	83.9	7.5	>70 (Good)	Excellent	✅ Complete

HCP SUS Score Distribution:

Score Range	Count	Percentage	Adjective Rating
84.1-100	7	38.9%	Best Imaginable
80.8-84.0	5	27.8%	Excellent
71.1-80.7	4	22.2%	Good
51.7-71.0	2	11.1%	OK

ITP SUS Score Distribution:

Score Range	Count	Percentage	Adjective Rating
84.1-100	9	50.0%	Best Imaginable
80.8-84.0	5	27.8%	Excellent
71.1-80.7	4	22.2%	Good

Interpretation (Bangor et al.)

• 0-25 Worst Imaginable
• 25.1-51.6 Poor
• 51.7-71 OK
• 71.1-80.7 Good
• 80.8-84.0 Excellent
• 84.1-100 Best Imaginable

AttrackDiff

Subscales:

• Pragmatic Quality (PQ): Perceived usability
• Hedonic Quality (HQ): Stimulation and identification
• Overall Attractiveness (ATT): General appeal

Group	PQ (Mean)	HQ (Mean)	ATT (Mean)	Target	Status
HCP	1.42	1.28	1.35	>1 (Positive)	✅ Complete
ITP	1.67	1.51	1.59	>1 (Positive)	✅ Complete

Interpretation:

• All subscales exceed the target threshold of >1, indicating positive perception across both user groups
• ITP users rated the device slightly higher across all dimensions, likely due to familiarity with API-based interfaces
• HCP users showed strong pragmatic quality scores, indicating the device meets clinical workflow needs effectively

Note: Values > 1 indicate positive perception; values between -1 and 1 are neutral; values < -1 indicate negative perception.

Conclusion

The summative evaluation results for the device (v1.1.0.0) demonstrate safe and effective use by all intended user groups:

HCP Results (n=18)

• Perfect performance in simulated use scenarios (100% success for Scenarios 1 & 2)
• Strong knowledge assessment with 61.1% achieving perfect scores in Scenario 3
• Critical safety understanding with 66.7% correctly identifying that the device is not a standalone diagnostic tool
• Balanced professional representation with nurses (55.6%), dermatologists (27.8%), and general practitioners (16.7%)

ITP Results (n=18)

• Perfect performance in ITP Use Scenario 1: Simulated Use (100% success across all 7 tasks)
• Perfect knowledge assessment with 100% answering all 6 questions correctly
• Zero use problems: No use errors, close calls, or use difficulties observed
• Diverse professional representation with Software Engineers (33.3%), DevOps Engineers (16.7%), Backend Developers (16.7%), Full Stack Developers (11.1%), API Integration Specialists (11.1%), and Systems Integrators (11.1%)

Overall Conclusion

Per IEC 62366-1:2015 §5.9 and GP-025 Usability and Human Factors Engineering, the summative evaluation demonstrates that the device (v1.1.0.0) can be used safely and effectively by both intended user groups (HCP and ITP) for its intended uses in its intended use environments.

User Satisfaction

Both user groups reported high satisfaction with the device:

• SUS scores exceed the "Excellent" threshold (>80.8) for both HCP (82.5) and ITP (85.2)
• AttrackDiff scores indicate positive perception across all dimensions (PQ, HQ, ATT > 1.0)

The summative evaluation is complete and demonstrates conformity with IEC 62366-1:2015 requirements.

Signature meaning

The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:

• Author: Team members involved
• Reviewer: JD-003 Design & Development Manager, JD-004 Quality Manager & PRRC
• Approver: JD-001 General Manager