T-TF-025-007 Summative evaluation report
Table of contents
Scope
This document applies to the medical device Legit.Health Plus (hereinafter, the device). It reports the summative evaluation results and concludes on device safety and effectiveness.
Summary of Results
- Participants: 18 HCP tested (ITP testing pending); Spanish healthcare professionals
- User tests:
- 3 HCP scenarios completed:
- Scenario 1 & 2: 100% success (18/18)
- Scenario 3: 61.1% perfect score (11/18 all OK)
- 1 use error, 4 close calls, 3 use difficulties in knowledge assessment
- 3 HCP scenarios completed:
- System Usability Scale (SUS):
- Data collection in progress
- Target scores: >70 ("Good" or better)
- Conclusion: HCP testing demonstrates safe and effective use; ITP testing pending for complete evaluation.
Detailed Results
Methods
Test Environment & Participants
- Locations:
- Hospital Universitario La Paz, Madrid
- Hospital Universitario 12 de Octubre, Madrid
- The manufacturer's offices
- Equipment: Samsung Galaxy Tab S7-S9 with access to the device version 1.1.0.0
- Recruitment:
- 18 HCP (completed)
- ITP (pending)
User Tests
- Scenarios: four user scenarios:
- For Healthcare Providers (HCPs):
- HCP Use Scenario 1: Simulated Use: No Lesion
- HCP Use Scenario 2: Simulated Use: Lesion
- HCP Use Scenario 3: Knowledge Assessment
- For IT Professionals (ITPs):
- ITP Use Scenario 1: Simulated Use
- For Healthcare Providers (HCPs):
- Metrics:
- Success rate
- Use-with-difficulties
- Close calls
- User errors
- Free commentary
Questionnaires
- SUS: 10 items scored 1-5
- AttrackDiff: 10 word-pair items (short version)
Results
Participant Characteristics
| Characteristic | HCP (n=18) | ITP | Total |
|---|---|---|---|
| Sex | |||
| Male | 16.7 % | TBD | - |
| Female | 83.3 % | TBD | - |
| Nationality | |||
| Spanish | 100 % | TBD | - |
| Profession | |||
| Nurse | 55.6 % | N/A | - |
| Dermatologist | 27.8 % | N/A | - |
| General Practitioner | 16.7 % | N/A | - |
| IT Professional | N/A | TBD | - |
User Test Summary
| Scenario | Success Rate | Use Errors | Close Calls | Use Difficulties | Error Description |
|---|---|---|---|---|---|
| HCP Use Scenario 1: Simulated Use: No Lesion | 18/18 (100 %) | 0 | 0 | 0 | N/A |
| HCP Use Scenario 2: Simulated Use: Lesion | 18/18 (100 %) | 0 | 0 | 0 | N/A |
| HCP Use Scenario 3: Knowledge Assessment | Variable* | 1 | 4 | 3 | See detailed breakdown below |
| ITP Use Scenario 1: Simulated Use | TBD | TBD | TBD | TBD | Testing pending |
*Scenario 3 breakdown: Q1: 94.4% OK, Q2: 94.4% OK, Q3: 100% OK, Q4: 66.7% OK
HCP Detailed Test Results
HCP Usability Testing Results
Comprehensive analysis of healthcare professionals' performance across all usability scenarios
Total Participants
18
Healthcare professionals tested
Scenarios 1 & 2
100%
Success rate (all OK)
Scenario 3 Perfect Score
61%
All questions answered correctly
Participant Demographics
Sex Distribution
Female
83% of participants
15
Male
17% of participants
3
Profession Distribution
Nurse
56% of participants
10
General Practitioner
17% of participants
3
Dermatologist
28% of participants
5
Scenario 3: Knowledge Assessment Performance by Question
Question 1: Understanding device report information
OK: 17
UD: 1
94% success
Question 2: Identifying probability of malignancy
OK: 17
UD: 1
94% success
Question 3: Recognizing detected conditions
OK: 18
100% success
Question 4: Understanding report is not a diagnosis
OK: 12
UD: 1
CC: 4
UE: 1
67% success
Scenario 3: Perfect Score Rate by Profession
Nurse(6/10 participants)
60%
General Practitioner(1/3 participants)
33%
Dermatologist(4/5 participants)
80%
Score Legend:
OK - SuccessUD - Use DifficultyCC - Close CallUE - Use Error
Individual Participant Results - Scenario 3
| Participant | Profession | Q1 | Q2 | Q3 | Q4 | Overall |
|---|---|---|---|---|---|---|
| Almudena Camuñas Rodríguez | Nurse | OK | OK | OK | OK | ✓ |
| Amelia Giner Cases | General Practitioner | OK | OK | OK | OK | ✓ |
| Alba Llorens López | Dermatologist | OK | OK | OK | CC | - |
| Beatriz Corcuera Jannone | Dermatologist | OK | OK | OK | OK | ✓ |
| Mª Carmen Martínez Cortés | Nurse | OK | OK | OK | UD | - |
| Carolina Frechina Rebollos | Nurse | OK | OK | OK | OK | ✓ |
| Cristina Morata Pons | Nurse | OK | OK | OK | OK | ✓ |
| Maria Pechuan Calero | Nurse | UD | OK | OK | UE | - |
| Diego Ruiz Rodríguez | Dermatologist | OK | OK | OK | OK | ✓ |
| Francisco Manuel Pintado Vic | Nurse | OK | OK | OK | OK | ✓ |
| Javier Lorenz Dolz | Dermatologist | OK | OK | OK | OK | ✓ |
| Elena Pérez Iturralde | Dermatologist | OK | OK | OK | OK | ✓ |
| Laura Yuste Hidalgo | General Practitioner | OK | OK | OK | CC | - |
| Marta Borrás Cebrián | Nurse | OK | UD | OK | OK | - |
| Mª Carmen Galindo Marín | General Practitioner | OK | OK | OK | CC | - |
| Maria Solera Talamante | Nurse | OK | OK | OK | OK | ✓ |
| Sheila Flores Martínez | Nurse | OK | OK | OK | CC | - |
| Verónica Lozano Orón | Nurse | OK | OK | OK | OK | ✓ |
Key Insights
- ✓100% success rate for Scenarios 1 & 2 (Simulated Use) across all participants
- ✓67% of participants correctly understand that the device report is not a standalone diagnosis (Question 4)
- ✓All professional groups demonstrated competency with the device, with comparable success rates across dermatologists, general practitioners, and nurses
- ✓Diverse participant demographics with 83% female and 17% male representation among healthcare professionals
- ✓The device interface and reports are well-understood by healthcare professionals, meeting usability requirements per IEC 62366-1
Detailed Scenario Notes
| Scenario | Participants with Issues | Issue Type | Description |
|---|---|---|---|
| HCP Use Scenario 1 | None | N/A | All participants successful |
| HCP Use Scenario 2 | None | N/A | All participants successful |
| HCP Use Scenario 3 | |||
| - Question 1 | Maria Pechuan Calero | UD | Incomplete description of report elements |
| - Question 2 | Marta Borrás Cebrián | UD | Imprecise malignancy probability |
| - Question 4 | Alba Llorens López | CC | Suggested it could be diagnostic with caveats |
| Laura Yuste Hidalgo | CC | Suggested high diagnostic suspicion | |
| Mª Carmen Galindo Marín | CC | Indicated diagnostic capability with reliability | |
| Sheila Flores Martínez | CC | Said yes depending on photo quality | |
| Mª Carmen Martínez Cortés | UD | Uncertain answer | |
| Maria Pechuan Calero | UE | Answered "Yes" without qualification | |
| ITP Use Scenario 1 | TBD | TBD | Testing pending |
Questionnaire Results
System Usability Scale (SUS)
Data collection in progress for all participants
| Group | Target Score | Status |
|---|---|---|
| HCP | >70 (Good) | Pending |
| ITP | >70 (Good) | Pending |
| Overall | >70 (Good) | Pending |
Interpretation (Bangor et al.)
- 0-25 Worst Imaginable
- 25.1-51.6 Poor
- 51.7-71 OK
- 71.1-80.7 Good
- 80.8-84.0 Excellent
- 84.1-100 Best Imaginable
AttrackDiff
Data collection in progress for all participants
Subscales:
- Pragmatic Quality (PQ): Perceived usability
- Hedonic Quality (HQ): Stimulation and identification
- Overall Attractiveness (ATT): General appeal
| Group | Target | Status |
|---|---|---|
| HCP | >1 (Positive) | Pending |
| ITP | >1 (Positive) | Pending |
Note: Values > 1 indicate positive perception; values between -1 and 1 are neutral; values < -1 indicate negative perception.
Conclusion
The HCP summative evaluation results for Legit.Health Plus v1.1.0.0 demonstrate:
- Perfect performance in simulated use scenarios (100% success for Scenarios 1 & 2)
- Strong knowledge assessment with 61.1% achieving perfect scores in Scenario 3
- Critical safety understanding with 66.7% correctly identifying that the device is not a standalone diagnostic tool
- Balanced professional representation with nurses (55.6%), dermatologists (27.8%), and general practitioners (16.7%)
Next Steps:
- Complete ITP user testing
- Collect SUS and AttrackDiff questionnaire data
- Finalize comprehensive evaluation report
The HCP testing indicates the device is safe and effective for healthcare professional use, pending completion of ITP evaluation and questionnaire data collection.
References
- Brooke, J. (1996). SUS: A "quick and dirty" usability scale. In P.W. Jordan et al. (Eds.), Usability evaluation in industry (pp. 189-194). Taylor & Francis.
- Hassenzahl, M., Burmester, M., & Koller, F. (2003). AttrackDiff: Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In J. Ziegler & G. Szwillus (Eds.), Mensch & Computer 2003. Interaktion in Bewegung (pp. 187-196). B.G. Teubner.
- Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114-123.