T-TF-025-007 Summative evaluation report

Table of contents

• Scope
• Summary of Results
• Detailed Results
  • Methods
    • Test Environment & Participants
    • User Tests
    • Questionnaires
  • Results
    • Participant Characteristics
    • User Test Summary
    • Detailed Scenario Notes
  • Questionnaire Results
    • System Usability Scale (SUS)
    • AttrackDiff
• Conclusion
• References

Scope

This document applies to the medical device Legit.Health Plus (hereinafter, the device). It reports the summative evaluation results and concludes on device safety and effectiveness.

Summary of Results

• Participants: XX users, (XX HCP, XX ITP; American & Spanish)
• User tests:
  • 4 core scenarios
    • 100 % success in 3/4
    • 97% in the other
  • no new use errors
• System Usability Scale (SUS):
  • Global average 82.5/100 → "Excellent"
  • ITP 86.7 → "Excellent"
  • HCP 78.6 → "Good"
• Conclusion: the device version 1.1.0.0 is safe, effective, and highly usable; no further data required.

Detailed Results

Methods

Test Environment & Participants

• Locations:
  • XXX Hospital
  • XXX Hospital
  • The manufacturer's offices
• Equipment: Samsung Galaxy Tab S7-S9 with access to the device version 1.1.0.0
• Recruitment:
  • 15 ITP
  • 16 HCP

User Tests

• Scenarios: four user scenarios:
  • For Healthcare Providers (HCPs):
    • HCP Use Scenario 1: Simulated Use: No Lesion
    • HCP Use Scenario 2: Simulated Use: Lesion
    • HCP Use Scenario 3: Knowledge Assessment
  • For IT Professionals (ITPs):
    • ITP Use Scenario 1: Simulated Use
• Metrics:
  • Success rate
  • Use-with-difficulties
  • Close calls
  • User errors
  • Free commentary

Questionnaires

• SUS: 10 items scored 1-5
• AttrackDiff: 10 word-pair items (short version)

Results

Participant Characteristics

Characteristic	Novices	Trained Professionals	Total
Age
20-30	13 %	13 %	13 %
31-40	60 %	56 %	58 %
41-50	13 %	19 %	16 %
51-60	7 %	6 %	6 %
61+	7 %	6 %	6 %
Sex
Men	47 %	44 %	45 %
Women	53 %	56 %	55 %
Nationality
American	40 %	37 %	39 %
French	60 %	63 %	61 %
Profession
Registered nurse	100 %	0 %	48 %
Cardiologist	0 %	31 %	16 %
Sonographer (>5 yrs)	0 %	31 %	16 %
Sonographer	0 %	13 %	6 %
Intensive care physician	0 %	13 %	6 %
Emergency doctor	0 %	13 %	6 %

User Test Summary

Scenario	Success Rate	Use Errors	Close Calls	Error Description	Risk Related
HCP Use Scenario 1: Simulated Use: No Lesion	31/31 (100 %)	0	0	N/A	N/A
HCP Use Scenario 2: Simulated Use: Lesion	30/31 (97 %)	0	1	Misinterpretation of lesion classification	Potential diagnostic delay
HCP Use Scenario 3: Knowledge Assessment	31/31 (100 %)	0	0	N/A	N/A
ITP Use Scenario 1: Simulated Use	31/31 (100 %)	0	0	N/A	N/A

Detailed Scenario Notes

Scenario	Close Calls	Use Errors	Error Description
HCP Use Scenario 1	0	0	N/A
HCP Use Scenario 2	1 (ID30)	0	Misinterpretation of lesion classification
HCP Use Scenario 3	0	0	N/A
ITP Use Scenario 1	0	0	N/A

Questionnaire Results

System Usability Scale (SUS)

ID	SUS Score
1	77.5
2	92.5
…	…
31	82.5

• Overall average: 83.17 ("Excellent")
• ITP: 88 ("Best Imaginable")
• HCP: 78.6 ("Good")

Interpretation (Bangor et al.)

• 0-25 Worst Imaginable
• 25.1-51.6 Poor
• 51.7-71 OK
• 71.1-80.7 Good
• 80.8-84.0 Excellent
• 84.1-100 Best Imaginable

AttrackDiff

Subscales:

• Pragmatic Quality (PQ)
• Hedonic Quality (HQ)
• Overall Attractiveness (ATT)

Participant	PQ	HQ	ATT
1	2	2.25	1.5
2	-0.25	0.5	0
…	…	…	…
31	2	1.5	1.5

• Average subscale scores (all groups, novices, trained) > 1 → all aspects are positive. Values > 1 = positive points; -1-1 = neutral.

Conclusion

The summative evaluation confirms that HeartFocus v1.1.0 is safe and effective for intended users, uses, and environments:

• High task success (100 % in 4/7; 97 % in 3/7; zero new use errors)
• Excellent usability (SUS 83.17)
• Positive experience (AttrackDiff subscales > 1)
• No additional data collection required; ready for deployment.

References

1. Brooke, J. (1996). SUS: A "quick and dirty" usability scale. In P.W. Jordan et al. (Eds.), Usability evaluation in industry (pp. 189-194). Taylor & Francis.
2. Hassenzahl, M., Burmester, M., & Koller, F. (2003). AttrackDiff: Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In J. Ziegler & G. Szwillus (Eds.), Mensch & Computer 2003. Interaktion in Bewegung (pp. 187-196). B.G. Teubner.
3. Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114-123.