Usability Testing Documentation Guide

Overview

This directory contains usability testing documentation for the device's Human Factors Engineering validation. The testing follows IEC 62366-1 standards and FDA guidance for medical device usability.

Directory Structure

usability/
├── t-tf-025-006/           # Summative evaluation questionnaires (expected answers)
│   └── t-tf-025-006.mdx    # Knowledge assessment questions and expected answers
├── t-tf-025-007/           # Test results and responses
│   ├── index.mdx           # Main summative evaluation report
│   └── 2025-10-22-hcp-results/
│       ├── *.pdf           # Original scanned questionnaires (handwritten)
│       └── usability_test_responses.json  # Structured data from PDFs
└── CLAUDE.md              # This file

Related Components

HcpUsabilityTestingResults Component

• Location: /apps/qms/src/components/HcpUsabilityTestingResults/
• Purpose: Visualizes and analyzes HCP usability testing data
• Documentation: See component's CLAUDE.md for implementation details
• Usage: Imported in t-tf-025-007/index.mdx to display comprehensive test results

Working with Usability Test Data

Understanding the Test Structure

The usability tests consist of knowledge assessment questions that Healthcare Professionals (HCPs) must answer after using the device. Each test includes:

1. Question 1: What information does the device report show?
   • Expected: Top 5 predictions, disease intensity quantification, clinical signs
2. Question 2: What is the probability of malignancy?
   • Expected: The specific percentage from the report
3. Question 3: What conditions were detected?
   • Expected: List of conditions shown in the report
4. Question 4: Can the report act as a diagnosis?
   • Expected: No (it's a support tool, not a diagnostic device)

Processing Handwritten Responses

When converting handwritten Spanish questionnaires to structured JSON:

Key Considerations

1. Language: All responses are in Spanish - preserve original text exactly as written
2. Handwriting Variations: Be generous in interpretation of unclear handwriting
3. Name Formatting: Convert ALL CAPS names to Title Case for consistency
4. Date Normalization: Standardize all dates to ISO 8601 format (YYYY-MM-DD)
5. Scoring Benevolence: When ambiguous, err on the side of accepting answers as correct

JSON Data Structure

The usability_test_responses.json file uses this enhanced structure:

{
  "usability_test_responses": [
    {
      "name": "Full Name in Title Case",
      "id": "DNI/NIE/passport number",
      "date": "YYYY-MM-DD",
      "profession": "Dermatologist|General Practitioner|Nurse",
      "scenarios": {
        "scenario_1": {
          "name": "Simulated Use: No Lesion",
          "score": "OK"
        },
        "scenario_2": {
          "name": "Simulated Use: Lesion",
          "score": "OK"
        },
        "scenario_3": {
          "name": "Knowledge Assessment",
          "questions": [
            {
              "question": 1,
              "answer": "Original answer text in Spanish",
              "score": "OK|UD|CC|UE"
            }
          ]
        }
      }
    }
  ]
}

Note: The structure now includes all three HCP use scenarios, with Scenarios 1 & 2 having simple pass/fail scores, and Scenario 3 containing detailed question-by-question assessment.

Scoring System

Based on IEC 62366-1 and FDA guidance:

• OK (Success): Correct answer according to expected response
• UD (Use Difficulty): Struggled but eventually answered correctly
• CC (Close Call): Initially incorrect but self-corrected
• UE (Use Error): Incorrect or incomplete answer

Scoring Guidelines

1. Be Benevolent: When interpretation is ambiguous, favor "OK" or "CC" over "UE"
2. Context Matters: Consider if the respondent shows understanding even if wording differs
3. Question 4 Special Handling:
   • "No" or variations → OK
   • "Can help but needs clinical validation" → OK
   • "Yes but with conditions" → CC
   • Unqualified "Yes" → UE

Data Processing Workflow

1. Extract PDF Data:

   • Read each PDF file from the results directory
   • OCR is not needed - PDFs contain readable text layers
   • Preserve exact Spanish text including accents

2. Normalize Personal Data:

   • Names: Convert from ALL CAPS to Title Case
   • Dates: Convert various formats (22/10/25, 22-OCT-2025, etc.) to YYYY-MM-DD
   • IDs: Preserve exactly as written

3. Structure Answers:

   • Convert from string array to object array
   • Add question numbers (1-4)
   • Apply scoring based on expected answers

4. Quality Checks:

   • Verify all 18 respondents are included
   • Ensure all 4 questions per respondent
   • Validate date formats are consistent
   • Check name formatting is Title Case

Important Notes

• Language Preservation: Keep all answers in original Spanish
• No Translation: Do not translate responses to English
• Exact Transcription: Preserve numbers, percentages, and medical terms exactly
• Scoring Documentation: When scoring is ambiguous, document reasoning
• Version Control: This data feeds into formal V&V records - maintain audit trail

Regulatory Context

This usability testing is part of:

• IEC 62366-1 compliance for medical device usability
• FDA Human Factors guidance implementation
• Risk management per ISO 14971
• Clinical validation requirements for Class IIa medical devices

The results demonstrate that healthcare professionals can correctly understand and interpret the device outputs, which is critical for safe and effective use.