TEST_016 Ensure API compliance with Base64 image format and FHIR standard

Test type

System

Linked activities

• MDS-449

Result

• Passed
• Failed

Description

This test run record documents the results of executing test plans designed to verify two main areas: confirming that the API only accepts images in Base64 format and validating that the data schemas of the request and response content adhere to the FHIR (Fast Healthcare Interoperability Resources) standard. The record includes detailed outcomes of the tests, highlighting any deviations from expected behavior, along with corresponding error messages and potential impacts on system functionality.

These test cases are being conducted for the first time on this version of the medical device.

Run environment

Here are the technical specifications of the runtime environment in which the tests were conducted:

• Operating system: Ubuntu 22.04.3 LTS (Linux kernel version: 5.10.0-051000-generic)
• Hardware specifications:
  • CPU:
    • Model name: AMD Ryzen Threadripper PRO 5995WX (x86_64)
    • Number of cores: 64
    • Thread(s) per core: 2
  • GPU:
    • CUDA version: 12.0
    • Devices:
      • NVIDIA RTX 6000 (49140 MiB)
      • NVIDIA RTX 6000 (49140 MiB)
  • RAM: 252 GB
  • Storage: 6.5 TB
  • Network:
    • Mean speed: 380 Mbps
    • Mean latency: 5 ms
• Other relevant software: No particular software was used.

Test case runs

The following test cases have been executed in this batch:

TEST_016_001

Outcome

• Passed
• Failed

Expected results

• The request with the well-formed payload is accepted, and the response is validated against the FHIR schema without errors. The response aims to closely follow the structure of the FHIR DiagnosticReport resource. While we adapt this resource to fit our specific use case, we ensure that the mandatory and most common keys are implemented.
• The API rejects the invalid JSON payload and responds with a 422 Unprocessable Content status code. The error message informs you that the request body is incorrectly formatted, and no report is returned.

Actual results

• The request with the well-formed payload is accepted, and the response is validated against the FHIR schema without errors. The response aims to closely follow the structure of the FHIR DiagnosticReport resource. While we adapt this resource to fit our specific use case, we ensure that the mandatory and most common keys are implemented:

Status code: 200
{
  "resourceType": "DiagnosticReport",
  "status": "preliminary",
  "category": {
    "coding": [
      {
        "system": "http://terminology.hl7.org/CodeSystem/v2-0074",
        "code": "IMG",
        "display": "Diagnostic Imaging"
      }
    ],
    "text": "Diagnostic Imaging"
  },
  "code": {
    "coding": [
      {
        "system": "http://loinc.org/",
        "code": "100063-7",
        "display": "Primary skin concern"
      },
      {
        "system": "http://loinc.org/",
        "code": "10206-1",
        "display": "Physical findings of Skin Narrative"
      }
    ],
    "text": "Skin Condition Analysis"
  },
  "device": {
    "name": "Legit.Health Plus",
    "manufacturer": "AI LABS GROUP SL",
    "category": {
      "system": "http://hl7.org/fhir/device-category",
      "code": "software",
      "display": "Software"
    },
    "type": {
      "system": "https://www.snomed.org/",
      "code": "468878001",
      "display": "Dermatology picture archiving and communication system application software"
    },
    "url": "https://medical-device.legit.health/"
  },
  "effectiveDateTime": "2024-07-05T09:27:53.142154",
  "conclusions": [
    {
      "coding": {
        "code": "XH5AW4",
        "display": "Haemangioma",
        "system": "https://icd.who.int/en",
        "systemAlias": "ICD-11"
      },
      "probability": 20.61
    },
    {
      "coding": {
        "code": "XH4L78",
        "display": "Nevus",
        "system": "https://icd.who.int/en",
        "systemAlias": "ICD-11"
      },
      "probability": 8.72
    },
    {
      "coding": {
        "code": "2F20.0Y, XH27A6",
        "display": "Compound nevus",
        "system": "https://icd.who.int/en",
        "systemAlias": "ICD-11"
      },
      "probability": 5.12
    },
    {
      "coding": {
        "code": "XH4846",
        "display": "Malignant melanoma",
        "system": "https://icd.who.int/en",
        "systemAlias": "ICD-11"
      },
      "probability": 4.78
    },
    {
      "coding": {
        "code": "XH2HG8",
        "display": "Spitz nevus",
        "system": "https://icd.who.int/en",
        "systemAlias": "ICD-11"
      },
      "probability": 4.76
    }
  ],
  "metrics": {
    "type": {
      "coding": [
        {
          "system": "urn:iso:std:iso:11073:10101",
          "code": "2",
          "display": "Metrics"
        }
      ],
      "text": "Diagnosis Metrics"
    },
    "category": "calculation",
    "sensitivity": 0.91,
    "specificity": 0.98
  },
  "preliminaryFindings": {
    "hasConditionSuspicion": 0.99,
    "needsSpecialistAttention": 1.0,
    "isMalignantSuspicion": 0.05,
    "isPreMalignantSuspicion": 0.05
  },
  "imagingStudySeries": [
    {
      "media": {
        "validity": {
          "metrics": {
            "quality": {
              "valid": true,
              "score": 60,
              "category": "Acceptable"
            },
            "domain": {
              "valid": true,
              "score": 100,
              "category": "Dermatological"
            }
          },
          "isValid": true
        },
        "modality": "Clinical"
      },
      "metrics": {
        "type": {
          "coding": [
            {
              "system": "urn:iso:std:iso:11073:10101",
              "code": "2",
              "display": "Metrics"
            }
          ],
          "text": "Diagnosis Metrics"
        },
        "category": "calculation",
        "sensitivity": 0.91,
        "specificity": 0.98
      },
      "conclusions": [
        {
          "coding": {
            "code": "XH5AW4",
            "display": "Haemangioma",
            "system": "https://icd.who.int/en",
            "systemAlias": "ICD-11"
          },
          "probability": 20.61
        },
        {
          "coding": {
            "code": "XH4L78",
            "display": "Nevus",
            "system": "https://icd.who.int/en",
            "systemAlias": "ICD-11"
          },
          "probability": 8.72
        },
        {
          "coding": {
            "code": "2F20.0Y, XH27A6",
            "display": "Compound nevus",
            "system": "https://icd.who.int/en",
            "systemAlias": "ICD-11"
          },
          "probability": 5.12
        },
        {
          "coding": {
            "code": "XH4846",
            "display": "Malignant melanoma",
            "system": "https://icd.who.int/en",
            "systemAlias": "ICD-11"
          },
          "probability": 4.78
        },
        {
          "coding": {
            "code": "XH2HG8",
            "display": "Spitz nevus",
            "system": "https://icd.who.int/en",
            "systemAlias": "ICD-11"
          },
          "probability": 4.76
        }
      ],
      "preliminaryFindings": {
        "hasConditionSuspicion": 0.99,
        "needsSpecialistAttention": 1.0,
        "isMalignantSuspicion": 0.05,
        "isPreMalignantSuspicion": 0.05
      }
    }
  ],
  "analysisDuration": "0.6285 secs"
}

• The response JSON from the API closely follows the structure of the DiagnosticReport resource as defined by the FHIR standard, sharing keys like status, category, code, effectiveDateTime, and conclusion.

• The API rejects the invalid JSON payload and responds with a 422 Unprocessable Content status code. The error message informs you that the request body is incorrectly formatted, and no report is returned:

Status code: 422
{
  "detail": [
    {
      "type": "missing",
      "loc": [
        "body",
        "media"
      ],
      "msg": "Field required",
      "input": {
        "user": {
          "reference": [
            "fake-patient-id"
          ]
        },
        "images": [
          {
            "contentType": "image/jpeg",
            "data": "/9j/4AAQSkZJRgABAQAAAQABAAD//gAIB..."
          }
        ]
      }
    },
    {
      "type": "extra_forbidden",
      "loc": [
        "body",
        "user"
      ],
      "msg": "Extra inputs are not permitted",
      "input": {
        "reference": [
          "fake-patient-id"
        ]
      }
    },
    {
      "type": "extra_forbidden",
      "loc": [
        "body",
        "images"
      ],
      "msg": "Extra inputs are not permitted",
      "input": [
        {
          "contentType": "image/jpeg",
          "data": "/9j/4AAQSkZJRgABAQAAAQABAAD//gAIB..."
        }
      ]
    }
  ]
}

• The API response contains only the error message displayed above. It clearly states that the required media field is missing and that the fields user and images are unrecognized.

Remarks

No comments to add. The test was carried out manually without any issues.

TEST_016_002

Outcome

• Passed
• Failed

Expected results

• Both endpoints successfully handle their respective requests containing the Base64 image and generate a well-structured preliminary report.
• The response from the /severity-assessment endpoint includes any image data in Base64 format. If everything worked correctly, you can find the image results in the response at the JSON path: patientEvolution.ascorad.media.attachments[0].data.

Actual results

• Both endpoints successfully handle their respective requests containing the Base64 image and generate a well-structured preliminary report:

• Preliminary report from diagnosis support service

Status code: 200
{
  "resourceType": "DiagnosticReport",
  "status": "preliminary",
  "category": {
    "coding": [
      {
        "system": "http://terminology.hl7.org/CodeSystem/v2-0074",
        "code": "IMG",
        "display": "Diagnostic Imaging"
      }
    ],
    "text": "Diagnostic Imaging"
  },
  "code": {
    "coding": [
      {
        "system": "http://loinc.org/",
        "code": "100063-7",
        "display": "Primary skin concern"
      },
      {
        "system": "http://loinc.org/",
        "code": "10206-1",
        "display": "Physical findings of Skin Narrative"
      }
    ],
    "text": "Skin Condition Analysis"
  },
  "device": {
    "name": "Legit.Health Plus",
    "manufacturer": "AI LABS GROUP SL",
    "category": {
      "system": "http://hl7.org/fhir/device-category",
      "code": "software",
      "display": "Software"
    },
    "type": {
      "system": "https://www.snomed.org/",
      "code": "468878001",
      "display": "Dermatology picture archiving and communication system application software"
    },
    "url": "https://medical-device.legit.health/"
  },
  "effectiveDateTime": "2024-07-05T10:45:23.801254",
  "conclusions": [
    {
      "coding": {
        "code": "EA80",
        "display": "Atopic dermatitis",
        "system": "https://icd.who.int/en",
        "systemAlias": "ICD-11"
      },
      "probability": 65.39
    },
    {
      "coding": {
        "code": "6B25.1",
        "display": "Excoriation disorder",
        "system": "https://icd.who.int/en",
        "systemAlias": "ICD-11"
      },
      "probability": 3.22
    },
    {
      "coding": {
        "code": "EB05",
        "display": "Urticaria",
        "system": "https://icd.who.int/en",
        "systemAlias": "ICD-11"
      },
      "probability": 3.07
    },
    {
      "coding": {
        "code": "EA80",
        "display": "Eczema",
        "system": "https://icd.who.int/en",
        "systemAlias": "ICD-11"
      },
      "probability": 2.79
    },
    {
      "coding": {
        "code": "1E91.Z",
        "display": "Herpes zoster",
        "system": "https://icd.who.int/en",
        "systemAlias": "ICD-11"
      },
      "probability": 2.49
    }
  ],
  "metrics": {
    "type": {
      "coding": [
        {
          "system": "urn:iso:std:iso:11073:10101",
          "code": "2",
          "display": "Metrics"
        }
      ],
      "text": "Diagnosis Metrics"
    },
    "category": "calculation",
    "sensitivity": 0.82,
    "specificity": 0.98
  },
  "preliminaryFindings": {
    "hasConditionSuspicion": 1.0,
    "needsSpecialistAttention": 1.0,
    "isMalignantSuspicion": 0.0,
    "isPreMalignantSuspicion": 0.0
  },
  "imagingStudySeries": [
    {
      "media": {
        "validity": {
          "metrics": {
            "quality": {
              "valid": true,
              "score": 87,
              "category": "Acceptable"
            },
            "domain": {
              "valid": true,
              "score": 100,
              "category": "Dermatological"
            }
          },
          "isValid": true
        },
        "modality": "Clinical"
      },
      "metrics": {
        "type": {
          "coding": [
            {
              "system": "urn:iso:std:iso:11073:10101",
              "code": "2",
              "display": "Metrics"
            }
          ],
          "text": "Diagnosis Metrics"
        },
        "category": "calculation",
        "sensitivity": 0.82,
        "specificity": 0.98
      },
      "conclusions": [
        {
          "coding": {
            "code": "EA80",
            "display": "Atopic dermatitis",
            "system": "https://icd.who.int/en",
            "systemAlias": "ICD-11"
          },
          "probability": 65.39
        },
        {
          "coding": {
            "code": "6B25.1",
            "display": "Excoriation disorder",
            "system": "https://icd.who.int/en",
            "systemAlias": "ICD-11"
          },
          "probability": 3.22
        },
        {
          "coding": {
            "code": "EB05",
            "display": "Urticaria",
            "system": "https://icd.who.int/en",
            "systemAlias": "ICD-11"
          },
          "probability": 3.07
        },
        {
          "coding": {
            "code": "EA80",
            "display": "Eczema",
            "system": "https://icd.who.int/en",
            "systemAlias": "ICD-11"
          },
          "probability": 2.79
        },
        {
          "coding": {
            "code": "1E91.Z",
            "display": "Herpes zoster",
            "system": "https://icd.who.int/en",
            "systemAlias": "ICD-11"
          },
          "probability": 2.49
        }
      ],
      "preliminaryFindings": {
        "hasConditionSuspicion": 1.0,
        "needsSpecialistAttention": 1.0,
        "isMalignantSuspicion": 0.0,
        "isPreMalignantSuspicion": 0.0
      }
    }
  ],
  "analysisDuration": "0.6216 secs"
}

• Preliminary report from severity assessment service

Status code: 200
{
  "resourceType": "DiagnosticReport",
  "status": "preliminary",
  "category": {
    "coding": [
      {
        "system": "http://terminology.hl7.org/CodeSystem/v2-0074",
        "code": "IMG",
        "display": "Diagnostic Imaging"
      }
    ],
    "text": "Diagnostic Imaging"
  },
  "code": {
    "coding": [
      {
        "system": "http://loinc.org/",
        "code": "100063-7",
        "display": "Primary skin concern"
      },
      {
        "system": "http://loinc.org/",
        "code": "10206-1",
        "display": "Physical findings of Skin Narrative"
      }
    ],
    "text": "Skin Condition Analysis"
  },
  "device": {
    "name": "Legit.Health Plus",
    "manufacturer": "AI LABS GROUP SL",
    "category": {
      "system": "http://hl7.org/fhir/device-category",
      "code": "software",
      "display": "Software"
    },
    "type": {
      "system": "https://www.snomed.org/",
      "code": "468878001",
      "display": "Dermatology picture archiving and communication system application software"
    },
    "url": "https://medical-device.legit.health/"
  },
  "effectiveDateTime": "2024-07-05T10:52:26.374451",
  "media": {
    "validity": {
      "metrics": {
        "quality": {
          "valid": true,
          "score": 87,
          "category": "Acceptable"
        },
        "domain": {
          "valid": true,
          "score": 100,
          "category": "Dermatological"
        }
      },
      "isValid": true
    },
    "modality": "Clinical"
  },
  "patientEvolution": {
    "ascorad": {
      "name": "ascorad",
      "score": {
        "text": "score",
        "value": 40.8,
        "interpretation": "Severe"
      },
      "items": {
        "surface": {
          "type": "visual-sign",
          "code": {
            "text": "surface-area"
          },
          "coverage": 40,
          "localCoverage": 4
        },
        "erythema": {
          "type": "visual-sign",
          "code": {
            "text": "erythema"
          },
          "intensity": 2,
          "intensityRaw": 59
        },
        "crusting": {
          "type": "visual-sign",
          "code": {
            "text": "crusting"
          },
          "intensity": 2,
          "intensityRaw": 82
        },
        "swelling": {
          "type": "visual-sign",
          "code": {
            "text": "swelling"
          },
          "intensity": 2,
          "intensityRaw": 56
        },
        "excoriation": {
          "type": "visual-sign",
          "code": {
            "text": "excoriation"
          },
          "intensity": 2,
          "intensityRaw": 73
        },
        "lichenification": {
          "type": "visual-sign",
          "code": {
            "text": "lichenification"
          },
          "intensity": 1,
          "intensityRaw": 29
        },
        "dryness": {
          "type": "visual-sign",
          "code": {
            "text": "dryness"
          },
          "intensity": 1,
          "intensityRaw": 47
        },
        "itchiness": {
          "type": "subjective-symptom",
          "code": {
            "text": "itchiness"
          },
          "intensity": 3
        },
        "sleeplessness": {
          "type": "subjective-symptom",
          "code": {
            "text": "sleeplessness"
          },
          "intensity": 2
        }
      },
      "media": {
        "attachments": [
          {
            "title": "segmentation",
            "contentType": "image/jpeg",
            "data": "/9j/4AAQSk...",
            "height": 527,
            "width": 817
          }
        ]
      }
    }
  },
  "analysisDuration": "0.3725 secs"
}

• The response from the /severity-assessment endpoint includes any image data in Base64 format. If everything worked correctly, you can find the image results in the response at the JSON path: patientEvolution.ascorad.media.attachments[0].data:

• If we look at the preliminary report generated by the severity assessment service, we can find a Base64 string for a clinical image processed by AI at the JSON path patientEvolution.ascorad.media.attachments[0].data. The string is truncated to keep the report readable and prevent overwhelming the readers with lengthy data.

Remarks

No comments to add. The test was carried out manually without any issues.

TEST_016_003

Outcome

• Passed
• Failed

Expected results

• Both endpoints are unable to process the request and return an error message indicating that the image must be encoded in Base64 format.

Actual results

• Both endpoints are unable to process the request and return an error message indicating that the image must be encoded in Base64 format:

• Response body from the /diagnosis-support endpoint:

Status code: 422
{
  "detail": [
    {
      "type": "value_error",
      "loc": [
        "body",
        "media",
        0,
        "data"
      ],
      "msg": "Value error, Only base64 data is allowed",
      "input": "b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\...'",
      "ctx": {
        "error": {}
      }
    }
  ]
}

• Response body from the /severity-assessment endpoint:

Status code: 422
{
  "detail": [
    {
      "type": "value_error",
      "loc": [
        "body",
        "media",
        "data"
      ],
      "msg": "Value error, Only base64 data is allowed",
      "input": "b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\...'",
      "ctx": {
        "error": {}
      }
    }
  ]
}

Remarks

No comments to add. The test was carried out manually without any issues.

Summary of results

• Total cases: 3
• Passed: 3
• Failed: 0
• Pass rate: 100 %

Defects and issues

Defect ID	Description	Severity	Status	Reported by	Assigned to	Activities generated	Remarks

Observations and recommendations

All planned test cases were executed successfully, with a 100% pass rate. This indicates that the system performed as expected under the defined scenarios.

Additionally, no defects were found during this testing cycle. The absence of defects suggests a high level of system stability and reliability.

Signature meaning

The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:

• Tester: JD-017, JD-009, JD-004
• Approver: JD-005