R-TF-012-016 Software usability test guide

A guide on what to do, not how to do it

Usability testing is a technique utilized in user-centered interaction design to evaluate a product by observing how real users interact with it.

Guiding users with technology-specific commands during a usability test can obscure their true behavior and natural interactions with the software. The main goal is to observe genuine user actions and reactions under conditions that mimic real-world usage. When users receive explicit instructions, they are less likely to encounter and identify issues that designers might not have anticipated, resulting in less authentic feedback about the user experience.

Tests are most effective when users can freely explore the UI, as this promotes problem-solving and helps testers identify areas where users naturally struggle without guidance. Over-guidance can make the software appear usable in a controlled test environment but fail to meet user needs in real-world scenarios. Therefore, to develop a truly user-friendly product, it's crucial to observe unguided interactions that reflect an authentic user experience.

User expertise

Users should have the following prior knowledge to effectively perform the usability tests designed for this version of the medical device's REST API:

Basic understanding of REST APIs:
- Knowledge of what a REST API is and how it operates.
- Familiarity with common HTTP methods (GET, POST, PUT, DELETE), and their appropriate use in different scenarios.
Experience with API authentication:
- Understanding of common API authentication mechanisms, including the use of API keys and access tokens.
Familiarity with JSON:
- Ability to read, write and parse JSON data, as it is a common data format used in REST APIs.
- Understanding of how to interpret and use a JSON path.
Knowledge of HTTP status codes:
- Familiarity with common HTTP status codes (e.g., 200, 404, 500) and their meanings.
Error handling knowledge:
- Familiarity with handling and interpreting error messages and responses from APIs.
Experience with API Clients:
- Basic skills using API clients like Postman, curl, or similar tools to make API requests.
Basic programming skills:
- Ability to write simple scripts in a programming language such as Python, JavaScript, or similar to interact with the API.
- Understanding of how to use libraries (e.g., requests in Python) for making HTTP requests.
Basic understanding of networking:
- Familiarity with concepts such as URLs, endpoints, requests, payloads and headers.
Comfort with technical documentation:
- Ability to read and follow technical documentation, including navigating through sections, understanding examples, and finding relevant information.

Equipment prerequisites

Stable internet connection: Perform a speed test at Fast and check that your connection speed is at least 30 Mbps, your upload speed is at least 20 Mbps, and that the download latency is below 20 ms. You can view all these metrics by clicking on “Show more info” once the speed test has finished.
REST API client: A tool or application to send HTTP requests (e.g., Postman, Python requests, curl)
Access to the API documentation: All participants involved in the usability testing have full access to the online API documentation relevant to the device being tested here. In case the instructions provided throughout this document are not clear enough, you can find in the API documentation more exhaustive information on relevant material like: descriptions of endpoints, the authentication method, request/response data schemas, error codes, or a detailed list of available scoring systems. API documentation could be very helpful in the troubleshooting process during usability testing.

Test scenarios

Our REST API is hosted at https://medical-device-pre.legit.health. All the usability tests you are about to conduct are on this API. The endpoints mentioned hereafter must be preceded by this web address or otherwise you will not find the services.

The order in which you must run the tests is determined by the numbering assigned to each one. However, this is only a requirement for the authentication tests (the first ones), since it is essential to have gained the knowledge and skills in authentication in order to move forward. Once these tests are completed, the subsequent test cases can be executed at any point in the usability testing process as long as the order within the category to which they belong (diagnosis support, severity assessment, etc.) is respected.

The estimated average time to complete all the usability tests for this exercise, including the setup stage, is about 1.5 hours.

You can access the API's Instructions for Use (IFU) and additional test support materials at this link.

For more detailed API documentation, refer to the relative path /docs from the base API URL. This documentation follows the format provided by the OpenAPI Specification. There, you'll find information on available endpoints and their operations (GET, POST, etc.), input and output schemas for each operation, authentication methods, as well as contact details, licensing, terms of use, and other relevant information.

Authentication data

To prevent misuse of our API, we have protected all endpoints of the medical device user interface. For this reason, you must identify yourself on every API call you make.

Your authentication credentials are the following:

Username: testuser@legit.api
Password: @634&lMjH52#ipK@

How to authenticate

Initially, you will need to log in using your username and password to receive an access token. This token will represent your authentication session. To do this, include the authentication credentials as Form-data in your request to the /login endpoint, using the following key names:

{
  "username": <Your-Username>,
  "password": <Your-Password>,
}

The access token you receive should look like this:

{
  "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."
}

For subsequent requests to any protected endpoint, include the access token in the request headers. This is what the header should look like:

{
  "Authorization": "Bearer <Your-Access-Token>"
}

Token expiration

Don't panic if the token you sent to the API is no longer valid. It has probably reached the end of its lifetime. For security requirements, we establish an expiration time for the token. This expiration time may vary, but the solution is always the same: acquire another token at the /login endpoint by following the previous authentication instructions.

While it might seem clear, keep your username, password, and access token confidential to prevent unauthorized access to your account.

Initial setup

Before initiating the usability tests for the REST API, you need to prepare adequately to ensure that the testing process is efficient and effective. This is a brief summary of the preliminary tasks you should complete to configure your testing environment before you start with the tests:

Downloading images from specified URLs, encoding them into Base64 format, and preprocessing them for submission through API requests.

Please, it is very important that you carefully follow the instructions below so that you have the necessary material to run the tests.

Encoding images to Base64

To begin the setup, you need to download a specific set of images, which will be used repeatedly as part of the API testing. Use the following URLs to download each image to any valid local path:

After downloading the images, the next step is to convert them into Base64 format. If you are unfamiliar with how to encode images to Base64, you can use the free online tool provided at Base64 Image Encoder.

Here's how to use the encoding tool:

Visit Base64 Image Encoder.
Drag and drop your downloaded image or use the file selector to upload your image to the tool.
Wait for the image to upload and be encoded. Click on the “Copy image” option displayed next to the converted image.

Download images as JPEG

The tool only recognizes images in JPEG (or JPG) format. If you are going to use it to encode the images, it is required that you save them with the suffix .jpeg or .jpg before uploading to the web.

Once you have the Base64 encoded data for each image, you will need to make a slight modification before using it in your API requests. Each Base64 string provided by the tool includes a prefix that looks something like data:image/{IMAGE_FORMAT};base64,. Paste the Base64 string into a text editor and remove this prefix, as it will not be recognized by the API when processing the payload of the request.

Example

If the encoded data looks like this:

data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD...

You should trim it to:

/9j/4AAQSkZJRgABAQAAAQABAAD...

With the Base64 strings prepared, you can now save each one in the location of your choice (either a variable in the programming language you are using, a configuration variable if you are working with an API client such as Postman, etc.). In each test you will be instructed on which field of the request to insert the encoded image(s) and other aspects necessary to format the body of the request correctly.

Decode Base64 images

During the usability tests, you may encounter tasks that require decoding Base64 encoded images. If you do not have a specific Base64-to-image decoder available, we recommend using the online tool provided by Base64 Guru. This tool is free, user-friendly and allows you to quickly decode and visualize Base64 images directly in your browser without needing to download any additional software.

Please, follow these steps to decode and visualize a Base64 image using this tool:

Open your web browser and navigate to Base64 Guru's image decoder.
On the webpage, you will find a text box where you can paste the Base64 encoded string. This is the data you received from the REST API during the usability test.
Click on the "Decode Base64 to Image" button. This action will decode the Base64 string and generate the corresponding image.
Once the image is decoded, it will be displayed directly in the browser window. You can view the image without needing to download it.

1. Authentication endpoint tests

Objective: Ensure the authentication process is user-friendly, secure, and reliable.
Endpoint: /login

Test case 1.1: Credential validation

Scenario: Successful authentication with valid credentials and acquisition of a valid access token.
Prerequisites:
- User account is already created and credentials are known.
Test data: No specific data is required for this test.
HTTP method: POST
Steps:
- Attempt to log in with correct username and password.
- Attempt to log in with an incorrect username, correct password, and vice versa.
- Attempt to log in with both incorrect username and password.

Expected outcome:

Successful login returns a JWT token. The API returns a message like this:

Access Token for testuser@legit.api: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhbGVqYW5kcm9AbGVnaXQuaGVhbHRoIiwiZXhwIjoxNzE4MTA5NDc5fQ.Idma9iBQo5DaRJfR-nDr8KAk_fc9ruM55WYoRiyIddXfb4DZAxDUmbl8bEBXNOJrozNGY5UFdqLjjstjHoFUoMW5tlsEzW4b9-tNS4U0FJqTNSk7wMId1i_USWNSJnHhWzcdYLPwnkz--77fDoy-H3KZ-2pGqtRHhxPzXODR_9fiuu_ye3NgHU-UCI_cdslNj8S_PdVoFs7WBmjPEiFOMQxIf_plTKXRsMjSSIxCNl6l0qVLXrE0iKBSqlyCyI4vkG149eEiZifzFjdZjzGIkZgOkEorFx-_IQiDzkssh2Ki9w07WADjKny6M3uk4pD4jLhDdaLG5tIXIHNIwcg9vfIqL7AJHXr8KTr1w2WRC41zOxSp7-Iy0QWemQXnEdpfAdRCUsWIYyZrU5pJor9Bn_3w4FG51s8ieCjybuDQNpRdtloUmQuW8Wjw_YiTSYasBbS1xn6WVmZk96himG9Mhg2Vw0aMlGNQ6J1yJe2BbugkFDk4j3f2FaOesBXY1yvBhGy_ldSmcz-5-k7kbW38xS5whMPAXtG0vNDagLWvu4c6r9354HRV9CJ5_K_UdjpMi-2CFGryxTUy8V1TlQ4fRiXo-VXxfHDImg8ZNX1HbYV1xsGdLpvHlpDMKDfnfm7UDQpMKWvuJ3b4pecFIu2GfEWNpjBbPa8HFW90VRdahH0

Failed login attempts return a 401 Unauthorized status code without disclosing which part of the credentials is incorrect. The error message should look like this:
```
{ "detail": "Invalid credentials" }
```

Additional notes: No additional comments are required for this test.

Test case 1.2: Token expiry and refresh

Scenario: Handling token expiration and refreshing tokens.
Prerequisites:
- User account is already created and credentials are known.
Test data: No specific data is required for this test.
HTTP method: POST
Steps:
- Log in to receive an access token.
- Wait for the token to expire (here we ask for your patience because the maximum expiration time allowed is up to 12 hours).
- Attempt to use the expired token to access the /diagnosis-support endpoint. For now, just leave the body of the request empty and only worry about sending the correct headers.
- Attempt to refresh the token.
Expected outcome:
- The API correctly identifies expired tokens and denies access to the protected endpoint, responding with a 401 Unauthorized status code. The error message returned should look like this:
```
{ "detail": "Invalid credentials" }
```
- Refreshing the token is straightforward and successful.
Additional notes: No additional comments are required for this test.

2. Diagnosis support endpoint tests

Objective: Ensure the diagnosis support service accurately handles image data and provides correct diagnoses.
Endpoint: /diagnosis-support

Test case 2.1: The simplest workflow

Scenario: A user is interested in doing a quick check of a patient's skin abnormality using a single image, and then saving the results in their storage system to later forward them to a physician.
Prerequisites:
- User account is already created and credentials are known.
- User downloaded the image(s) labeled "Melanoma" in the setup stage. Then, the user converted the image(s) to Base64 and saved the encoded image(s) in a location close to the working environment.

Test data: The JSON object required for this test as the request body is:

{
  "subject": {"reference": "fake-patient-id"},
  "media": [
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Melanoma-Image>,
    }
  ]
}

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Submit the request to the API including the JSON object available in "Test data".
- Record the HTTP status code of the response(s) and capture the response body.
- Save the preliminary clinical results (the whole response body) provided by the API in a JSON file within a secure environment. Name the file as you prefer, just make sure you don't accidentally overwrite any existing files.

Expected outcome:

The API responds with a 200 OK status code for the request you sent.

The API returns an accurate differential diagnosis for the "Melanoma" image. You can find this information in the top-level conclusions section of the JSON contained in the response:

"conclusions": [
  {
    "coding": {
      "code": "XH4846",
      "display": "Malignant melanoma",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 99.82
  },
  {
    "coding": {
      "code": "2F20.2",
      "display": "Congenital nevus",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 0.02
  },
  {
    "coding": {
      "code": "2F20.1",
      "display": "Atypical nevus",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 0.02
  },
  {
    "coding": {
      "code": "2F20.0Y",
      "display": "Nevus incipiens",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 0.01
  },
  {
    "coding": {
      "code": "2F20",
      "display": "Melanocytic nevus",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 0.01
  }
]

The JSON values you receive may not exactly match the values in the JSON above, but they should be very close or at least match in the order of probability in which the predicted pathologies are listed.

The JSON file is successfully stored to disk, without causing storage-related errors (e.g. insufficient disk space) or accidentally overwriting other files.

Additional notes: No additional comments are required for this test.

Test case 2.2: Beyond the boundaries of the standard operating flow

Scenario: A user sends low quality image(s) or/and image(s) outside the dermatological domain, expecting an accurate and reliable diagnosis.
Prerequisites:
- User account is already created and credentials are known.
- User downloaded the images labeled "Low Quality" and "Out Of Skin Domain" during the setup stage. Then, the user converted the images to Base64 and saved the encoded images in a location close to the working environment.

Test data: The JSON object required for this test as the request body is:

{
  "subject": {"reference": "fake-patient-id"},
  "media": [
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Image>,
    }
  ]
}

Replace the <Base64-Encoded-Image> placeholder as appropriate in each step of the test.

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Submit the request to the API including the JSON object available in "Test data" with the "Low Quality" image.
- Submit the request to the API including the JSON object available in "Test data" with the "Out Of Skin Domain" image.
- Record the HTTP status code of the response(s) and capture the response body.

Expected outcome:

The API responds with a 200 OK status code for all the requests you sent.
The API handles the low-quality image gracefully, but the result indicates that the image has not passed the quality check. You can find this information under the top-level key imagingStudySeries of the JSON. The data type of the value associated with this key is a list, in this case with a single element. The relevant key in this element is the one that indicates whether the submitted image has passed the quality control, and it is located in the media.validity.metrics.quality.valid path:
```
"imagingStudySeries": [
  {
    "media": {
      "validity": {
        "metrics": {
          "quality": {
            "valid": false,
            "score": 48,
            "category": "Low quality"
          },
          ...,
        },
        ...,
      },
      ...,
    },
    ...,
  }.
  ...,
]
```
The value of the score field may be slightly different from yours, but the important thing is that the valid field is false in both cases.

Non-dermatological image does not result in false positive. You can find this information in the key given by the JSON path imagingStudySeries[0].media.validity.metrics.domain.valid:

"imagingStudySeries": [
  {
    "media": {
      "validity": {
        "metrics": {
          "domain": {
            "valid": false,
            "score": 9,
            "category": "Non dermatological"
          },
          ...,
        },
        ...,
      },
      ...,
    },
    ...,
  }.
  ...,
]

The diagnosis for the non-dermatological image is completely wrong, it does not reflect what the image represents. If you check the top-level conclusions key, you will see that the preliminary diagnosis predicted for the out-of-domain image does not have any meaning:

"conclusions": [
  {
    "coding": {
      "code": "XH4L78",
      "display": "Nevus",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 19.37
  },
  {
    "coding": {
      "code": "ND56.0",
      "display": "Hematoma",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 8.21
  },
  {
    "coding": {
      "code": "EE80.0",
      "display": "Granuloma annulare",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 7.66
  },
  {
    "coding": {
      "code": "EB60",
      "display": "Lichen sclerosus",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 4.92
  },
  {
    "coding": {
      "code": "XH5AW4",
      "display": "Angioma",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 3.86
  }
]

Additional notes: No additional comments are required for this test.

Test case 2.3: Missing or extraneous field in the request

Scenario: A user might either omit a mandatory JSON key in the request body or include an extra key that isn't allowed by the data schema, either due to a misspelling or because it's entirely unrelated.
Prerequisites:
- User account is already created and credentials are known.
- User downloaded the image(s) labeled "Melanoma" in the setup stage. Then, the user converted the image(s) to Base64 and saved the encoded image(s) in a location close to the working environment.

Test data: The three JSON objects required by this test are given below together with their identifiers:

Body request 1

{
  "subject": { "reference": "fake-patient-id" },
  "media": [
    {
      "contentType": "image/jpeg"
      // We have omitted the mandatory `data` key from JSON
    }
  ]
}

Body request 2

{
  "subject": {"reference": "fake-patient-id"},
  "media": [
    {
      "kontentType": "image/jpeg", // We misspelled the "contentType" key
      "data": <Base64-Encoded-Melanoma-Image>,
    }
  ],
}

Body request 3

{
  "subject": {"reference": "fake-patient-id"},
  "media": [
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Melanoma-Image>,
    }
  ],
  "patientAddress": "1234 Elm Street, Springfield, IL 62704" // We added an extraneous key
}

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Submit three distinct requests to the API, each containing a different JSON from "Test data".
- Record the HTTP status code of the response(s) and capture the response body.

Expected outcome:

The API fails in processing each of the requests and responds with a 422 Unprocessable Content status code in all three cases.

For the request with the JSON "Body request 1", the response body should contain an error message similar to:

{
  "detail": [
    {
      "type": "missing",
      "loc": ["body", "media", 0, "data"],
      "msg": "Field required",
      "input": {
        "contentType": "image/jpeg"
      }
    }
  ]
}

For the request with the JSON "Body request 2", the response body should contain an error message similar to:

{
  "detail": [
    {
      "type": "extra_forbidden",
      "loc": ["body", "media", 0, "kontentType"],
      "msg": "Extra inputs are not permitted",
      "input": "image/jpeg"
    }
  ]
}

For the request with the JSON "Body request 3", the response body should contain an error message similar to:

{
  "detail": [
    {
      "type": "extra_forbidden",
      "loc": ["body", "patientAddress"],
      "msg": "Extra inputs are not permitted",
      "input": "1234 Elm Street, Springfield, IL 62704"
    }
  ]
}

Additional notes: No additional comments are required for this test.

Test case 2.4: Submission of multiple images for a more reliable diagnosis

Scenario: A user wants a more precise diagnosis by submitting two acceptable-to-high quality clinical images of a skin abnormality. Each image captures the same potential lesion from different angles.
Prerequisites:
- User account is already created and credentials are known.
- User has downloaded the images labeled "Hematoma1" and "Hematoma2" during the setup stage, converted them to Base64, and has the encoded images readily available for use.

Test data: The JSON object required for this test as the request body is:

{
  "subject": {"reference": "fake-patient-id"},
  "media": [
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Hematoma1-Image>,
    },
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Hematoma2-Image>,
    },
  ]
}

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Submit a request to the API endpoint with a JSON body containing the encoded images.
- Record the HTTP status code of the response(s) and capture the response body.

Expected outcome:

The API successfully processes the images and returns a 200 OK status code.
The top-level imagingStudySeries key of the JSON is a list containing two JSON objects, one for each image sent. Each of these objects contains the results of the AI analysis, and if you inspect them you can check the image-specific preliminary diagnosis (conclusions), validity metrics and other information that may be useful to you.

The top-level conclusions key of the JSON contains a differential diagnosis resulting from aggregating the results of the differential diagnoses per image. In this case, the patient's most likely skin condition should be hematoma:

"conclusions": [
  {
    "coding": {
      "code": "ND56.0",
      "display": "Hematoma",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 89.51
  },
  {
    "coding": {
      "code": "XH5AW4",
      "display": "Haemangioma",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 6.97
  },
  {
    "coding": {
      "code": "EH6Z",
      "display": "Drug eruption",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 1.1
  },
  {
    "coding": {
      "code": "XH4L78",
      "display": "Nevus",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 0.92
  },
  {
    "coding": {
      "code": "EA83.0Z",
      "display": "Lichen simplex chronicus",
      "system": "https://icd.who.int/en",
      "systemAlias": "ICD-11"
    },
    "probability": 0.41
  }
]

Additional notes: No additional comments are required for this test.

Test case 2.5: Handling corrupted images

Scenario: A user accidentally sends a corrupted image along with a valid image.
Prerequisites:
- User account is already created and credentials are known.
- User has downloaded the images labeled "Hematoma1" and "Hematoma2" during the setup stage, converted them to Base64, and has the encoded images readily available for use.

Test data: The JSON object required for this test as the request body is:

{
  "subject": {"reference": "fake-patient-id"},
  "media": [
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Hematoma1-Image>,
    },
    {
      "contentType": "image/jpeg",
      "data": <Malformed-Hematoma2-Image>,
    },
  ]
}

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Make a copy of the Base64 string of the "Hematoma2" image and insert any number of random characters both at the beginning and at the end of the string (it can be any character from your keyboard). This way we can fake a corrupted image.
- Submit a request with a JSON body containing the image you just corrupted and the well-encoded image of "Hematoma1".
- Record the HTTP status code of the response(s) and capture the response body.
Expected outcome:
- The API identifies the corrupted image and returns a 422 Unprocessable Content status code with an error message indicating that one of the images could not be processed. The error message you receive may look something like this:
```
{
  "detail": [
    {
      "type": "value_error",
      "loc": ["body", "media", 1, "data"],
      "msg": "Value error, Only base64 data is allowed",
      "input": "ude7dd$%%fjn/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAg...",
      "ctx": { "error": {} }
    }
  ]
}
```
  The msg key in the JSON can also contain the following error messages, depending on the characters you have chosen to corrupt the image:
  - {"msg": "Value error, Discontinuous padding not allowed"}
  - {"msg": "Value error, Excess data after padding"}
  In either case, the first characters of the JSON input key should match the ones you added at the beginning of the Base64 string to alter the image.
Additional notes: No additional comments are required for this test.

Test case 2.6: Excessive number of images

Scenario: A user mistakenly attempts to submit six images, exceeding the established limit.
Prerequisites:
- User account is already created and credentials are known.
- User has downloaded the images labeled "Acne", "Alopecia areata", "Generalized pustular psoriasis", "Melanoma", "Hematoma1" and "Hematoma2" during the setup stage, converted them to Base64, and has the encoded images readily available for use.

Test data: The JSON object required for this test as the request body is:

{
  "subject": {"reference": "fake-patient-id"},
  "media": [
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Acne-Image>,
    },
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Alopecia-Areata-Image>,
    },
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Generalized-Pustular-Psoriasis-Image>,
    },
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Melanoma-Image>,
    },
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Hematoma1-Image>,
    },
    {
      "contentType": "image/jpeg",
      "data": <Base64-Encoded-Hematoma2-Image>,
    },
  ]
}

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Submit a request with the JSON body containing the Base64 encoded versions of the "Acne", "Alopecia areata", "Generalized pustular psoriasis", "Melanoma", "Hematoma1" and "Hematoma2" images.
- Record the HTTP status code of the response(s) and capture the response body.
Expected outcome:
- The API responds with a 422 Unprocessable Content status code with a message explaining that the number of images exceeds the allowed limit. The error message should look like this:
```
{
  "detail": [
    {
      "type": "too_long",
      "loc": ["body", "media"],
      "msg": "List should have at most 5 items after validation, not 6",
      "input": [...],
      "ctx": {"field_type": "List", "max_length": 5, "actual_length": 6}
    }
  ]
}
```
  As the value of the input key you should expect a list with a massive amount of content, and that is because it contains all the encoded images you have sent to the API.
Additional notes: No additional comments are required for this test.

3. Severity assessment endpoint tests

Objective: Ensure the severity assessment endpoint is intuitive, user-friendly and meets the user's needs for skin condition follow-up.
Endpoint: /severity-assessment

Test case 3.1: Evaluating the severity of skin conditions of different natures

Scenario: A user is interested in using our AI capabilities for quantifying the intensity, count and extent of visual clinical signs present in different skin conditions, as well as to calculate one or more scoring systems for the conditions.
Prerequisites:
- User account is already created and credentials are known.
- User has downloaded the images labeled "Acne", "Alopecia areata" and "Generalized pustular psoriasis" during the setup stage, converted them to Base64, and has the encoded images readily available for use.

Test data: The following JSON objects are the body requests that you have to send to the API in the corresponding test steps:

For the "Acne" image:

{
    "subject": {"reference": "fake-patient-id"},
    "media": {
        "contentType": "image/jpeg",
        "data": <Base64-Encoded-Acne-Image>,
    },
    "known_condition": {
        "conclusion": {
            "code": "ED80",
            "display": "Acne",
            "system_alias": "ICD-11",
        }
    },
    "body_site": "face",
    "scoring_systems": ["aladin"],
}

For the "Alopecia areata" image:

{
    "subject": {"reference": "fake-patient-id"},
    "media": {
        "contentType": "image/jpeg",
        "data": <Base64-Encoded-Alopecia-Areata-Image>,
    },
    "known_condition": {
        "conclusion": {
            "code": "ED70",
            "display": "Alopecia",
            "system_alias": "ICD-11",
        }
    },
    "body_site": "head_top",
    "scoring_systems": ["asalt"],
}

For the "Generalized pustular psoriasis" image:

{
    "subject": {"reference": "fake-patient-id"},
    "media": {
        "contentType": "image/jpeg",
        "data": <Base64-Encoded-Generalized-Pustular-Psoriasis-Image>,
    },
    "known_condition": {
        "conclusion": {
            "code": "EA90.4",
            "display": "Pustular psoriasis",
            "system_alias": "ICD-11",
        }
    },
    "body_site": "torso",
    "scoring_systems": ["agppga"],
}

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Submit the JSON body request for the "Acne" image. Here we are interested in counting the number of lesions and assessing the severity of acne using the ALADIN scoring system.
- Submit the JSON body request for the "Alopecia areata" image. Here we are interested in measuring the extent covered by the lesion and assessing the severity of the alopecia using the SALT scoring system.
- Submit the JSON body request for the "Generalized pustular psoriasis" image. Here we are interested in quantifying the intensity of the visual signs that characterize the condition and assessing the severity of pustular psoriasis using the AGPPGA scoring system.
- Record the HTTP status code of the response(s) and capture the response body.
Expected outcome:
- The API successfully processes all requests and returns a 200 OK status code for each response.
- The calculated score for each scoring system you have specified in the request can be fetched from the JSON path given by patientEvolution.<Scoring-System-Name>.score.value. Some scoring systems also provide an interpretation of that value that can be extracted from the interpretation key, located at the same level as the value key.
- For skin conditions where lesions can be counted or the surface area covered by the lesion measured, the JSON contained in the response will include one or more images that are the result of processing the original image using the information predicted by the AI. The processed images are Base64 encoded and can be found in the JSON path patientEvolution.<Scoring-System-Name>.media.attachments[0].data.
  - In the case of ALADIN, the returned image includes the bounding boxes that delimit each acne lesion found. The decoded image should look similar to this one:
  - In the case of ASALT, the returned image shows the surface segmentations for both the scalp and the zone with hair loss for a patient suffering from alopecia. The decoded image should look similar to this one:
- The body of the API response includes other valuable information, which you can find in the following JSON paths:
  - patientEvolution.<Scoring-System-Name>.items: the intensity of the visual signs analyzed in the image of generalized pustular psoriasis, or the percentage of scalp and hair loss in the case of alopecia.
  - patientEvolution.<Scoring-System-Name>.media.detections: the pixel coordinates of the bounding boxes enclosing the inflammatory lesions associated with acne.
Additional notes: No additional comments are required for this test.

Test case 3.2: Scoring system requiring a supporting questionnaire

Scenario: A user wants to monitor the severity of a skin condition using an automatic scoring system that requires filling out a questionnaire with information that cannot be predicted by AI.
Prerequisites:
- User account is already created and credentials are known.
- User has downloaded the image labeled "Atopic dermatitis" during the setup stage, converted it to Base64, and has the encoded image readily available for use.

Test data: The JSON object required by this test as the body request is:

{
    "subject": {"reference": "fake-patient-id"},
    "media": {
        "contentType": "image/jpeg",
        "data": <Base64-Encoded-Atopic-Dermatitis-Image>,
    },
    "known_condition": {
        "conclusion": {
            "code": "EA80",
            "display": "Atopic dermatitis",
            "system_alias": "ICD-11",
        }
    },
    "body_site": "hand",
    "scoring_systems": ["ascorad"],
    "questionnaire_response": [
        {
            "questionnaire": "ascorad",
            "item": [
                {"code": "surface", "answer": [{"value": 78}]},
                {"code": "itchiness", "answer": [{"value": 4}]},
                {"code": "sleeplessness", "answer": [{"value": 6}]},
            ],
        },
    ]
}

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Submit to the API the request with the JSON object provided in "Test data".
- Submit the same request again but this time remove the questionnaire_response key and its value from the JSON body.
- Record the HTTP status code of the response(s) and capture the response body.
Expected outcome:
- The API successfully responds with a 200 OK status code for the request that includes the ASCORAD questionnaire responses. You should see the final score of the scoring system and other visual items predicted by the AI in the JSON response body.
- When the questionnaire data is excluded from the request, since it is a mandatory questionnaire, the API fails to process the request and responds with a status code 500 Internal Server Error without providing any details about the error occurred.
Additional notes: No additional comments are required for this test.

Test case 3.3: Inappropriate scoring system for the condition

Scenario: A user wants to know the result of a scoring system for a known condition, but has specified an inappropriate scoring system for that condition, as the scoring system is not intended to quantify the severity of that disease.
Prerequisites:
- User account is already created and credentials are known.
- User has downloaded the image labeled "Acne", converted it to Base64, and has the encoded image readily available for use.

Test data: The JSON object required by this test as the body request is:

{
    "subject": {"reference": "fake-patient-id"},
    "media": {
        "contentType": "image/jpeg",
        "data": <Base64-Encoded-Acne-Image>,
    },
    "known_condition": {
        "conclusion": {
            "code": "ED80",
            "display": "Acne",
            "system_alias": "ICD-11",
        }
    },
    "body_site": "head_top",
    "scoring_systems": ["asalt"],
}

Body site has changed

Note that we also had to change the value of the body_site key to head_top because face is not a valid option in the ASALT scoring system.

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Submit to the API the JSON object available in "Test data".
- Record the HTTP status code of the response(s) and capture the response body.
Expected outcome:
- The API successfully processes the request and returns a response with a 200 OK status code.
- In the annotated image located at the JSON path patientEvolution.asalt.media.attachments[0].data we can notice the AI model of ASALT is doing its best to identify the visual signs of alopecia, failing in the attempt. That is because the inadequate scoring system (and therefore AI model) is being used. If we decode the Base64 image, we can indeed verify this:
Additional notes: No additional comments are required for this test.

Test case 3.4: Scoring system unspecified or misspelled

Scenario: A user sends an empty list of scoring systems or mistypes the name of one or multiple scoring systems.
Prerequisites:
- User account is already created and credentials are known.
- User has downloaded the image labeled "Acne", converted it to Base64, and has the encoded image readily available for use.

Test data: Two JSON objects are required for this test, each properly labeled for use in the corresponding request:

Body request 1

{
    "subject": {"reference": "fake-patient-id"},
    "media": {
        "contentType": "image/jpeg",
        "data": <Base64-Encoded-Acne-Image>,
    },
    "known_condition": {
        "conclusion": {
            "code": "ED80",
            "display": "Acne",
            "system_alias": "ICD-11",
        }
    },
    "body_site": "face",
    "scoring_systems": [], // Note the list is empty
}

Body request 2

{
    "subject": {"reference": "fake-patient-id"},
    "media": {
        "contentType": "image/jpeg",
        "data": <Base64-Encoded-Acne-Image>,
    },
    "known_condition": {
        "conclusion": {
            "code": "ED80",
            "display": "Acne",
            "system_alias": "ICD-11",
        }
    },
    "body_site": "face",
    "scoring_systems": ["heladin"],
}

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Submit to the API the JSON object marked as "Body request 1" in "Test data".
- Submit to the API the JSON object marked as "Body request 2" in "Test data".
- Record the HTTP status code of the response(s) and capture the response body.

Expected outcome:

When sending an empty list, the API fails to process the request and responds with a 500 Internal Server Error, without providing any further details about the issue.

When a misspelled scoring system is sent, the API responds with a 422 Unprocessable Content status code. The error message will include a list of the exact names of all available scoring systems in the device:

{
  "detail": [
    {
      "type": "enum",
      "loc": ["body", "scoring_systems", 0],
      "msg": "Input should be 'agppga', 'aihs4', 'aladin', 'apasi', 'apulsi', 'asalt', 'ascorad', 'auas', 'dlqi', 'gags', 'ihs4', 'nsil', 'pasi', 'pga', 'pure4', 'resvech', 's7pc', 'scorad', 'scovid', 'uas' or 'uct'",
      "input": "heladin",
      "ctx": {
        "expected": "'agppga', 'aihs4', 'aladin', 'apasi', 'apulsi', 'asalt', 'ascorad', 'auas', 'dlqi', 'gags', 'ihs4', 'nsil', 'pasi', 'pga', 'pure4', 'resvech', 's7pc', 'scorad', 'scovid', 'uas' or 'uct'"
      }
    }
  ]
}

Additional notes: No additional comments are required for this test.

Test case 3.5: Sending more than one image

Scenario: A user mistakenly sends more than one image of the same lesion in a single request, assuming the behavior is similar to the /diagnosis-support endpoint.
Prerequisites:
- User account is already created and credentials are known.
- User downloaded the images labeled "Hematoma1" and "Hematoma2". Then, the user converted the images to Base64 and saved the encoded images in a location close to the working environment.

Test data: The JSON object required by this test as the body request is:

{
    "subject": {"reference": "fake-patient-id"},
    "media": [
        {
            "contentType": "image/jpeg",
            "data": <Base64-Encoded-Hematoma1-Image>,
        },
        {
            "contentType": "image/jpeg",
            "data": <Base64-Encoded-Hematoma2-Image>,
        },
    ],
    "known_condition": {
        "conclusion": {
            "code": "ND56.0",
            "display": "Hematoma",
            "system_alias": "ICD-11",
        }
    },
    "body_site": "leg_right",
    "scoring_systems": ["nsil"],
}

HTTP method: POST
Steps:
- If the token obtained in previous tests has expired, log in again to receive a fresh access token.
- Submit the request to the API including the JSON object available in "Test data".
- Record the HTTP status code of the response(s) and capture the response body.

Expected outcome:

The API fails in processing the request and responds with a 422 Unprocessable Content status code.

The response body should contain an error message similar to the one below. This error message should be clear, guiding you to submit only one image per request.

{
  "detail": [
    {
      "type": "model_attributes_type",
      "loc": ["body", "media"],
      "msg": "Input should be a valid dictionary or object to extract fields from",
      "input": [
        {
          "contentType": "image/jpeg",
          "data": "/9j/4AAQSkZJRgA...NFFIY2iiigD/9k="
        },
        {
          "contentType": "image/jpeg",
          "data": "/9j/4AAQSkZJRgA...UY9KcKDTGf/2Q=="
        }
      ]
    }
  ]
}

Additional notes: No additional comments are required for this test.

Record signature meaning

Author: JD-004
Reviewer: JD-003
Approver: JD-005

User expertise​

Equipment prerequisites​

Test scenarios​

Authentication data​

How to authenticate​

Initial setup​

Encoding images to Base64​

Example​

Decode Base64 images​

1. Authentication endpoint tests​

Test case 1.1: Credential validation​

Test case 1.2: Token expiry and refresh​

2. Diagnosis support endpoint tests​

Test case 2.1: The simplest workflow​

Test case 2.2: Beyond the boundaries of the standard operating flow​

Test case 2.3: Missing or extraneous field in the request​

Test case 2.4: Submission of multiple images for a more reliable diagnosis​

Test case 2.5: Handling corrupted images​

Test case 2.6: Excessive number of images​

3. Severity assessment endpoint tests​

Test case 3.1: Evaluating the severity of skin conditions of different natures​

Test case 3.2: Scoring system requiring a supporting questionnaire​

Test case 3.3: Inappropriate scoring system for the condition​

Test case 3.4: Scoring system unspecified or misspelled​

Test case 3.5: Sending more than one image​

Record signature meaning​

User expertise

Equipment prerequisites

Test scenarios

Authentication data

How to authenticate

Initial setup

Encoding images to Base64

Example

Decode Base64 images

1. Authentication endpoint tests

Test case 1.1: Credential validation

Test case 1.2: Token expiry and refresh

2. Diagnosis support endpoint tests

Test case 2.1: The simplest workflow

Test case 2.2: Beyond the boundaries of the standard operating flow

Test case 2.3: Missing or extraneous field in the request

Test case 2.4: Submission of multiple images for a more reliable diagnosis

Test case 2.5: Handling corrupted images

Test case 2.6: Excessive number of images

3. Severity assessment endpoint tests

Test case 3.1: Evaluating the severity of skin conditions of different natures

Test case 3.2: Scoring system requiring a supporting questionnaire

Test case 3.3: Inappropriate scoring system for the condition

Test case 3.4: Scoring system unspecified or misspelled

Test case 3.5: Sending more than one image

Record signature meaning