R-TF-012-033 Software Tests Plan

Test identification

Testing phases

This test plan defines all the tests to verify all the requirements of the medical device in the following successive testing phases:

• Unit tests and Integration tests
• Verification tests

Requirements are defined in the software requirements folder.

Test progression

The tests progression depends on the testing phase:

• Unit and Integration tests: There is no order in the execution of the Unit and Integration tests. All the Unit and Integration tests are run simultaneously.
• Verification tests: Verification tests can only be done if the Unit and Integration tests are successful. Then, Verification tests progression is defined according to the following rationale:
  • Non-Regression Tests (NRT) are done at first (if they are planned in the test plan),
  • Tests for new features and bug fixes are done afterwards,
  • Tests by environment are done at the end

Data recording, post-processing and analysis

• The crashes and functional errors occurring during test executions are collected into sentry.io cloud service.
• The Test data sets are stored in the md-legit-health-plus git repository.

Test identification and content

Each test case must include a set of defined fields to ensure proper traceability, reproducibility, and consistency across all testing phases. Two categories of tests are defined: Unit and Integration Tests and Verification Tests. The following subsections describe the minimum required fields for each.

Unit and Integration Tests

Each Unit or Integration Test must contain at least the following information:

• Identifier: Automatically generated unique identifier by the CI pipeline or test framework.
• Test Suite / Module: Logical grouping of related test cases (e.g., software component, package, or class).
• Test Name: Descriptive title of the test case.
• Objective: Brief statement of what the test verifies at code or interface level.
• Expected Result: The expected output or condition for the test to pass.
• Status: Result of the test execution (e.g., Passed, Failed, Skipped).
• Execution Environment: CI/CD environment or local test configuration.
• Test Date / Software Version: Version or commit hash of the tested build.
• Automated Evidence: Link or reference to the automated test report (e.g., HTML, XML, or JSON report generated by the pipeline).

Verification Tests

Each Verification Test must contain the following information:

• Identifier: Unique test case reference
• Title: Concise name describing the test purpose.
• Objective: Description of the functionality or requirement to be verified.
• Traceability: Reference(s) to the corresponding Software Requirement(s) (SRS).
• Fix Version(s): Software version(s) in which the test has been executed.
• Label(s): Special tags such as:
  • NRT – Non-Regression Test
  • Commissioning_Test – Executed only in commissioning phase
• Preconditions / Assumptions: Environmental or logical conditions required before executing the test.
• Test Data and Procedure: Description of the input data and step-by-step execution method.
• Expected Result: Description of the expected behavior or output.
• Data Recording and Analysis: Method used to record and analyze results.
• Postconditions: Conditions that must be validated after the test execution.
• Result / Status: Execution result (Passed, Failed, Blocked, Not Run).

Defect identification and content

During the test campaign, if the expected result is incorrect for a test case, then a defect is raised as an issue in the repository. This issue is created by either a developer or a tester (depending on the test phase) with following information when available:

• Issue type: bug
• Summary: a short title for the defect
• Affected version
• Description, including:
  • The steps to reproduce are written down into the issue, and reproducibility rate if any
  • The actual and expected results are specified
  • The technical environment
    • Version where the defect is found
    • Any other details from the test context
• Link to the Test Case, if detected during the test execution

This defect will appear in the "Software Test report" of the tested version, in the defects list.

Test environment

This chapter present all necessary set-up, tools and the personnel in order to execute the tests properly.

Unit test and integration test

Unit and integration tests can be executed in two different environments:

• Continuous Integration (CI) environment: Unit and integration tests are automatically executed in the CI pipeline for each code change (pull request) and for each release candidate. The CI environment is defined in the GitHub Actions workflow files.
• Local development environment: Developers can execute unit and integration tests locally on their development machines. The local environment must match the CI environment as closely as possible to ensure consistency in test results.

Verification test

The hardware requirements for the server used to execute verification tests for our medical devices are as follows:

• NVIDIA GPU with CUDA and a total of 96 GB of GPU memory
• The number of GPUs does not matter
• Storage: 250 GB minimum
• RAM: 64 GB
• CPU: 64 cores (2 threads per core).

This hardware configuration is necessary to ensure that the verification tests can be executed efficiently and effectively, given the computational demands of the AI models used in our medical device.

For the client side, a standard computer with internet access is sufficient to interact with the API endpoints.

Softeware test tools

Unit and integration test tools

to perform unit and integration tests, the following tools are used:

• Github Actions: for automating the execution of unit and integration tests in the CI pipeline.
• Pytest: as the testing framework for writing and executing unit and integration tests.

Verification test tools

To perform verification tests, the following tools are used:

• Pytest: as the testing framework for writing and executing verification tests.

Test data and documentation

Unit tests and integration tests

The unit and integration tests are located in the md-legit-health-plus git repository. This project contains different apps that implement the various components of the medical device software. Each app has its own directory structure, which includes a tests folder where the unit and integration tests are stored:

• control_plane
• experts
• report_builder
• report_exporter
• ui/api_gateway

Each tests folder contains test files organized by functionality, along with any necessary test data or fixtures. The Readme files in each app provide additional information on how to run the tests and any specific dependencies or configurations required.

Verification tests

The verification tests are defined in the folder verification in the md-legit-health-plus-verification git repository. Pytest is used for test case management.

These verification tests are executed against a preproduction environment that mirrors the production setup. This environment includes the same hardware specifications, software configurations, and network settings to ensure that the tests accurately reflect real-world usage scenarios. Since from the client side it is only necessary to have internet access to interact with the API endpoints, no special hardware is required on the client side for verification testing.

Non-Regression Tests

Some tests are classified as Non-Regression Tests (NRT) to ensure that previously validated functionalities continue to work as expected after changes or updates to the software. These NRTs are executed first during the verification testing phase.

Installation, setup and maintenance

To perform verification tests, the test environment must be set up according to the following steps:

• Provision the server with the specified hardware requirements.
• Install the medical device software on the server, following the deployment procedures outlined in the project documentation.
• Configure the network settings to allow access to the API endpoints from the client side.
• Run the verification tests using Pytest.

Personnel

The unit and integration tests are primarily executed by the development team as part of their development workflow. Each developer is responsible for writing and running tests for the code they develop.

The verification tests are designed by JD-007 and are executed either by him, either by other member of the developer team.

Planned tests

Unit and integration tests

Unit and integration tests are defined and executed as part of the CI pipeline through GitHub Actions. If any of these tests fail, the code changes are rejected, and the issues must be resolved before proceeding.

Unit and integration tests coverage

The unit and integration tests coverage report is generated using pytest-cov, a plugin for Pytest that measures code coverage. The coverage report is stored in the CI pipeline artifacts, can be accessed through the GitHub Actions interface. The report provides insights into which parts of the codebase are covered by tests and highlights areas that may require additional testing.

Coverage documents are stored in the Design History File of the related version.

Verification tests

Verification tests are defined in the verification folder of the md-legit-health-plus-verification git repository. Pytest is used for test case management.

Commisioning phase

For the comissioning phase, a selection of various tests has been carried out to ensure the proper functioning of the medical device in its intended environment. These tests include:

• Non regression tests
• End to end tests are also conducted to validate the integration and overall functionality of the system.
• Addiontaly, we perform specific tests focused on the commissioning aspects of the device to ensure it meets all necessary requirements and intented use cases.

These last two types of tests are labeled "Commissioning_Test" to ease their identification. Additionally, these tests are not necessarily linked to Software Requirements, as they may cover broader aspects of the system's operation and integration.

For this phase, a beta testing environment is used to replicate the production set-up as closely as possible.

List of planned tests

Requirements traceabilit

The traceability between the Software Requirements and the planned tests is maintained through the unique identifiers assigned to each requirement and test case. Each verification test case includes a "Traceability" field that references the corresponding Software Requirement(s) (SRS) it is designed to verify.