Annex II — Examples of clinical evaluation / performance evaluation strategies

Source: MDCG 2020-1 (March 2020), pages 18-21

Annex II — Examples of CLINICAL EVALUATION (MDR) / PERFORMANCE EVALUATION (IVDR) strategies

The high-level examples provided here are for guidance purposes only and aim to provide general indications on how to develop a CLINICAL EVALUATION / PERFORMANCE EVALUATION strategy. The strategy presented in each example is not a confirmation of the pathway for a CLINICAL EVALUATION / PERFORMANCE EVALUATION of the device, as other factors need to be considered.

Moreover, the proposed pathway reflects the specific intended purpose, or the healthcare context or situation, in which the device is used as described in the example itself. Any change to the intended purpose or the healthcare context / situation in which that same device is used might result in a different approach.

Data source	Examples
Peer-reviewed, relevant scientific literature	- Existing data from studies conducted with the subject device or equivalent device
CLINICAL INVESTIGATION / CLINICAL PERFORMANCE STUDIES	- Prospective or retrospective studies - Existing manufacturer data - Data from equivalent devices - Data from curated databases/registries/reference databases - Data from outside the EU with justification on applicability
Published experience gained by routine diagnostic testing	- REAL-WORLD PERFORMANCE DATA - Data obtained from PMPF/ PMCF

a) MDSW intended to analyse sleep quality data

An independent MDSW intended to take into account accelerometer and microphone data to determine quality of sleep and to estimate the expected success rate of CPAP (continuous positive airway pressure) treatment for sleep apnoea.

The Manufacturer claims that the MDSW

• determines the quality of sleep that impacts the general well-being.
• monitors quality of sleep in patients with sleep disorders such as sleep apnoea (using phone sensors/wearable devices)
• estimates the expected success rate of CPAP therapy.

Valid Clinical Association

To establish VALID CLINICAL ASSOCIATION, review literature.

• Objective quality of sleep is measured by sleep duration, efficiency and fragmentation. It is further well-established that quality of sleep impacts general well-being such as concentration, risk-factors for cardiovascular disease, mood, cognitive abilities, etc.

• It is not well-established that the success of CPAP therapy can be predicted by monitoring the quality of sleep.
• Address the association of accelerometer and microphone data to established quality of sleep parameters (e.g. sleep duration, efficiency and fragmentation).

The VALID CLINICAL ASSOCIATION has been not established without gaps for prediction of success of CPAP therapy, which requires generation of missing clinical data.

Technical Performance

• Confirm with verification and validation tests that the app can reliably and reproducibly calculate sleep quality scoring.
• Confirm compatibility between the MDSW and the device equipped with the sensors to ensure data can be utilised in the intended way.

Clinical Performance

• In addition to the USABILITY assessment, the manufacturer would perform a retrospective study on previously obtained data to confirm that success of CPAP therapy can be predicted based on the quality of sleep.

b) MDSW intended for image segmentation

An independent MDSW intended to allow automatic detection of organs and anatomical structures (such as the aorta) in CT scans with the accuracy of a radiologist.

The Manufacturer claims that the MDSW:

• detects abdominal aortic aneurisms on abdominal CT scans,
• detects compression fractures on vertebrae,
• detects liver cysts.

Valid Clinical Association

To establish VALID CLINICAL ASSOCIATION, review literature.

• The normal shape and size of anatomy is well established.
• Segmentation techniques on cross-sectional images correlates well with the actual size and shape.

The VALID CLINICAL ASSOCIATION has been established without gaps identified.

Technical Performance

• Confirm with verification and validation tests the basic technical performance such as display, modification, window levelling of images, measurements including confirmation of accuracy, sensitivity and reliability of the MDSW as per the expected performance.

Clinical Performance

• USABILITY assessment including the intended user groups in conjunction with the VALID CLINICAL ASSOCIATION and validation of TECHNICAL PERFORMANCE results has been determined as sufficient to demonstrate conformity with relevant GSPRs.
• In cases where data is available, a retrospective analysis can be performed. In cases where data does not represent the variability of input parameters, for the CLINICAL PERFORMANCE of the segmentation algorithm, the missing data could be generated in a prospective CLINICAL INVESTIGATION.

c) MDSW intended to detect inflammatory bowel diseases (IBD)

Self-testing independent MDSW intended for the semi-quantitative detection of calprotectin from a faecal sample. Reagents are added to the sample resulting in a colour change. The sample is then photographed on a smartphone, and the image is evaluated by an MDSW application (app) running on the phone. The MDSW app detects the colour change in the sample and interprets the concentration of calprotectin. The test is intended as an aid in monitoring and staging of patients with inflammatory bowel disease (IBD).

Manufacturer's claims that the MDSW app

• aids in monitoring and staging the disease level of patients with inflammatory bowel diseases (IBD).
• aids in differentiation between IBD and functional bowel disorders.
• helps patients avoid unnecessary clinical visits.

Scientific Validity

To establish SCIENTIFIC VALIDITY, review literature.

• The SCIENTIFIC VALIDITY could address how the calprotectin level corresponds to the IBD level and stages. Furthermore, it should address, whether calprotectin levels are suitable to differentiate between IBD and functional bowel disorders.
• It is well-established that calprotectin concentration in faecal matter can be reliably measured in test strips by change of colour.
• The colour intensity is directly representative of the concentration of calprotectin.

Analytical Performance

• Confirm the MDSW app can detect reliably and accurately the colour of the test strip compared to human observation, taking into account environmental factors.

Clinical Performance

• The manufacturer should assess the initial performance and feasibility by creating CLINICAL PERFORMANCE metrics, taking into account sensitivity, specificity and confidence intervals.
• Any claims regarding CLINICAL BENEFIT should be supported by sufficient clinical performance data.
• USABILITY should be confirmed by the manufacturer.

d) Active devices containing MDSW to enable their intended purpose

Active devices, such as diagnostic or therapeutic devices, that include MDSW which drives the device in a way that, without the software it would not be able to fulfil its intended purpose. This software does not perform a medical purpose on its own.

The CLINICAL EVALUATION of the MDSW should not be performed independently but should be performed together with the driven device.

e) MDSW which provides an additional user-interface to control an insulin pump

A MDSW intended to virtualise controls of an insulin pump additionally on a smartphone app by connecting to it.

As the software is driving the insulin pump, it is not performing a medical purpose on its own, nor is it creating information on its own for medical purposes.

The CLINICAL EVALUATION of the MDSW app should not be performed independently but should be performed together with the driven insulin pump.

f) MDSW intended to analyse exhaled CO2 in a life-sustaining device in order to control ventilator settings

The MDSW uses physiological data of the patient (e.g. exhaled CO2, blood oxygen saturation) to control a ventilation device (e.g. frequency, volume and pressure).

The MDSW allows the device to maintain the pre-set value at a desired target (defined by the clinician) without periodic user adjustments needed. This MDSW is part of a closed-loop system.

The CLINICAL EVALUATION should not be limited to the MDSW and should include pre-clinical and clinical investigations, encompassing the entire closed-loop system.