Research and planning
This page is for internal planning only. It will not be included in the final response to BSI.
What BSI is asking
Where is the specification of the test environment (hardware, software, configuration) used to generate the AI model integration verification test results in master.csv?
Underlying concern (GSPR 17.2 / EN 62304 §5.5, §5.6): BSI wants to verify that integration verification was conducted in a controlled, documented environment. Without knowing the test environment, results are not reproducible.
Root cause analysis
The test environment information is documented across several documents, but it is not consolidated in one easily-findable location:
- R-TF-012-035 Software Test Report (lines 37-46): Describes test levels and environments (Development, Staging/QA, Preproduction, Production)
- R-TF-028-006 AI/ML Release Report (Integration Verification Package section, lines 3000-3149): Specifies runtime configuration in the manifest (PyTorch ≥1.12.0, preprocessing parameters, temperature scaling, TTA settings)
- R-TF-012-033 Software Test Plan: Defines environment configuration baselines (API configuration, build identity, container baseline, runtime configuration SHA-256)
- ai-models-integration-tests.csv header: Each test case specifies the plan as "Medical Device 1.1.0.0 (sha: 5f8549e02f3f362db8930906cf6dfdedf232119a) — AI Models Integration Plan — Preproduction"
The specific environment details that DO exist in our documentation:
| Parameter | Value | Source document |
|---|---|---|
| Environment | Preproduction (AWS) | R-TF-012-035, ai-models-integration-tests.csv |
| Software version | 1.1.0.0 (SHA: 5f8549e02f3f362db8930906cf6dfdedf232119a) | ai-models-integration-tests.csv |
| PyTorch version | ≥1.12.0 | R-TF-028-006 manifest |
| Python version | ≥3.8 | R-TF-028-006 manifest |
| Image preprocessing | Resize [384, 384], ImageNet normalization | R-TF-028-006 manifest |
| Runtime config | TTA enabled, temperature scaling 1.05 | R-TF-028-006 manifest |
| Acceptance criteria (classification) | Probability match ≤ 1e-5 per element | R-TF-028-006 |
| Acceptance criteria (intensity) | Score match ≤ 1e-4 | R-TF-028-006 |
| Acceptance criteria (segmentation) | IoU ≥ 0.9999 | R-TF-028-006 |
| Pass rate | 100% | R-TF-028-006 |
| Evidence storage | s3://legit-health-plus/integration-verification/ | R-TF-012-035, ai-models-integration-tests.csv |
Relevant QMS documents
| Document | Path | Relevance |
|---|---|---|
| Software Test Report | apps/qms/docs/legit-health-plus-version-1-1-0-0/product-verification-and-validation/software/R-TF-012-035-Software-Test-Report/index.mdx | Test environment and levels (lines 37-46) |
| Software Test Plan | apps/qms/docs/legit-health-plus-version-1-1-0-0/product-verification-and-validation/software/R-TF-012-033-Software-Test-Plan/ | Environment configuration baselines |
| AI/ML Release Report | apps/qms/docs/legit-health-plus-version-1-1-0-0/product-verification-and-validation/artificial-intelligence/r-tf-028-006-aiml-release-report.mdx | Integration Verification Package specification (lines 3000-3149) |
| AI Models Integration Tests | apps/qms/docs/legit-health-plus-version-1-1-0-0/product-verification-and-validation/software/ai-models-integration-tests.csv | 76 test cases, plan/run IDs, version/SHA info |
Gap analysis
- Already had: Test environment information distributed across R-TF-012-033, R-TF-012-035, R-TF-028-006, and the CSV header.
- BSI couldn't find: A consolidated, easily-locatable test environment specification for the AI model integration tests specifically.
- Needs updating: Add a clear "Test Environment" subsection to the Software Test Report (R-TF-012-035) under § AI Models Integration Verification, consolidating all environment details in one place.
Response strategy
Regulatory mapping for this response:
| Requirement | How our response addresses it |
|---|---|
| GSPR 17.2 | Software V&V performed in a controlled, documented environment per state-of-the-art lifecycle practices |
| Annex II 6.1(b) | Design and manufacturing information (including V&V environment) is documented |
| Annex II 6.2(f) | V&V results are traceable to a specified environment |
| EN 62304 §5.5.2 | Integration test plan defines the test environment (R-TF-012-033, § Test Environment Controls) |
| EN 62304 §5.6.2 | System test plan defines the test environment (R-TF-012-033, § Environment Controls) |
| EN 82304-1 §6.2 | Validation evidence includes environment specification |
Action required:
-
Point BSI to the existing documentation: The test environment is specified across R-TF-012-033, R-TF-012-035, and R-TF-028-006. Provide specific section references per the table above.
-
Add a consolidated "Test Environment" subsection to the Software Test Report (R-TF-012-035) under "Detailed Results — AI Models Integration Verification" that explicitly states: environment name, software version, commit SHA, runtime dependencies (PyTorch version, Python version), preprocessing configuration, and acceptance criteria.
Response tone: "The test environment specification for AI model integration verification is documented in the following locations, per EN 62304 §5.5.2 and §5.6.2: R-TF-028-006, § Integration Verification Package (runtime configuration, preprocessing parameters, and acceptance criteria); R-TF-012-033, § Test Environment Controls (environment configuration baselines including build identity, container baseline, and runtime configuration SHA-256). For clarity, and to satisfy Annex II 6.2(f) traceability, we have added a consolidated test environment specification to R-TF-012-035, § AI Models Integration Verification. See updated documentation [red-lined]."
Action items:
| # | Action | Owner | Document affected | Priority |
|---|---|---|---|---|
| 5 | Add consolidated test environment section to Test Report | Gerardo | R-TF-012-035 | High |