R-TF-028 AI/ML Design Checks
Table of contents
Purpose
This checklist is used to verify that the Design Phase of the AI/ML development lifecycle has been completed in accordance with procedure GP-028 AI Development. It ensures that the AI/ML Description, Development Plan, Data Collection Instructions, Data Annotation Instructions, and initial Risk Matrix are complete, coherent, and provide a sufficient basis for proceeding to the Development Phase.
Instructions
The verifier must assess each item in the checklist. For each item, select "Yes," "No," or "N/A" and provide comments where necessary, especially for any "No" answers. All "No" items must be resolved before the Design Phase can be considered complete and approved.
Checklist
R-TF-028-001 AI/ML Description
| Check Item | Yes/No/NA | Comments |
|---|---|---|
| 1. Is the purpose of the algorithm package clearly defined? | ||
| 2. Is the ICD Category Distribution algorithm's function (ViT model, probability output) described? | ||
| 3. Is the Binary Indicators' derivation logic (summation via matrix) clearly described? | ||
| 4. Are the performance endpoints and success criteria for Top-k Accuracy (≥55%, ≥70%, ≥80%) explicitly stated? | ||
| 5. Are the performance endpoints and success criteria for Binary Indicator AUC (≥0.80) explicitly stated? | ||
| 6. Are the overall data specifications (scale, diversity, expert annotation) defined? | ||
| 7. Are cybersecurity, transparency, and integration aspects sufficiently described? |
R-TF-028-002 AI/ML Development Plan
| Check Item | Yes/No/NA | Comments |
|---|---|---|
| 1. Is the project team and their responsibilities clearly defined? | ||
| 2. Is the development environment (hardware, software, tools) specified? | ||
| 3. Does the Data Management Plan cover data collection, curation, partitioning, and test set sequestration? | ||
| 4. Does the Training & Evaluation Plan cover model architecture, training methodology, and post-processing (e.g., calibration, TTA)? | ||
| 5. Does the Release Plan specify the deliverables for the software integration team? | ||
| 6. Does the plan include a comprehensive AI/ML Risk Management Plan? |
Data Collection Instructions (R-TF-028-003, R-TF-028-004)
| Check Item | Yes/No/NA | Comments |
|---|---|---|
1. Are there clear and distinct instructions for retrospective data collection from public sources (R-TF-028-003)? | ||
2. Are there clear and distinct instructions for prospective data collection from the clinical study (R-TF-028-004)? | ||
| 3. For both, are the target population, inclusion/exclusion criteria, and ethical considerations (GDPR, consent) defined? | ||
| 4. Are the technical protocols for data retrieval, de-identification, and transfer clearly specified? |
Data Annotation Instructions (R-TF-028-004, R-TF-028-005)
| Check Item | Yes/No/NA | Comments |
|---|---|---|
1. Is the protocol for creating the Binary Indicator Mapping Matrix clear and unambiguous (R-TF-028-004)? | ||
2. Are there clear instructions for annotating Visual Signs (R-TF-028-005), covering intensity, count (bounding boxes), and extent (polygons)? | ||
| 3. Are the qualifications for the medical expert annotators clearly specified? | ||
| 4. Is the Quality Control process, including the use of multiple annotators and a consensus mechanism, well-defined for each task? |
*R-TF-028-011 AI/ML Risk Matrix
| Check Item | Yes/No/NA | Comments |
|---|---|---|
| 1. Have initial AI/ML risks related to data management (e.g., data bias, inaccurate labels) been identified? | ||
| 2. Have initial AI/ML risks related to model training and evaluation (e.g., overfitting) been identified? | ||
| 3. Has an initial assessment of severity and likelihood been performed for the identified risks? |
Conclusion
State the overall outcome of the design check.
☐ Design Phase Approved: All checks have been successfully passed. The project is cleared to proceed to the Development Phase.
☐ Design Phase Not Approved: One or more checks have failed. The responsible team members must address the comments and resubmit the design documentation for verification.
Overall Comments: Provide any summary comments or required actions here.
Verification
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:
- Author: Team members involved
- Reviewer: JD-003, JD-004
- Approver: JD-001