R-TF-015-002 Preclinical and clinical evaluation record_2023_001

Scope

This literature review compiles and evaluates state-of-the-art publications on computer vision in medicine. The overall objective of this strategy is to identify, select and collect the relevant literature to determine whether the device can achieve its intended purpose.

Medical device information

Manufacturer contact details

	Manufacturer data
Legal manufacturer name	AI Labs Group S.L.
Address	Street Gran Vía 1, BAT Tower, 48001, Bilbao, Bizkaia (Spain)
SRN	ES-MF-000025345
Person responsible for regulatory compliance	Alfonso Medela, Saray Ugidos
E-mail	office@legit.health
Phone	+34 638127476
Trademark	Legit.Health

Medical device characterization

	Information
Device name	Legit.Health Plus (hereinafter, the device)
Model and type	NA
Version	1.1.0.0
Basic UDI-DI	8437025550LegitCADx6X
Certificate number (if available)	MDR 792790
EMDN code(s)	Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software)
GMDN code	65975
Class	Class IIb
Classification rule	Rule 11
Novel product (True/False)	FALSE
Novel related clinical procedure (True/False)	FALSE
SRN	ES-MF-000025345

Intended purpose

Intended use

The device is a computational software-only medical device intended to support health care providers in the assessment of skin structures, enhancing efficiency and accuracy of care delivery, by providing:

• quantification of intensity, count, extent of visible clinical signs
• interpretative distribution representation of possible International Classification of Diseases (ICD) categories.

Quantification of intensity, count and extent of visible clinical signs

The device provides quantifiable data on the intensity, count and extent of clinical signs such as erythema, desquamation, and induration, among others; including, but not limited to:

• erythema,
• desquamation,
• induration,
• crusting,
• xerosis (dryness),
• swelling (oedema),
• oozing,
• excoriation,
• lichenification,
• exudation,
• wound depth,
• wound border,
• undermining,
• hair loss,
• necrotic tissue,
• granulation tissue,
• epithelialization,
• nodule,
• papule
• pustule,
• cyst,
• comedone,
• abscess,
• draining tunnel,
• inflammatory lesion,
• exposed wound, bone and/or adjacent tissues,
• slough or biofilm,
• maceration,
• external material over the lesion,
• hypopigmentation or depigmentation,
• hyperpigmentation,
• scar,
• ictericia

Image-based recognition of visible ICD categories

The device is intended to provide an interpretative distribution representation of possible International Classification of Diseases (ICD) categories that might be represented in the pixels content of the image.

Device description

The device is a computational software-only medical device leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures. Its principal function is to provide a wide range of clinical data from the analyzed images to assist healthcare practitioners in their clinical evaluations and allow healthcare provider organisations to gather data and improve their workflows.

The generated data is intended to aid healthcare practitioners and organizations in their clinical decision-making process, thus enhancing the efficiency and accuracy of care delivery.

The device should never be used to confirm a clinical diagnosis. On the contrary, its result is one element of the overall clinical assessment. Indeed, the device is designed to be used when a healthcare practitioner chooses to obtain additional information to consider a decision.

Intended medical indication

The device is indicated for use on images of visible skin structure abnormalities to support the assessment of all diseases of the skin incorporating conditions affecting the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral and genital), the dermis, the cutaneous vasculature and the subcutaneous tissue (subcutis).

Intended patient population

The device is intended for use on images of skin from patients presenting visible skin structure abnormalities, across all age groups, skin types, and demographics.

Intended user

The medical device is intended for use by healthcare providers to aid in the assessment of skin structures.

User qualification and competencies

In this section we specificy the specific qualifications and competencies needed for users of the device, to properly use the device, provided that they already belong to their professional category. In other words, when describing the qualifications of HCPs, it is assumed that healthcare professionals (HCPs) already have the qualifications and competencies native to their profession.

Healthcare professionals

No official qualifications are needes, but it is advisable if HCPs have some competencies:

• Knowledge on how to take images with smartphones.

IT professionals

IT professionals are responsible for the integration of the medical device into the healthcare organisation's system.

No specific official qualifications are needed, but it is advisable that IT professionals using the device have the following competencies:

• Basic knowledge of FHIR
• Understanding of the output of the device.

Use environment

The device is intended to be used in the setting of healthcare organisations and their IT departments, which commonly are situated inside hospitals or other clinical facilities.

The device is intended to be integrated into the healthcare organisation's system by IT professionals.

Operating principle

The device is computational medical tool leveraging computer vision algorithms to process images of the epidermis, the dermis and its appendages, among other skin structures.

Body structures

The device is intended to use on the epidermis, its appendages (hair, hair follicle, sebaceous glands, apocrine sweat gland apparatus, eccrine sweat gland apparatus and nails) and associated mucous membranes (conjunctival, oral and genital), the dermis, the cutaneous vasculature and the subcutaneous tissue (subcutis).

In fact, the device is intended to use on visible skin structures. As such, it can only quantify clinical signs that are visible, and distribute the probabilities across ICD categories that are visible.

Variants and models

The device does not have any variants.

Expected lifetime

Its expected lifetime is considered unlimited because the device will be updated with each improvement opportunity extracted from the information and analysis of the data provided by the continuous and systematic post-market data follow-up.

List of any accessories

The device does not have any accessories.

The device can be used from any device with an internet connection.

Explanation of any novel features

The device is the result of an incremental improvement of an existing technology. It has been developed to improve the current state of the art to process photographs of skin structure and then processes them with artificial intelligence algorithms.

We detail the novel features of the devide within the Legit.Health Plus Description and specifications 2023_001 document.

Literature search methodology

The literature search methodology establishes the strategy used to search and compile scientific articles, as well as the criteria used to include/exclude and evaluate the doctuments compiled.

Review team undertaking the literature search

The literature search protocol has been developed and executed by professionals with expertise in information retrieval and understanding of the scope of the clinical evaluation set out by the manufacturer, who have been compensated for the time taken to perform the review.

The involvement of the literature review team will help to optimize literature retrieval to identify all relevant published literature. The review team has prepared the protocol to describe an objective, non-biased systematic search and review methods, using information found in the IFU and risk management process to develop the clinical research questions.

Period covered by the search

The present literature search complements the last Clinical Evaluation literature search (2011 to 2020) by analyzing the work published in a 3-year period from 2021 to 2023, and further extends the above search to include the condition of facial palsy.

Goals of the literature search

This literature search was conducted to include the relevant scientific articles that are related to:

• Clinical trials that probe the safety and usefulness of Computer-Aided Systems for obtaining interpretative distribution representation of possible International Classification of Diseases (ICD) categories that might be represented in the pixels content of the image and quantifiable data on the intensity, count and extent of clinical signs such as erythema, desquamation, and induration, among others.
• State-of-the-art algorithms and methods in the fields of visible conditions, specially dermatoses and facial palsy, to obtain an interpretative distribution representation of possible International Classification of Diseases (ICD) categories that might be represented in the pixels content of the image, and quantifiable data on the intensity, count and extent of clinical signs such as erythema, desquamation, and induration, among others.
• Datasets that are useful for the development of technology to obtain an interpretative distribution representation of possible International Classification of Diseases (ICD) categories that might be represented in the pixels content of the image, and quantifiable data on the intensity, count and extent of clinical signs such as erythema, desquamation, and induration, among others, in skin structures.
• Smartphone applications that implement or evaluate skin structures to obtain an interpretative distribution representation of possible International Classification of Diseases (ICD) categories that might be represented in the pixels content of the image, and quantifiable data on the intensity, count and extent of clinical signs such as erythema, desquamation, and induration, among others.
• Reviews and surveys of the available literature related to obtain an interpretative distribution representation of possible International Classification of Diseases (ICD) categories that might be represented in the pixels content of the image, and quantifiable data on the intensity, count and extent of clinical signs such as erythema, desquamation, and induration, among others, in skin structures.

Data sources

For the scientific article compilation we have used different sources of information:

Medical Literature Databases

• PubMed (MEDLINE): comprises more than 36 million citations for biomedical literature from MEDLINE, life science journals, and online books.
• Cochrane Library: collection of databases that contain high-quality, independent evidence to inform healthcare decision-making. The Cochrane Library is owned by Cochrane and published by Wiley.

Google Scholar

Google Scholar provides a simple way to broadly search for scholarly literature across a wide variety of disciplines and sources: articles, theses, books, abstracts and court opinions. Different from the Medical Literature Databases, Google Scholar also includes medical articles that have been published in non-medical journals and conferences and that might not have been indexed by the previous Medical Literature Databases.

Article Citations

Scientific articles are also a meaningful source of information when describing their related work. Relevant articles that are cited within a given published manuscript are also considered for their evaluation.

The PICO format

The Literature Search methodology is based on the PICO format.

This format has been designed for high-quality clinical research evidence by defining a set of relevant keywords in four different categories (Population, Intervention, Comparison, and Outcome), that are then used to construct the search queries. Since our research is not limited to clinical research, by also to articles related to state-of-the-art Deep Learning algorithms, paper reviews, and the deployment of Deep Learning solutions, we have modified these categories to also include keywords related to non-clinical terms.

	Description
Population Patient Problem (P)	Diseases that the software is able to identify (skin cancer, melanoma, chronic skin conditions).
Intervention (I)	Intervention, prognostic factor or exposure.
Comparator (C)	Known software, algorithm, or method with the same characteristics as the device.
Outcome (O)	Diagnosis, identification, analysis, or assessment of certain skin pathologies.

Screening, selection, and organizing literature

The articles retrieved from each data source go through an initial set of filters to ensure their quality and adequacy to the goals of this literature review:

Metadata check

Duplicate articles, non-English articles, and articles published before 2021 are not considered for the Data Appraisal.

Scope analysis

We read the abstract, motivation, and conclusions of each article to discern if they match the scope of topics pursued in the Literature Review. Articles that are our of the scope of this literature review are filtered out. After this initial look, we save a brief summary of the content and contribution of the articles.

Publication source quality analysis

For each article, we analyze the impact factor of the journal, conference, or book where they have been published. The impact factor is a metric that reflects the ratio between the number of citations received in a year and the number of citable publications that were published in the same source during the two preceding years. In our case, we retrieve the Impact Factor from Academic Accelerator that has been calculated for the year 2021. Finally, we filter out the articles that have an Impact Factor under 3.5, which stands for a low-quality publication source. Since the Impact Factor measures the quality of the publication source and not the quality of the publication itself, we also included for the Data Appraisal the papers that we considered relevant during the Scope Analysis step.

The articles that have passed the screening quality evaluation are then forwarded to their appraisal.

Appraisal of selected literature

The Data Appraisal evaluates the suitability of the selected articles for establishing the safety and performance of the device. To this end, we take into consideration three parameters: the methodological quality and scientific validity of the information (Q), the relevance to the device and its intended use (R), and the contribution to the overall evaluation (C). Articles are finally accepted or rejected after applying the following weight-based system. In this process, we also compile some notes related to the content and contribution of the articles.

Data characteristics	Methodological quality Q	Relevance, R	Contribution, C	Weighted value, W (W=Q+R+C)	Appraisal data	Notes
Very relevant information in relation to the product and its intended use	Up to 40	Up to 30	Up to 30	W ≥ 70	Accepted	Pivotal data
Relevant information in relation to the product and its intended use	Up to 40	Up to 30	Up to 30	30 < W < 70	Accepted	Other data
Little relevant information in relation to the product and its intended use	Up to 40	Up to 30	Up to 30	W ≤ 30	Rejected	No contribution, rejected

Methodological quality and scientific validity of the information (Q):

• The scientific article properly describes the relevant related work.
• The scientific article compares its results and conclusions with the relevant related work.
• Clinical trials use data extracted from RCTs (Randomized Clinical Trials), meta-analysis and clinical trials, from publications based on scientific evidence or from previous experience on the market.
• The study design is adequate in terms of type, sample size, relevance of endpoints, randomization of patients, inclusion and exclusion criteria, stratification of patients, prognostic factors, follow-up, recording and reporting of serious adverse events, reliability of the methods used for quantifying outcomes, or procedures for retrieving complete information.
• Data consists of an adequate number of observations that can confirm or refute the claims of the device under evaluation.
• The methods for data processing and statistics are suitable.
• It complies with Good Clinical Practice (GCP) according to EN ISO 14155 or equivalent standards.
• The information is disclosed adequately, including a summary, introduction, methods, results, discussion and conclusions.

Relevance to the device and its intended use (R):

• It is representative of the device under evaluation, including data concerning the medical conditions that are managed with the device, other devices and medical alternatives, or equivalent devices.
• The aspects covered are related to clinical safety and performance, claims, hazards and hazardous situations, management of risks, establishment of current knowledge and state of the art, determination and justification of the benefit/risk ratio and the acceptability of undesirable side-effects, or the determination of equivalence.
• It is representative to the intended purpose or claims of the device under evaluation.
• It is representative to the user group, patient population, medical indication, age group, gender, type and severity of the medical condition, or range of time.
• It develops algorithms or techniques to improve the performance of skin-related image analysis.

Contribution to the overall evaluation (C):

• Device and/or related techniques intended use. The contribution of the analyzed data in relation to the device or technique features and in relation to the intended use must be strong and clear.
• Device and/or related techniques risk analysis. The contribution of the analyzed data in relation to the device or techniques of risk analysis must be strong and clear, based on keywords definition.

Literature Review results

Screening

For each one of the PICO components we have defined the set of keywords as:

Category	Keywords
Population, Patient, Problem (P)	dermatosis, skin cancer, chronic skin conditions, inflammatory skin diseases, malignant skin lesions, pigmented skin lesions, melanoma, basal cell carcinoma, squamous cell carcinoma, atypical nevus, acne, psoriasis, urticaria, atopic dermatitis, onychomycosis, melasma, solar lentigo, dermatofibroma, skin diseases, skin lesions
Intervention (I)	clinical image, digital imaging, web application, smartphone, dermatoscopy, camera, CAD, dermatoscope
Comparator (C)	artificial intelligence, machine learning, deep learning, computer vision, deep neural networks, convolutional neural networks, metaoptima, automated
Outcome (O)	diagnosis, diagnosis support, followup, segmentation, detection, estimation, classification, assessment, severity assessment, improving

In the following, we describe how we use these keywords to form queries for each source of information, the retrieved articles, and the outcome of the Data Screening.

PubMed

Following the PICO format and using the PubMed advanced search tool, we have defined a global query to retrieve the articles whose title or abstract relate to the query:

(("dermatosis") OR ("skin cancer") OR ("chronic skin conditions") OR ("inflammatory skin diseases") OR ("malignant skin lesions") OR ("pigmented skin lesions") OR ("melanoma") OR ("basal cell carcinoma") OR ("squamous cell carcinoma") OR ("atypical nevus") OR ("acne") OR ("psoriasis") OR ("urticaria") OR ("atopic dermatitis") OR ("onychomycosis") OR ("melasma") OR ("solar lentigo") OR ("dermatofibroma") OR ("skin diseases") OR ("skin lesions")) AND (("clinical image") OR ("digital imaging") OR ("web application") OR ("smartphone") OR ("dermatoscopy") OR ("camera") OR ("CAD")) AND (("artificial intelligence") OR ("machine learning") OR ("deep learning") OR ("computer vision") OR ("deep neural networks") OR ("convolutional neural networks") OR ("metaoptima") OR ("automated")) AND (("diagnosis") OR ("diagnosis support") OR ("followup") OR ("segmentation") OR ("detection") OR ("estimation") OR ("classification") OR ("assessment") OR ("severity assessment") OR ("improving"))

As a result, we have compiled 112 articles of which 52 articles have passed the Data Screening:

The editable table is here: https://docs.google.com/spreadsheets/d/1QrUi-Q0REVfh-X6YBD8ExUm0WsYbRoXZXyodLp_fJIo/edit?usp=sharing

ID	Name	Journal/Conference	Impact factor	In scope? Not duplicated?	Appraisal included
001	Explainable artificial intelligence in skin cancer recognition: A systematic review	European Journal of Cancer	10.002	Y	Y
002	Dermatoscopy	Clinics in dermatology	2.797	Y	N
003	Legal and ethical considerations of artificial intelligence in skin cancer diagnosis	The Australasian journal of dermatology	2.481	N	N
004	DermIA: Machine Learning to Improve Skin Cancer Screening	Journal of digital imaging	4.903	Y	Y
005	Artificial intelligence and melanoma: A comprehensive review of clinical, dermoscopic, and histologic applications	Pigment cell & melanoma research	4.159	N	N
006	Deep Learning for Clinical Image Analyses in Oral Squamous Cell Carcinoma: A Review	JAMA otolaryngology-- head & neck surgery	8.961	N	N
007	Non-Melanoma Skin Cancer Detection in the Age of Advanced Technology: A Review	Cancers	6.575	Y	Y
008	Artificial Intelligence as a Diagnostic Tool in Non-Invasive Imaging in the Assessment of Coronary Artery Disease	Medical sciences (Basel, Switzerland)	3.642	N	N
009	Telemedicine and e-Health in the Management of Psoriasis: Improving Patient Outcomes A Narrative Review	Psoriasis (Auckland, N.Z.)	N/A	Y	Y
010	Computational Intelligence-Based Melanoma Detection and Classification Using Dermoscopic Images	Computational intelligence and neuroscience	3.120	Y	N
011	Differential Diagnosis of Rosacea Using Machine Learning and Dermoscopy	Clinical, cosmetic and investigational dermatology	2.765	Y	N
012	Evaluation of a smartphone application for diagnosis of skin diseases	Postepy dermatologii i alergologii	1.664	Y	N
013	Artificial intelligence for the automated single-shot assessment of psoriasis severity	Journal of the European Academy of Dermatology and Venereology : JEADV	9.228	Y	Y
014	A survey, review, and future trends of skin lesion segmentation and classification	Computers in biology and medicine	6.698	Y	Y
015	Digital skin imaging applications, part I: Assessment of image acquisition technique features	Skin Research and Technology	2.240	Y	Y
016	Melanoma segmentation using deep learning with test-time augmentations and conditional random fields	Scientific reports	4.997	Y	Y
017	Development and Validation of a Model to Predict Posttraumatic Stress Disorder and Major Depression After a Motor Vehicle Collision	JAMA psychiatry	25.936	N	N
018	DenseNet-II: an improved deep convolutional neural network for melanoma cancer detection	Soft computing	3.732	Y	Y
019	Automatic skin disease diagnosis using deep learning from clinical image and patient information	Skin health and disease	N/A	Y	Y
020	An interpretable CNN-based CAD system for skin lesion diagnosis	Artificial intelligence in medicine	7.011	Y	Y
021	A cell phone app for facial acne severity assessment	Applied intelligence (Dordrecht, Netherlands)	5.019	Y	Y
022	Deep learning detection of melanoma metastases in lymph nodes	European journal of cancer (Oxford, England : 1990)	10.002	N	N
023	Automated detection of mouse scratching behaviour using convolutional recurrent neural network	Scientific reports	4.997	N	N
024	Extravasation Screening and Severity Prediction from Skin Lesion Image using Deep Neural Networks	Annual International Conference of the IEEE Engineering in Medicine & Biology Society	0.346	Y	N
025	Monitoring of Pigmented Skin Lesions Using 3D Whole Body Imaging	Computer methods and programs in biomedicine	7.027	Y	Y
026	Computer-Aided Assessment of Melanocytic Lesions by Means of a Mitosis Algorithm	Diagnostics	3.992	N	N
027	Machine learning based skin lesion segmentation method with novel borders and hair removal techniques	PloS one	3.752	Y	Y
028	The role of mobile teledermoscopy in skin cancer triage and management during the COVID-19 pandemic	Indian journal of dermatology, venereology and leprology	2.217	Y	N
029	Medical tumor image classification based on Few-shot learning	IEEE/ACM transactions on computational biology and bioinformatics	3.702	N	N
030	Convolutional neural network assistance significantly improves dermatologists' diagnosis of cutaneous tumours using clinical images	European journal of cancer (Oxford, England : 1990)	10.002	Y	Y
031	Skin Lesion Segmentation Using an Ensemble of Different Image Processing Methods	Diagnostics	3.992	Y	Y
032	Attention Cost-Sensitive Deep Learning-Based Approach for Skin Cancer Detection and Classification	Cancers	6.575	Y	Y
033	The Role of Pathology-Based Methods in Qualitative and Quantitative Approaches to Cancer Immunotherapy	Cancers	6.575	N	N
034	ExAID: A multimodal explanation framework for computer-aided diagnosis of skin lesions	Computer methods and programs in biomedicine	7.027	Y	Y
035	Automatic Acne Object Detection and Acne Severity Grading Using Smartphone Images and Artificial Intelligence	Diagnostics	3.992	Y	Y
036	Computer-aided clinical image analysis for non-invasive assessment of tumor thickness in cutaneous melanoma	BMC research notes	0.527	N	N
037	An IoMT-Based Melanoma Lesion Segmentation Using Conditional Generative Adversarial Networks	Sensors	3.900	Y	Y
038	Smartphone-Based Hyperspectral Imaging Low-Cost Application for Telemedicine	Studies in health technology and informatics	0.277	N	N
039	Human Monkeypox Classification from Skin Lesion Images with Deep Pre-trained Network using Mobile Application	Journal of medical systems	4.920	Y	Y
040	A Visually Interpretable Deep Learning Framework for Histopathological Image-Based Skin Cancer Diagnosis	IEEE journal of biomedical and health informatics	7.021	N	N
041	ZooME: Efficient Melanoma Detection Using Zoom-in Attention and Metadata Embedding Deep Neural Network	Annual International Conference of the IEEE Engineering in Medicine & Biology Society	0.346	Y	N
042	Melanoma classification from dermatoscopy images using knowledge distillation for highly imbalanced data	Computers in biology and medicine	6.698	Y	Y
043	Adopting low-shot deep learning for the detection of conjunctival melanoma using ocular surface images	Computer methods and programs in biomedicine	7.027	N	N
044	A rotation meanout network with invariance for dermoscopy image classification and retrieval	Computers in biology and medicine	6.698	Y	Y
045	Implementation of artificial intelligence algorithms for melanoma screening in a primary care setting	PloS one	3.752	Y	Y
046	A Two-Stage Automatic Color Thresholding Technique	Sensors	3.900	N	N
047	A Deep CNN Transformer Hybrid Model for Skin Lesion Classification of Dermoscopic Images Using Focal Loss	Diagnostics	3.992	Y	Y
048	Image analysis of cutaneous melanoma histology: a systematic review and meta-analysis	Scientific reports	4.997	N	N
049	Application of an artificial intelligence system for endoscopic diagnosis of superficial esophageal squamous cell carcinoma	World journal of gastroenterology	5.374	N	N
050	A deep learning based multimodal fusion model for skin lesion diagnosis using smartphone collected clinical images and metadata	Frontiers in surgery	2.350	Y	N
051	A novel approach toward skin cancer classification through fused deep features and neutrosophic environment	Frontiers in public health	6.461	Y	Y
052	Developing a Recognition System for Diagnosing Melanoma Skin Lesions Using Artificial Intelligence Algorithms	Computational and mathematical methods in medicine	2.809	Y	N
053	A machine learning-based, decision support, mobile phone application for diagnosis of common dermatological diseases	Journal of the European Academy of Dermatology and Venereology : JEADV	9.228	Y	Y
054	AutoRadiomics: A Framework for Reproducible Radiomics Research	Frontiers in radiology	N/A	N	N
055	Development and Clinical Evaluation of an Artificial Intelligence Support Tool for Improving Telemedicine Photo Quality	JAMA dermatology	11.816	Y	Y
056	The Role of Machine Learning and Deep Learning Approaches for the Detection of Skin Cancer	Healthcare (Basel, Switzerland)	N/A	Y	Y
057	Diagnosis of Oral Squamous Cell Carcinoma Using Deep Neural Networks and Binary Particle Swarm Optimization on Histopathological Images: An AIoMT Approach	Computational intelligence and neuroscience	3.120	N	N
058	Automatic identification of benign pigmented skin lesions from clinical images using deep convolutional neural network	BMC biotechnology	3.329	Y	Y
059	Classification and diagnostic prediction of breast cancer metastasis on clinical data using machine learning algorithms	Scientific reports	4.997	N	N
060	Reevaluation of missed lung cancer with artificial intelligence	Respiratory medicine case reports	0.354	N	N
061	Accuracy of a Smartphone-Based Artificial Intelligence Application for Classification of Melanomas, Melanocytic Nevi, and Seborrheic Keratoses	Diagnostics	3.992	Y	Y
062	Hair removal in dermoscopy images using variational autoencoders	Skin research and technology	2.240	Y	N
063	Lesion identification and malignancy prediction from clinical dermatological images	Scientific reports	4.997	Y	Y
064	Ms RED: A novel multi-scale residual encoding and decoding network for skin lesion segmentation	Medical image analysis	13.828	Y	Y
065	Data-Efficient Sensor Upgrade Path Using Knowledge Distillation	Sensors	3.900	N	N
066	Quantitative active super-resolution thermal imaging: The melanoma case study	Biomolecular concepts	0.730	N	N
067	Ensemble Method of Convolutional Neural Networks with Directed Acyclic Graph Using Dermoscopic Images: Melanoma Detection Application	Sensors	3.900	Y	Y
068	Incorporating clinical knowledge with constrained classifier chain into a multimodal deep network for melanoma detection	Computers in biology and medicine	6.698	Y	Y
069	NDB-UFES: An oral cancer and leukoplakia dataset composed of histopathological images and patient data	Data in brief	0.131	N	N
070	A Novel Approach for the Shape Characterisation of Non-Melanoma Skin Lesions Using Elliptic Fourier Analyses and Clinical Images	Journal of clinical medicine	4.964	Y	Y
071	Multiclass skin lesion localization and classification using deep learning based features fusion and selection framework for smart healthcare	Neural networks : the official journal of the International Neural Network Society	N/A	Y	Y
072	Light-Dermo: A Lightweight Pretrained Convolution Neural Network for the Diagnosis of Multiclass Skin Lesions	Diagnostics	3.992	Y	Y
073	Evaluation of Computer-Aided Detection (CAD) in Screening Automated Breast Ultrasound Based on Characteristics of CAD Marks and False-Positive Marks	Diagnostics	3.992	N	N
074	Homology-Based Image Processing for Automatic Classification of Histopathological Images of Lung Tissue	Cancers	6.575	N	N
075	Multi-Task and Few-Shot Learning-Based Fully Automatic Deep Learning Platform for Mobile Diagnosis of Skin Diseases	IEEE journal of biomedical and health informatics	7.021	N	N
076	A Multi-Feature Fusion Framework for Automatic Skin Cancer Diagnostics	Diagnostics	3.992	Y	Y
077	Improving Skin cancer Management with ARTificial Intelligence (SMARTI): protocol for a preintervention/postintervention trial of an artificial intelligence system used as a diagnostic aid for skin cancer management in a specialist dermatology setting	BMJ open	3.007	Y	N
078	Computer Aided Diagnosis of Melanoma Using Deep Neural Networks and Game Theory: Application on Dermoscopic Images of Skin Lesions	International journal of molecular sciences	6.208	Y	Y
079	Deeply Supervised Skin Lesions Diagnosis with Stage and Branch Attention	IEEE journal of biomedical and health informatics	7.021	Y	Y
080	The Role in Teledermoscopy of an Inexpensive and Easy-to-Use Smartphone Device for the Classification of Three Types of Skin Lesions Using Convolutional Neural Networks	Diagnostics	3.992	Y	Y
081	Validation of a Market-Approved Artificial Intelligence Mobile Health App for Skin Cancer Screening: A Prospective Multicenter Diagnostic Accuracy Study	Dermatology (Basel, Switzerland)	N/A	Y	Y
082	Non-melanoma skin cancer diagnosis: a comparison between dermoscopic and smartphone images by unified visual and sonification deep learning algorithms	Journal of cancer research and clinical oncology	4.322	Y	Y
083	The Classification of Six Common Skin Diseases Based on Xiangya-Derm: Development of a Chinese Database for Artificial Intelligence	Journal of medical Internet research	7.077	Y	Y
084	Evaluation of Erythema Severity in Dermatoscopic Images of Canine Skin: Erythema Index Assessment and Image Sampling Reliability	Sensors	3.900	N	N
085	Performance of a deep neural network in teledermatology: a single-centre prospective diagnostic study	Journal of the European Academy of Dermatology and Venereology : JEADV	9.228	Y	Y
086	Comparative study on artificial intelligence systems for detecting early esophageal squamous cell carcinoma between narrow-band and white-light imaging	World journal of gastroenterology	5.374	N	N
087	Imtidad: A Reference Architecture and a Case Study on Developing Distributed AI Services for Skin Disease Diagnosis over Cloud, Fog and Edge	Sensors	3.900	Y	Y
088	Over-Detection of Melanoma-Suspect Lesions by a CE-Certified Smartphone App: Performance in Comparison to Dermatologists, 2D and 3D Convolutional Neural Networks in a Prospective Data Set of 1204 Pigmented Skin Lesions Involving Patients’ Perception	Cancers	6.575	Y	Y
089	Design and validation of a new machine-learning-based diagnostic tool for the differentiation of dermatoscopic skin cancer images	PloS one	3.752	Y	Y
090	A CAD system for automatic dysplasia grading on H&E cervical whole-slide images	Scientific reports	4.997	N	N
091	Artificial intelligence-based computer-aided diagnosis system supports diagnosis of lymph node metastasis in esophageal squamous cell carcinoma: A multicenter study	Heliyon	3.776	N	N
092	Reimagining leprosy elimination with AI analysis of a combination of skin lesion images with demographic and clinical data	Lancet regional health. Americas	N/A	Y	Y
093	A smart LED therapy device with an automatic facial acne vulgaris diagnosis based on deep learning and internet of things application	Computers in biology and medicine	6.698	Y	Y
094	Stain color translation of multi-domain OSCC histopathology images using attention gated cGAN	Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society	7.422	N	N
095	AI-based smartphone apps for risk assessment of skin cancer need more evaluation and better regulation	British journal of cancer	9.082	N	N
096	Dermatoscopy of combined blue nevi: a multicentre study of the International Dermoscopy Society	Journal of the European Academy of Dermatology and Venereology : JEADV	9.228	Y	Y
097	Bridging the digital divide among advanced age skin cancer patients	Journal of plastic, reconstructive & aesthetic surgery : JPRAS	N/A	N	N
098	Melanoma: update on dermatoscopy, artificial intelligence for diagnosis, histopathology, genetics, surgery and systemic medical treatment	Italian journal of dermatology and venereology	N/A	N	N
099	Dermatoscopy of Cutaneous Lichen Planus Attempt to Translate Metaphoric Terminology Into Descriptive Terminology	Dermatology practical & conceptual	N/A	N	N
100	Experiences Regarding Use and Implementation of Artificial Intelligence-Supported Follow-Up of Atypical Moles at a Dermatological Outpatient Clinic: Qualitative Study	JMIR dermatology	N/A	N	N
101	Applying an adaptive Otsu-based initialization algorithm to optimize active contour models for skin lesion segmentation	Journal of X-ray science and technology	2.442	Y	N
102	MNet-10: A robust shallow convolutional neural network model performing ablation study on medical images assessing the effectiveness of applying optimal data augmentation technique	Frontiers in medicine	4.900	N	N
103	An attention-based weakly supervised framework for spitzoid melanocytic lesion diagnosis in whole slide images	Artificial intelligence in medicine	7.011	N	N
104	Ex vivo fluorescent confocal microscopy images of oral mucosa: Tissue atlas and evaluation of the learning curve	Journal of biophotonics	3.390	N	N
105	The dermoscopic inverse approach significantly improves the accuracy of human readers for lentigo maligna diagnosis	Journal of the American Academy of Dermatology	15.487	N	N
106	Perilesional sun damage as a diagnostic clue for pigmented actinic keratosis and Bowen's disease	Journal of the European Academy of Dermatology and Venereology : JEADV	9.228	N	N
107	The need for action by evaluators and decision makers in Europe to ensure safe use of medical software	Frontiers in medical technology	N/A	N	N
108	Agreement Between Experts and an Untrained Crowd for Identifying Dermoscopic Features Using a Gamified App: Reader Feasibility Study	JMIR medical informatics	N/A	N	N
109	Assessing the Potential for Patient-led Surveillance After Treatment of Localized Melanoma (MEL-SELF): A Pilot Randomized Clinical Trial	JAMA dermatology	11.816	N	N
110	Response to "Reply to 'A deep learning-based smartphone platform for cutaneous lupus erythematosus classification assistance: Simplifying the diagnosis of complicated diseases.' Has the complicated disease been simplified too much?" Artificial intelligence system is helpful for diagnosis of cutaneous lupus erythematosus	Journal of the American Academy of Dermatology	15.487	N	N
111	A prospective, randomized, single-blinded, crossover trial to investigate the effect of a wearable device in addition to a daily symptom diary for the remote early detection of SARS-CoV-2 infections (COVID-RED): a structured summary of a study protocol for a randomized controlled trial	Trials	2.728	N	N
112	A prospective, randomized, single-blinded, crossover trial to investigate the effect of a wearable device in addition to a daily symptom diary for the Remote Early Detection of SARS-CoV-2 infections (COVID-RED): a structured summary of a study protocol for a randomized controlled trial	Trials	2.728	N	N

Cochrane

Following the PICO format and using the Cochrane advanced search tool, we have defined a global query to retrieve the articles whose title, abstract, or keywords relate to the query:

("dermatosis" OR "skin cancer" OR "chronic skin conditions" OR "inflammatory skin diseases" OR "malignant skin lesions" OR "pigmented skin lesions" OR "melanoma" OR "basal cell carcinoma" OR "squamous cell carcinoma" OR "atypical nevus" OR "acne" OR "psoriasis" OR "urticaria" OR "atopic dermatitis" OR "onychomycosis" OR "melasma" OR "solar lentigo" OR "dermatofibroma" OR "skin diseases" OR "skin lesions")  AND ("clinical image" OR "digital imaging" OR "web application" OR "smartphone" OR "dermatoscopy" OR "camera" OR "CAD")  AND ("artificial intelligence" OR "machine learning" OR "deep learning" OR "computer vision" OR "deep neural networks" OR "convolutional neural networks" OR "metaoptima" OR "automated") AND ("diagnosis" OR "diagnosis support" OR "followup" OR "segmentation" OR "detection" OR "estimation" OR "classification" OR "assessment" OR "severity assessment" OR "improving")

As a result, we have compiled 14 articles of which 1 articles have passed the Data Screening:

The editable table is here: https://docs.google.com/spreadsheets/d/1QrUi-Q0REVfh-X6YBD8ExUm0WsYbRoXZXyodLp_fJIo/edit?usp=sharing

ID	Name	Journal/Conference	Impact factor	In scope? Not duplicated?	Appraisal included
173	Menopausal status, ultrasound and biomarker tests in combination for the diagnosis of ovarian cancer in symptomatic women	Cochrane Database of Systematic Reviews	N/A	N	N
174	Clinical assessment for the detection of oral cavity cancer and potentially malignant disorders in apparently healthy adults	Cochrane Database of Systematic Reviews	N/A	N	N
175	Mobile phone-based interventions for improving contraception use	Cochrane Database of Systematic Reviews	N/A	N	N
176	Electronic symptom monitoring for patients with advanced cancer	The Cochrane Database of Systematic Reviews	N/A	N	N
177	Dermoscopy Augmented Histology Trial, Consensus Agreement Diagnosis Made by Dermatopathology Experts	Not-published yet	N/A	N	N
178	Integrated Basic Science Within the Instructional Design of Pattern Recognition Training	Not-published yet	N/A	N	N
179	At-Home Dermoscopy Artificial Intelligence for Optimizing Early Triage of Skin Cancer	Not-published yet	N/A	N	N
180	AI Augmented Training for Skin Specialists	Not-published yet	N/A	N	N
181	Assessment of deep neural networks for the diagnosis of benign and malignant skin neoplasms in comparison with dermatologists: A retrospective validation study	PLoS medicine	N/A	N	N
182	Perilesional sun damage as a diagnostic clue for pigmented actinic keratosis and Bowen's disease	Journal of the European Academy of Dermatology and Venereology	N/A	N	N
183	Dermoscopy Augmented Histology Trial	Not-published yet	N/A	N	N
184	Development and assessment of an artificial intelligence-based tool for skin condition diagnosis by primary care physicians and nurse practitioners in teledermatology practices	JAMA network open	N/A	Y	Y
185	Study of AI Chromoendoscopy system for detection of early stage esophageal squamous cell carcinoma: a single-center, randomized, open-label, controlled, clinical trials		N/A	N	N
186	Coronavirus (COVID-19): remote care through telehealth	Journal of Primary Health Care	N/A	N	N

Google Scholar

Since the Google Scholar search tool does not allow as much detail as PubMed and Cochrane, we have defined 3 different queries to retrieve the articles. From the retrieved list of articles, sorted by relevance, we compile the first 20 results (2 pages):

Query 1: skin (lesion OR disease OR cancer) (detection OR classification)

As a result, 15 articles have passed the Data Screening:

ID	Name	Journal/Conference	Impact factor	In scope? Not duplicated?	Appraisal included
113	Skin cancer detection: a review using deep learning techniques	International journal of environmental research and public health	4.614	Y	Y
114	A machine learning model for skin disease classification using convolution neural network	International Journal of Computing	0.313	Y	N
115	Deep learning techniques for skin lesion analysis and melanoma cancer detection: a survey of state-of-the-art	Artificial Intelligence Review	9.588	N	N
116	Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts	European Journal of Cancer	10.002	Y	Y
117	An enhanced technique of skin cancer classification using deep convolutional neural network with transfer learning models	Machine Learning with Applications	N/A	Y	Y
118	Multiclass skin cancer classification using EfficientNets-a first step towards preventing skin cancer	Neuroscience Informatics	N/A	Y	Y
119	Intelligence Skin Cancer Detection using IoT with a Fuzzy Expert System	In 2022 International Conference on Cyber Resilience (ICCR)	N/A	Y	Y
120	Machine learning and deep learning methods for skin lesion classification and diagnosis: a systematic review	Diagnostics	3.992	Y	Y
121	Skin lesion classification based on deep convolutional neural networks architectures	Journal of Applied Science and Technology Trends	N/A	Y	Y
122	Multi-class skin lesion detection and classification via teledermatology	IEEE journal of biomedical and health informatics	7.021	Y	Y
123	Monkeypox skin lesion detection using deep learning models: A feasibility study	arXiv	N/A	N	N
124	Intelligent skin cancer detection applying autoencoder, MobileNetV2 and spiking neural networks	Chaos, Solitons & Fractals	9.922	Y	Y
125	Skin lesion analyser: an efficient seven-way multi-class skin cancer classification using MobileNet	Advanced Machine Learning Technologies and Applications: Proceedings of AMLTA 2020	N/A	Y	Y
126	Multiclass skin lesion classification using hybrid deep features selection and extreme learning machine	Sensors	3.900	Y	Y
127	Soft attention improves skin cancer classification performance	In Interpretability of Machine Intelligence in Medical Image Computing	N/A	Y	Y
128	An attention-based mechanism to combine images and metadata in deep learning models applied to skin cancer classification	IEEE journal of biomedical and health informatics	7.021	Y	Y
129	Uncertainty quantification in skin cancer classification using three-way decision-based Bayesian deep learning	Computers in biology and medicine	6.698	Y	Y
130	Skin cancer detection from dermoscopic images using deep learning and fuzzy k-means clustering	Microscopy research and technique	2.893	Y	N
131	A convolutional neural network framework for accurate skin cancer detection	Neural Processing Letters	2.565	N	N
132	Single model deep learning on imbalanced small datasets for skin lesion classification	IEEE transactions on medical imaging	11.037	Y	Y

Query 2: skin (lesion OR disease OR cancer) (detection OR classification) smartphone

As a result, 7 articles have passed the Data Screening:

ID	Name	Journal/Conference	Impact factor	In scope? Not duplicated?	Appraisal included
133	Review of smartphone mobile applications for skin cancer detection: what are the changes in availability, functionality, and costs to users over time?	International Journal of Dermatology	3.204	Y	N
134	A smartphone based application for skin cancer classification using deep learning with clinical images and lesion information	arXiv	N/A	N	N
135	Skin cancer diagnosis using convolutional neural networks for smartphone images: A comparative study	Journal of Radiation Research and Applied Sciences	N/A	Y	Y
136	Human monkeypox classification from skin lesion images with deep pre-trained network using mobile application	Journal of Medical Systems	4.920	N	N
137	Accuracy of commercially available smartphone applications for the detection of melanoma	The British Journal of Dermatology	11.113	N	N
138	A smartphone-based application for an early skin disease prognosis: Towards a lean healthcare system via computer-based vision	Advanced Engineering Informatics	7.862	Y	Y
139	Machine learning and deep learning methods for skin lesion classification and diagnosis: a systematic review	Diagnostics	3.992	N	N
140	New AI-algorithms on smartphones to detect skin cancer in a clinical setting—A validation study	Plos one	3.752	Y	Y
141	Artificial intelligence algorithm with SVM classification using dermascopic images for melanoma diagnosis	Journal of Artificial Intelligence and Capsule Networks	N/A	Y	Y
142	The development of skin lesion detection application in smart handheld devices using deep neural networks	Multimedia Tools and Applications	2.577	Y	N
143	Artificial intelligence and machine learning algorithms for early detection of skin cancer in community and primary care settings: a systematic review	The Lancet Digital Health	N/A	Y	Y
144	Skin cancer disease images classification using deep learning solutions	Multimedia Tools and Applications	2.577	Y	N
145	Over-Detection of Melanoma-Suspect Lesions by a CE-Certified Smartphone App: Performance in Comparison to Dermatologists, 2D and 3D Convolutional Neural Networks in a Prospective Data Set of 1204 Pigmented Skin Lesions Involving Patients’ Perception	Cancers	6.575	N	N
146	Artificial intelligence in the detection of skin cancer	Journal of the American Academy of Dermatology.	15.487	Y	Y
147	Real-time skin cancer detection using neural networks on an embedded device	Bachelor's thesis, University of Twente	N/A	N	N
148	A deep learning based multimodal fusion model for skin lesion diagnosis using smartphone collected clinical images and metadata	Frontiers in Surgery	2.350	N	N
149	Smartphone-based Skin Cancer Detection using Image Processing and Convolutional Neural Network	International Conference on Computing Communication and Networking Technologies (ICCCNT)	N/A	Y	Y
150	The role in teledermoscopy of an inexpensive and easy-to-use smartphone device for the classification of three types of skin lesions using convolutional neural networks	Diagnostics	3.992	N	N
151	Automatic skin disease diagnosis using deep learning from clinical image and patient information	Skin Health and Disease	N/A	N	N
152	A mobile augmented reality application for supporting real-time skin lesion analysis based on deep learning	Journal of Real-Time Image Processing	2.293	Y	N

Query 3: skin (lesion OR disease OR cancer) diagnosis (cnn OR convolutional OR deep learning OR artificial intelligence)

As a result, 4 articles have passed the Data Screening:

ID	Name	Journal/Conference	Impact factor	In scope? Not duplicated?	Appraisal included
153	DSCC_Net: Multi-Classification Deep Learning Models for Diagnosing of Skin Cancer Using Dermoscopic Images	Cancers	6.575	Y	Y
154	Ovary cancer diagnosing empowered with machine learning	International Conference on Business Analytics for Technology and Security (ICBATS)	N/A	N	N
155	Deep Learning-Based Classification of Dermoscopic Images for Skin Lesions	Sakarya University Journal of Computer and Information Sciences	N/A	Y	Y
156	Deep Learning-Based Cancer Detection-Recent Developments, Trend and Challenges	CMES-Computer Modeling in Engineering & Sciences	2.027	Y	N
157	The classification of six common skin diseases based on Xiangya-Derm: development of a Chinese database for artificial intelligence	Journal of Medical Internet Research	7.077	N	N
158	Detection of melanoma with hybrid learning method by removing hair from dermoscopic images using image processing techniques and wavelet transform	Biomedical Signal Processing and Control	5.076	Y	Y
159	A fully automated approach involving neuroimaging and deep learning for Parkinson’s disease detection and severity prediction	PeerJ Computer Science	2.411	N	N
160	Deep learning as a new tool in the diagnosis of mycosis fungoides	Archives of Dermatological Research	3.033	N	N
161	Use of deep learning approaches in cancer diagnosis	Deep Learning for Cancer Diagnosis	N/A	N	N
162	Cancer detection in breast cells using a hybrid method based on deep complex neural network and data mining	Journal of Cancer Research and Clinical Oncology	4.322	N	N
163	Crccn-net: Automated framework for classification of colorectal tissue using histopathological images	Biomedical Signal Processing and Control	5.076	N	N
164	PoxNet22: A fine-tuned model for the classification of monkeypox disease using transfer learning	IEEE Access	3.476	Y	N
165	MITNET: a novel dataset and a two-stage deep learning approach for mitosis recognition in whole slide images of breast cancer tissue	Neural Computing and Applications	5.102	N	N
166	DermoCC-GAN: A new approach for standardizing dermatological images using generative adversarial networks	Computer Methods and Programs in Biomedicine	7.027	Y	Y
167	Deep learning models for cancer stem cell detection: a brief review	Frontiers in Immunology	8.787	N	N
168	RADIC: A tool for diagnosing COVID-19 from chest CT and X-ray scans using deep learning and quad-radiomics	Chemometrics and Intelligent Laboratory Systems	4.175	N	N
169	Tubule-U-Net: a novel dataset and deep learning-based tubule segmentation framework in whole slide images of breast cancer	Scientific Reports	4.997	N	N
170	The Deep Learning Method Differentiates Patients with Bipolar Disorder from Controls with High Accuracy Using EEG Data	Clinical EEG and Neuroscience	2.046	N	N
171	A Novel Deep Learning-Based Approach for Segmentation of Different Type Caries Lesions on Panoramic Radiographs	Diagnostics	3.992	N	N
172	Artificial intelligence in oncology: From bench to clinic	Seminars in Cancer Biology (Vol. 84, pp. 113-128). Academic Press.	17.012	N	N

Cited articles

Two articles, cited by the article with ID 14, have passed the Data Screening:

ID	Name	Journal/Conference	Impact factor	In scope? Not duplicated?	Appraisal included
187	A survey on deep learning for skin lesion segmentation	Medical Image Analysis	13.828	Y	Y
188	Characteristics of publicly available skin cancer image datasets: a systematic review	The Lancet Digital Health	N/A	Y	Y

Data appraisal

All the articles that passed the Data Screening have been reviewed thoroughly to assess their Quality, Relevance, and Contribution. As a result:

• 31 articles have been evaluated as highly relevant.
• 25 articles have been evaluated as relevant.
• 25 articles have been evaluated as non-relevant.

The editable table is here: https://docs.google.com/spreadsheets/d/1eIGPeoYd3BqZFDWA8h6jRL36UBbsy0J6nNunvM-g12g/edit?usp=sharing

ID	Name	Authors	Journal/Conference	Q (<40)	R (<30)	C (<30)	Weighted value
001	Explainable artificial intelligence in skin cancer recognition: A systematic review	Hauser, K., Kurz, A., Haggenmueller, S., Maron, R. C., von Kalle, C., Utikal, J. S., ... & Brinker, T. J.	European Journal of Cancer	40	20	10	70
004	DermIA: Machine Learning to Improve Skin Cancer Screening	Shoen, Ezra; Shoen, Ezra	Journal of digital imaging	0	20	5	25
007	Non-Melanoma Skin Cancer Detection in the Age of Advanced Technology: A Review	Stafford, Haleigh; Buell, Jane; Yaniv, Dan	Cancers	30	25	15	70
009	Telemedicine and e-Health in the Management of Psoriasis: Improving Patient Outcomes A Narrative Review	Havelin, Alison; Hampton, Philip; Hampton, Philip	Psoriasis (Auckland, N.Z.)	25	20	5	50
013	Artificial intelligence for the automated single-shot assessment of psoriasis severity	Okamoto, T; Kawai, M; Kawamura, T	Journal of the European Academy of Dermatology and Venereology : JEADV	10	5	5	20
014	A survey, review, and future trends of skin lesion segmentation and classification	Hasan, Md Kamrul; Ahamad, Md Asif; Yang, Guang	Computers in biology and medicine	38	27	10	75
015	Digital skin imaging applications, part I: Assessment of image acquisition technique features	Sun, Mary D; Kentley, Jonathan; Halpern, Allan C	Skin Research and Technology	35	25	25	85
016	Melanoma segmentation using deep learning with test-time augmentations and conditional random fields	Ashraf, Hassan; Waris, Asim; Niazi, Imran Khan	Scientific reports	35	25	15	75
018	DenseNet-II: an improved deep convolutional neural network for melanoma cancer detection	Girdhar, Nancy; Sinha, Aparna; Gupta, Shivang	Soft computing	5	20	0	25
019	Automatic skin disease diagnosis using deep learning from clinical image and patient information	Muhaba, K A; Dese, K; Simegn, G L	Skin health and disease	35	30	10	75
020	An interpretable CNN-based CAD system for skin lesion diagnosis	López-Labraca, Javier; González-Díaz, Iván; Fueyo-Casado, Alejandro	Artificial intelligence in medicine	40	25	25	90
021	A cell phone app for facial acne severity assessment	Wang, Jiaoju; Luo, Yan; Zhang, Jianglin	Applied intelligence (Dordrecht, Netherlands)	40	28	28	96
025	Monitoring of Pigmented Skin Lesions Using 3D Whole Body Imaging	Ahmedt-Aristizabal, David; Nguyen, Chuong; Wang, Dadong	Computer methods and programs in biomedicine	20	5	10	35
027	Machine learning based skin lesion segmentation method with novel borders and hair removal techniques	Rehman, Mohibur; Ali, Mushtaq; Mustafa Hilal, Anwer	PloS one	20	20	10	50
030	Convolutional neural network assistance significantly improves dermatologists' diagnosis of cutaneous tumours using clinical images	Ba, Wei; Wu, Huan; Li, Cheng X	European journal of cancer (Oxford, England : 1990)	35	30	8	73
031	Skin Lesion Segmentation Using an Ensemble of Different Image Processing Methods	Tamoor, Maria; Naseer, Asma; Zafar, Kashif	Diagnostics	15	15	5	35
032	Attention Cost-Sensitive Deep Learning-Based Approach for Skin Cancer Detection and Classification	Ravi, Vinayakumar; Ravi, Vinayakumar	Cancers	15	10	0	25
034	ExAID: A multimodal explanation framework for computer-aided diagnosis of skin lesions	Lucieri, Adriano; Bajwa, Muhammad Naseer; Ahmed, Sheraz	Computer methods and programs in biomedicine	30	25	20	75
035	Automatic Acne Object Detection and Acne Severity Grading Using Smartphone Images and Artificial Intelligence	Huynh, Quan Thanh; Nguyen, Phuc Hoang; Ngo, Hoan Thanh	Diagnostics	15	25	10	50
037	An IoMT-Based Melanoma Lesion Segmentation Using Conditional Generative Adversarial Networks	Ali, Zeeshan; Naz, Sheneela; Kim, Yongsung	Sensors	15	20	10	45
039	Human Monkeypox Classification from Skin Lesion Images with Deep Pre-trained Network using Mobile Application	Sahin, Veysel Harun; Oztel, Ismail; Yolcu Oztel, Gozde	Journal of medical systems	5	15	0	20
042	Melanoma classification from dermatoscopy images using knowledge distillation for highly imbalanced data	Adepu, Anil Kumar; Sahayam, Subin; Arramraju, Rashmika	Computers in biology and medicine	35	25	25	85
044	A rotation meanout network with invariance for dermoscopy image classification and retrieval	Zhang, Yilan; Xie, Fengying; Liu, Jie	Computers in biology and medicine	25	20	15	60
045	Implementation of artificial intelligence algorithms for melanoma screening in a primary care setting	Giavina-Bianchi, Mara; de Sousa, Raquel Machado; Machado, Birajara Soares	PloS one	35	25	15	75
047	A Deep CNN Transformer Hybrid Model for Skin Lesion Classification of Dermoscopic Images Using Focal Loss	Nie, Yali; Sommella, Paolo; Lundgren, Jan	Diagnostics	10	20	5	35
051	A novel approach toward skin cancer classification through fused deep features and neutrosophic environment	Abdelhafeez, Ahmed; Mohamed, Hoda K; Khalil, Nariman A	Frontiers in public health	5	10	5	20
053	A machine learning-based, decision support, mobile phone application for diagnosis of common dermatological diseases	Pangti, R; Mathur, J; Gupta, S	Journal of the European Academy of Dermatology and Venereology : JEADV	35	30	10	75
055	Development and Clinical Evaluation of an Artificial Intelligence Support Tool for Improving Telemedicine Photo Quality	Vodrahalli, Kailas; Ko, Justin; Daneshjou, Roxana	JAMA dermatology	30	30	15	75
056	The Role of Machine Learning and Deep Learning Approaches for the Detection of Skin Cancer	Mazhar, Tehseen; Haq, Inayatul; Goh, Lucky Poh Wah	Healthcare (Basel, Switzerland)	15	10	0	25
058	Automatic identification of benign pigmented skin lesions from clinical images using deep convolutional neural network	Ding, Hui; Zhang, Eejia; Lin, Tong	BMC biotechnology	20	25	5	50
061	Accuracy of a Smartphone-Based Artificial Intelligence Application for Classification of Melanomas, Melanocytic Nevi, and Seborrheic Keratoses	Liutkus, Jokubas; Kriukas, Arturas; Valiukeviciene, Skaidra	Diagnostics	25	20	5	50
063	Lesion identification and malignancy prediction from clinical dermatological images	Xia, Meng; Kheterpal, Meenal K; Henao, Ricardo	Scientific reports	25	20	10	55
064	Ms RED: A novel multi-scale residual encoding and decoding network for skin lesion segmentation	Dai, Duwei; Dong, Caixia; Luo, Nana	Medical image analysis	35	25	20	80
067	Ensemble Method of Convolutional Neural Networks with Directed Acyclic Graph Using Dermoscopic Images: Melanoma Detection Application	Foahom Gouabou, Arthur Cartel; Damoiseaux, Jean-Luc; Merad, Djamal	Sensors	10	10	10	30
068	Incorporating clinical knowledge with constrained classifier chain into a multimodal deep network for melanoma detection	Wang, Yuheng; Cai, Jiayue; Lee, Tim K	Computers in biology and medicine	15	10	5	30
070	A Novel Approach for the Shape Characterisation of Non-Melanoma Skin Lesions Using Elliptic Fourier Analyses and Clinical Images	Courtenay, Lloyd A; Barbero-García, Inés; Román-Curto, Concepción	Journal of clinical medicine	10	5	5	20
071	Multiclass skin lesion localization and classification using deep learning based features fusion and selection framework for smart healthcare	Maqsood, Sarmad; Damaševičius, Robertas; Damaševičius, Robertas	Neural networks : the official journal of the International Neural Network Society	5	5	5	15
072	Light-Dermo: A Lightweight Pretrained Convolution Neural Network for the Diagnosis of Multiclass Skin Lesions	Baig, Abdul Rauf; Abbas, Qaisar; Ahmed, Alaa E S	Diagnostics	15	15	10	40
076	A Multi-Feature Fusion Framework for Automatic Skin Cancer Diagnostics	Bakheet, Samy; Alsubai, Shtwai; Alqahtani, Abdullah	Diagnostics	25	15	10	50
078	Computer Aided Diagnosis of Melanoma Using Deep Neural Networks and Game Theory: Application on Dermoscopic Images of Skin Lesions	Foahom Gouabou, Arthur Cartel; Collenne, Jules; Merad, Djamal	International journal of molecular sciences	25	20	20	65
079	Deeply Supervised Skin Lesions Diagnosis with Stage and Branch Attention	Dai, Wei; Liu, Rui; Liu, Jun	IEEE journal of biomedical and health informatics	30	25	20	75
080	The Role in Teledermoscopy of an Inexpensive and Easy-to-Use Smartphone Device for the Classification of Three Types of Skin Lesions Using Convolutional Neural Networks	Veronese, Federica; Branciforti, Francesco; Savoia, Paola	Diagnostics	30	25	20	75
081	Validation of a Market-Approved Artificial Intelligence Mobile Health App for Skin Cancer Screening: A Prospective Multicenter Diagnostic Accuracy Study	Sangers, Tobias; Reeder, Suzan; Wakkee, Marlies	Dermatology (Basel, Switzerland)	35	30	15	80
082	Non-melanoma skin cancer diagnosis: a comparison between dermoscopic and smartphone images by unified visual and sonification deep learning algorithms	Dascalu, A; Walker, B N; David, E O	Journal of cancer research and clinical oncology	30	10	5	45
083	The Classification of Six Common Skin Diseases Based on Xiangya-Derm: Development of a Chinese Database for Artificial Intelligence	Huang, Kai; Jiang, Zixi; Zhao, Shuang	Journal of medical Internet research	30	30	20	80
085	Performance of a deep neural network in teledermatology: a single-centre prospective diagnostic study	Muñoz-López, C; Ramírez-Cornejo, C; Navarrete-Dechent, C	Journal of the European Academy of Dermatology and Venereology : JEADV	35	30	10	75
087	Imtidad: A Reference Architecture and a Case Study on Developing Distributed AI Services for Skin Disease Diagnosis over Cloud, Fog and Edge	Janbi, Nourah; Mehmood, Rashid; Yigitcanlar, Tan	Sensors	40	20	20	80
088	Over-Detection of Melanoma-Suspect Lesions by a CE-Certified Smartphone App: Performance in Comparison to Dermatologists, 2D and 3D Convolutional Neural Networks in a Prospective Data Set of 1204 Pigmented Skin Lesions Involving Patients’ Perception	Jahn, Anna Sophie; Navarini, Alexander Andreas; Maul, Lara Valeska	Cancers	15	5	0	20
089	Design and validation of a new machine-learning-based diagnostic tool for the differentiation of dermatoscopic skin cancer images	Tajerian, Amin; Kazemian, Mohsen; Akhavan Malayeri, Ava	PloS one	5	5	5	15
092	Reimagining leprosy elimination with AI analysis of a combination of skin lesion images with demographic and clinical data	Barbieri, Raquel R; Xu, Yixi; Moraes, Milton O	Lancet regional health. Americas	35	25	15	75
093	A smart LED therapy device with an automatic facial acne vulgaris diagnosis based on deep learning and internet of things application	Phan, Duc Tri; Ta, Quoc Bao; Oh, Junghwan	Computers in biology and medicine	30	10	20	60
096	Dermatoscopy of combined blue nevi: a multicentre study of the International Dermoscopy Society	Stojkovic-Filipovic, J; Tiodorovic, D; Kittler, H	Journal of the European Academy of Dermatology and Venereology : JEADV	30	20	10	60
113	Skin cancer detection: a review using deep learning techniques	Dildar, M., Akram, S., Irfan, M., Khan, H. U., Ramzan, M., Mahmood, A. R., ... & Mahnashi, M. H.	International journal of environmental research and public health	40	20	0	60
116	Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts	Haggenmüller, S., Maron, R. C., Hekler, A., Utikal, J. S., Barata, C., Barnhill, R. L., ... & Brinker, T. J.	European Journal of Cancer	20	20	5	45
117	An enhanced technique of skin cancer classification using deep convolutional neural network with transfer learning models	Ali, M. S., Miah, M. S., Haque, J., Rahman, M. M., & Islam, M. K.	Machine Learning with Applications	5	10	0	15
118	Multiclass skin cancer classification using EfficientNets-a first step towards preventing skin cancer	Ali, K., Shaikh, Z. A., Khan, A. A., & Laghari, A. A.	Neuroscience Informatics	10	10	0	20
119	Intelligence Skin Cancer Detection using IoT with a Fuzzy Expert System	Al-Dmour, N. A., Salahat, M., Nair, H. K., Kanwal, N., Saleem, M., & Aziz, N.	In 2022 International Conference on Cyber Resilience (ICCR)	30	15	15	60
120	Machine learning and deep learning methods for skin lesion classification and diagnosis: a systematic review	Kassem, M. A., Hosny, K. M., Damaševičius, R., & Eltoukhy, M. M.	Diagnostics	35	20	5	60
121	Skin lesion classification based on deep convolutional neural networks architectures	Saeed, J., & Zeebaree, S.	Journal of Applied Science and Technology Trends	10	20	5	35
122	Multi-class skin lesion detection and classification via teledermatology	Khan, M. A., Muhammad, K., Sharif, M., Akram, T., & de Albuquerque, V. H. C.	IEEE journal of biomedical and health informatics	20	20	20	60
124	Intelligent skin cancer detection applying autoencoder, MobileNetV2 and spiking neural networks	Toğaçar, M., Cömert, Z., & Ergen, B.	Chaos, Solitons & Fractals	10	10	5	25
125	Skin lesion analyser: an efficient seven-way multi-class skin cancer classification using MobileNet	Chaturvedi, S. S., Gupta, K., & Prasad, P. S.	Advanced Machine Learning Technologies and Applications: Proceedings of AMLTA 2020	5	10	0	15
126	Multiclass skin lesion classification using hybrid deep features selection and extreme learning machine	Afza, F., Sharif, M., Khan, M. A., Tariq, U., Yong, H. S., & Cha, J.	Sensors	10	10	5	25
127	Soft attention improves skin cancer classification performance	Datta, S. K., Shaikh, M. A., Srihari, S. N., & Gao, M.	In Interpretability of Machine Intelligence in Medical Image Computing	25	25	25	75
128	An attention-based mechanism to combine images and metadata in deep learning models applied to skin cancer classification	Pacheco, A. G., & Krohling, R. A.	IEEE journal of biomedical and health informatics	25	25	25	75
129	Uncertainty quantification in skin cancer classification using three-way decision-based Bayesian deep learning	Abdar, M., Samami, M., Mahmoodabad, S. D., Doan, T., Mazoure, B., Hashemifesharaki, R., ... & Nahavandi, S.	Computers in biology and medicine	35	25	20	80
132	Single model deep learning on imbalanced small datasets for skin lesion classification	Yao, P., Shen, S., Xu, M., Liu, P., Zhang, F., Xing, J., ... & Xu, R. X.	IEEE transactions on medical imaging	35	30	30	95
135	Skin cancer diagnosis using convolutional neural networks for smartphone images: A comparative study	Medhat, S., Abdel-Galil, H., Aboutabl, A. E., & Saleh, H.	Journal of Radiation Research and Applied Sciences	5	5	0	10
138	A smartphone-based application for an early skin disease prognosis: Towards a lean healthcare system via computer-based vision	Shahin, M., Chen, F. F., Hosseinzadeh, A., Koodiani, H. K., Shahin, A., & Nafi, O. A.	Advanced Engineering Informatics	10	5	5	20
140	New AI-algorithms on smartphones to detect skin cancer in a clinical setting—A validation study	Kränke, T., Tripolt-Droschl, K., Röd, L., Hofmann-Wellenhof, R., Koppitz, M., & Tripolt, M.	Plos one	35	30	10	75
141	Artificial intelligence algorithm with SVM classification using dermascopic images for melanoma diagnosis	Balasubramaniam, V.	Journal of Artificial Intelligence and Capsule Networks	1	5	5	11
143	Artificial intelligence and machine learning algorithms for early detection of skin cancer in community and primary care settings: a systematic review	Jones, O. T., Matin, R. N., van der Schaar, M., Bhayankaram, K. P., Ranmuthu, C. K. I., Islam, M. S., ... & Walter, F. M.	The Lancet Digital Health	35	20	20	75
146	Artificial intelligence in the detection of skin cancer	Beltrami, E. J., Brown, A. C., Salmon, P. J., Leffell, D. J., Ko, J. M., & Grant-Kels, J. M.	Journal of the American Academy of Dermatology.	35	20	0	55
149	Smartphone-based Skin Cancer Detection using Image Processing and Convolutional Neural Network	Rahman, S., Raihan, M., & Mithila, S. K.	International Conference on Computing Communication and Networking Technologies (ICCCNT)	10	10	0	20
153	DSCC_Net: Multi-Classification Deep Learning Models for Diagnosing of Skin Cancer Using Dermoscopic Images	Tahir, M., Naeem, A., Malik, H., Tanveer, J., Naqvi, R. A., & Lee, S. W.	Cancers	10	10	5	25
155	Deep Learning-Based Classification of Dermoscopic Images for Skin Lesions	SÖNMEZ, A. F., ÇAKAR, S., CEREZCİ, F., KOTAN, M., DELİBAŞOĞLU, İ., & Gülüzar, Ç. İ. T.	Sakarya University Journal of Computer and Information Sciences	10	10	5	25
158	Detection of melanoma with hybrid learning method by removing hair from dermoscopic images using image processing techniques and wavelet transform	Suiçmez, Ç., Kahraman, H. T., Suiçmez, A., Yılmaz, C., & Balcı, F.	Biomedical Signal Processing and Control	10	10	5	25
166	DermoCC-GAN: A new approach for standardizing dermatological images using generative adversarial networks	Salvi, M., Branciforti, F., Veronese, F., Zavattaro, E., Tarantino, V., Savoia, P., & Meiburger, K. M.	Computer Methods and Programs in Biomedicine	30	20	15	65
184	Development and assessment of an artificial intelligence-based tool for skin condition diagnosis by primary care physicians and nurse practitioners in teledermatology practices	Jain, A., Way, D., Gupta, V., Gao, Y., de Oliveira Marinho, G., Hartford, J., ... & Liu, Y.	JAMA network open	35	30	10	75
187	A survey on deep learning for skin lesion segmentation	Mirikharaji, Z., Abhishek, K., Bissoto, A., Barata, C., Avila, S., Valle, E., ... & Hamarneh, G.	Medical Image Analysis	40	30	20	90
188	Characteristics of publicly available skin cancer image datasets: a systematic review	Wen, D., Khan, S. M., Xu, A. J., Ibrahim, H., Smith, L., Caballero, J., ... & Matin, R. N.	The Lancet Digital Health	40	30	10	80

Literature Review results - Facial Palsy

Screening

For each one of the PICO components we have defined the set of keywords as:

	Keywords
Population, Patient, Problem (P)	facial palsy, facial paralysis, facial nerve palsy
Intervention (I)	photo, video, expression, proportion, clinical image, smartphone, monitoring
Comparator (C)	artificial intelligence, machine learning, deep learning, computer vision, deep neural networks, convolutional neural networks, regression, neural network
Outcome (O)	prediction, assessment, severity assessment, improving

In the following, we describe how we use these keywords to form queries for each source of information, the retrieved articles, and the outcome of the Data Screening. Since this screening process does not contain as many articles as the previous one, we do not filter out the compiled articles with the impact factor of their publication source.

PubMed

Following the PICO format and using the PubMed advanced search tool, we have defined a global query to retrieve the articles whose title or abstract relate to the query:

(("facial palsy") OR ("facial paralysis") OR ("facial nerve palsy")) AND ("Photo") OR ("video") OR ("expression") OR ("proportion") OR ("clinical image") OR ("smartphone") OR ("monitoring") AND ("artificial intelligence") OR ("machine learning") OR ("deep learning") OR ("computer vision") OR ("deep neural networks") OR ("convolutional neural networks") OR ("regression") OR ("neural network") AND (("prediction") OR ("assessment") OR ("severity assessment") OR ("improving"))

As a result, we have compiled 21 articles of which 20 articles have passed the Data Screening:

The editable table is here: https://docs.google.com/spreadsheets/d/1eIGPeoYd3BqZFDWA8h6jRL36UBbsy0J6nNunvM-g12g/edit?usp=sharing

ID	Name	Journal/Conference	In scope? Not duplicated?	Appraisal included
001	Reliability and validity of emotrics in the assessment of facial palsy.	Journal of personalized medicine	Y	Y
002	Artificial intelligence-driven video analysis for novel outcome measures after smile reanimation surgery.	Facial plastic surgery & aesthetic medicine	Y	Y
003	Toward an automatic system for computer-aided assessment in facial palsy.	Facial plastic surgery & aesthetic medicine	Y	Y
004	Automated spontaneity assessment after smile reanimation: a machine learning approach.	Plastic and reconstructive surgery	Y	Y
005	A smartphone-based automatic diagnosis system for facial nerve palsy.	Sensors (Basel, Switzerland)	Y	Y
006	Machine learning models for predicting facial nerve palsy in parotid gland surgery for benign tumors.	The Journal of surgical research	Y	Y
007	Clinician-graded electronic facial paralysis assessment: the eface.	Plastic and reconstructive surgery	Y	Y
008	The research for the function evaluation of facial nerve and the mechanisms of rehabilitation training.	Medicine	N	N
009	Artificial neural network as a tool to predict facial nerve palsy in parotid gland surgery for benign tumors.	Medical sciences (Basel, Switzerland)	Y	Y
010	Prediction of long-term facial nerve outcomes with intraoperative nerve monitoring.	Otology & neurotology : official publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology	Y	Y
011	The emerging role of artificial intelligence tools for outcome measurement in facial reanimation surgery: a review.	Facial plastic surgery & aesthetic medicine	Y	Y
012	A survey on computer vision for assistive medical diagnosis from faces.	IEEE journal of biomedical and health informatics	Y	Y
013	Functional outcome of dual reinnervation with cross-facial nerve graft and masseteric nerve transfer for facial paralysis.	Plastic and reconstructive surgery	Y	Y
014	Predicting perceived disfigurement from facial function in patients with unilateral paralysis.	Plastic and reconstructive surgery	Y	Y
015	Reliability between in-person and still photograph assessment of facial function in facial paralysis using the eface facial grading system.	Facial plastic surgery & aesthetic medicine	Y	Y
016	Objective method of assessing and presenting the house-brackmann and regional grades of facial palsy by production of a facogram.	Otology & neurotology : official publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology	Y	Y
017	Facial reanimation surgery restores affect display.	Otology & neurotology : official publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology	Y	Y
018	Deepstroke: an efficient stroke screening framework for emergency rooms with multimodal adversarial deep learning.	Medical image analysis	Y	Y
019	In the eye of the beholder: changes in perceived emotion expression after smile reanimation.	Plastic and reconstructive surgery	Y	Y
020	Quantitative analysis of facial paralysis using local binary patterns in biomedical videos.	IEEE transactions on bio-medical engineering	Y	Y
021	Objective outcomes analysis following microvascular gracilis transfer for facial reanimation: a review of 10 years' experience.	JAMA facial plastic surgery	Y	Y

Cochrane

Following the PICO format and using the Cochrane advanced search tool, we have defined a global query to retrieve the articles whose title, abstract, or keywords relate to the query:

("facial palsy" OR "facial paralysis" OR "facial nerve palsy") AND ("Photo" OR "video" OR "expression" OR "proportion" OR "clinical image" OR "smartphone" OR "monitoring") AND ("artificial intelligence" OR "machine learning" OR "deep learning" OR "computer vision" OR "deep neural networks" OR "convolutional neural networks" OR "regression" OR "neural network") AND ("prediction" OR "assessment" OR "severity assessment" OR "improving")

As a result, we have compiled 4 articles of which 0 articles have passed the Data Screening:

ID	Name	Journal/Conference	In scope? Not duplicated?	Appraisal included
042	Changing perception: facial reanimation surgery improves attractiveness and decreases negative facial perception.	The Laryngoscope	N	N
043	Changing perception: facial reanimation surgery improves attractiveness and decreases negative facial perception.	The Laryngoscope	N	N
044	A randomised controlled trial of the use of aciclovir and/or prednisolone for the early treatment of bell's palsy: the bells study.	--	N	N
045	Efficacy of nerve blocks for episodic migraine.	--	N	N

Google Scholar

Since the Google Scholar search tool does not allow as much detail as PubMed and Cochrane, we have defined a simplified query to retrieve the articles. From the retrieved list of articles, sorted by relevance, we compile the first 20 results (2 pages):

Query: facial (palsy OR paralysis) (prediction OR assessment OR improving) (regression OR machine learning OR computer vision OR neural network OR artificial intelligence)

As a result, 10 articles have passed the Data Screening:

ID	Name	Journal/Conference	In scope? Not duplicated?	Appraisal included
022	Classification of facial paralysis based on machine learning techniques.	BioMedical Engineering OnLine	Y	Y
023	A critical assessment and review of artificial intelligence in facial paralysis analysis: uncovering the truth.	FACE	Y	Y
024	Deep learning for the assessment of facial nerve palsy: opportunities and challenges.	Facial Plastic Surgery	Y	Y
025	Using artificial intelligence to measure facial expression following facial reanimation surgery.	Plastic and reconstructive surgery	N	N
026	Applications of artificial intelligence, machine learning, and deep learning on facial plastic surgeries.	In Cosmetic and reconstructive facial plastic surgery: A review of medical and biomedical engineering and science concepts (pp 281-306)	Y	Y
027	Advances and future directions in the care of patients with facial paralysis.	Operative Techniques in Otolaryngology-Head and Neck Surgery	N	N
028	Toward an automatic system for computer-aided assessment in facial palsy.	Facial Plastic Surgery & Aesthetic Medicine	N	N
029	A privacy preserving diagnostic collaboration framework for facial paralysis using federated learning.	Engineering Applications of Artificial Intelligence	N	N
030	The emerging role of artificial intelligence tools for outcome measurement in facial reanimation surgery: a review.	Facial Plastic Surgery & Aesthetic Medicine	N	N
031	Automatic facial palsy, age and gender detection using a raspberry pi.	BioMedInformatics	Y	Y
032	Artificial intelligence-driven video analysis for novel outcome measures after smile reanimation surgery.	Facial Plastic Surgery & Aesthetic Medicine	N	N
033	In the eye of the beholder: changes in perceived emotion expression after smile reanimation.	Plastic and Reconstructive Surgery	N	N
034	Automatic evaluation of facial paralysis with transfer learning and improved resnet34 neural network.	In 2023 15th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC) (pp 218-222)	Y	Y
035	Computer vision system for facial palsy detection.	Journal of Techniques	Y	Y
036	Automated spontaneity assessment after smile reanimation: a machine learning approach.	Plastic and Reconstructive Surgery	N	N
037	A narrative review of artificial intelligence (ai) for objective assessment of aesthetic endpoints in plastic surgery.	Aesthetic Plastic Surgery	N	N
038	The auto-eface: machine learning-enhanced program yields automated facial palsy assessment tool.	Plastic and Reconstructive Surgery	Y	Y
039	Artificial intelligence for automatic pain assessment: research methods and perspectives.	Pain Research and Management	N	N
040	Svm and logistic regression for facial palsy detection utilizing facial landmark features.	In Proceedings of the 2022 Fourteenth International Conference on Contemporary Computing (pp 43-48)	Y	Y
041	Smart prediction system for facial paralysis.	In 2020 7th International Conference on Electrical and Electronics Engineering (ICEEE) (pp 321-327)	Y	Y

Cited articles

One article, cited by the article with ID 026, have passed the Data Screening:

ID	Name	Journal/Conference	In scope? Not duplicated?	Appraisal included
046	AFLFP: A database with annotated facial landmarks for facial palsy	IEEE Transactions on Computational Social Systems	Y	Y

Data appraisal

All the articles that passed the Data Screening have been reviewed thoroughly to assess their Quality, Relevance, and Contribution. As a result:

• 15 articles have been evaluated as highly relevant.
• 10 articles have been evaluated as relevant.
• 6 articles have been evaluated as non-relevant.

ID	Name	Authors	Journal/Conference	Q (<40)	R (<30)	C (<30)	Weighted Value
001	Reliability and validity of emotrics in the assessment of facial palsy.	Kim, Min Gi; Bae, Cho Rong; Kim, Dae Yul	Journal of personalized medicine	25	25	30	80
002	Artificial intelligence-driven video analysis for novel outcome measures after smile reanimation surgery.	Kollar, Branislav; Schneider, Laura; Eisenhardt, Steffen U	Facial plastic surgery & aesthetic medicine	30	20	20	70
003	Toward an automatic system for computer-aided assessment in facial palsy.	Guarin, Diego L; Yunusova, Yana; Jowett, Nate	Facial plastic surgery & aesthetic medicine	30	25	20	75
004	Automated spontaneity assessment after smile reanimation: a machine learning approach.	Dusseldorp, Joseph R; Guarin, Diego L; Hadlock, Tessa A	Plastic and reconstructive surgery	10	10	5	25
005	A smartphone-based automatic diagnosis system for facial nerve palsy.	Kim, Hyun Seok; Kim, So Young; Park, Kwang Suk	Sensors (Basel, Switzerland)	15	20	10	45
006	Machine learning models for predicting facial nerve palsy in parotid gland surgery for benign tumors.	Chiesa-Estomba, Carlos Miguel; Echaniz, Oier; Graña, Manuel	The Journal of surgical research	30	25	20	75
007	Clinician-graded electronic facial paralysis assessment: the eface.	Banks, Caroline A; Bhama, Prabhat K; Hadlock, Tessa A	Plastic and reconstructive surgery	35	30	20	85
009	Artificial neural network as a tool to predict facial nerve palsy in parotid gland surgery for benign tumors.	Chiesa-Estomba, Carlos M; Sistiaga-Suarez, Jon A; Medela, Alfonso	Medical sciences (Basel, Switzerland)	30	25	20	75
010	Prediction of long-term facial nerve outcomes with intraoperative nerve monitoring.	Isaacson, Brandon; Kileny, Paul R; El-Kashlan, Hussam K	Otology & neurotology : official publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology	20	25	10	55
011	The emerging role of artificial intelligence tools for outcome measurement in facial reanimation surgery: a review.	Fuzi, Jordan; Meller, Catherine; Dusseldorp, Joseph	Facial plastic surgery & aesthetic medicine	40	25	10	75
012	A survey on computer vision for assistive medical diagnosis from faces.	Thevenot, Jerome; Lopez, Miguel Bordallo; Hadid, Abdenour	IEEE journal of biomedical and health informatics	35	15	10	60
013	Functional outcome of dual reinnervation with cross-facial nerve graft and masseteric nerve transfer for facial paralysis.	Kollar, Branislav; Weiss, Jakob B W; Eisenhardt, Steffen U	Plastic and reconstructive surgery	20	10	5	35
014	Predicting perceived disfigurement from facial function in patients with unilateral paralysis.	Lyford-Pike, Sofia; Helwig, Nathaniel E; Hadlock, Tessa A	Plastic and reconstructive surgery	30	10	5	45
015	Reliability between in-person and still photograph assessment of facial function in facial paralysis using the eface facial grading system.	Malka, Ronit; Miller, Matthew; Banks, Caroline	Facial plastic surgery & aesthetic medicine	30	20	20	70
016	Objective method of assessing and presenting the house-brackmann and regional grades of facial palsy by production of a facogram.	O'Reilly, Brian F; Soraghan, John J; He, Shu	Otology & neurotology : official publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology	20	20	10	50
017	Facial reanimation surgery restores affect display.	Dey, Jacob K; Ishii, Masaru; Ishii, Lisa E	Otology & neurotology : official publication of the American Otological Society, American Neurotology Society [and] European Academy of Otology and Neurotology	30	20	10	60
018	Deepstroke: an efficient stroke screening framework for emergency rooms with multimodal adversarial deep learning.	Cai, Tongan; Ni, Haomiao; Wong, Stephen T C	Medical image analysis	35	25	15	75
019	In the eye of the beholder: changes in perceived emotion expression after smile reanimation.	Dusseldorp, Joseph R; Guarin, Diego L; Hadlock, Tessa A	Plastic and reconstructive surgery	30	25	20	75
020	Quantitative analysis of facial paralysis using local binary patterns in biomedical videos.	He, Shu; Soraghan, John J; Xing, Dongshan	IEEE transactions on bio-medical engineering	35	10	10	55
021	Objective outcomes analysis following microvascular gracilis transfer for facial reanimation: a review of 10 years' experience.	Bhama, Prabhat K; Weinberg, Julie S; Hadlock, Tessa A	JAMA facial plastic surgery	30	20	10	60
022	Classification of facial paralysis based on machine learning techniques.	Gaber, A., Taher, M. F., Wahed, M. A., Shalaby, N. M., & Gaber, S.	BioMedical Engineering OnLine	25	30	16	71
023	A critical assessment and review of artificial intelligence in facial paralysis analysis: uncovering the truth.	Mitchell, D. T., Allen, D. Z., Greives, M. R., & Nguyen, P. D.	FACE	25	30	25	80
024	Deep learning for the assessment of facial nerve palsy: opportunities and challenges.	Boochoon, K., Mottaghi, A., Aziz, A., & Pepper, J. P.	Facial Plastic Surgery	35	30	10	75
026	Applications of artificial intelligence, machine learning, and deep learning on facial plastic surgeries.	Tokgöz, E., & Carro, M. A.	In Cosmetic and reconstructive facial plastic surgery: A review of medical and biomedical engineering and science concepts (pp 281-306)	30	10	0	40
031	Automatic facial palsy, age and gender detection using a raspberry pi.	Amsalam, A. S., Al-Naji, A., Daeef, A. Y., & Chahl, J.	BioMedInformatics	10	5	10	25
034	Automatic evaluation of facial paralysis with transfer learning and improved resnet34 neural network.	Fu, R., & Zhou, G.	In 2023 15th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC) (pp 218-222)	5	10	5	20
035	Computer vision system for facial palsy detection.	Amsalam, A. S., Al-Naji, A., Daeef, A. Y., & Chahl, J.	Journal of Techniques	5	10	5	20
038	The auto-eface: machine learning-enhanced program yields automated facial palsy assessment tool.	Miller, M. Q., Hadlock, T. A., Fortier, E., & Guarin, D. L.	Plastic and Reconstructive Surgery	30	30	20	80
040	Svm and logistic regression for facial palsy detection utilizing facial landmark features.	Arora, A., Sinha, A., Bhansali, K., Goel, R., Sharma, I., & Jayal, A.	In Proceedings of the 2022 Fourteenth International Conference on Contemporary Computing (pp 43-48)	15	5	5	25
041	Smart prediction system for facial paralysis.	Ridha, A. M., Shehieb, W., Yacoub, P., Al-Balawneh, K., & Arshad, K.	In 2020 7th International Conference on Electrical and Electronics Engineering (ICEEE) (pp 321-327)	14	5	10	29
046	AFLFP: A database with annotated facial landmarks for facial palsy	Xia, Y., Nduka, C., Kannan, R. Y., Pescarini, E., Berner, J. E., & Yu, H.	IEEE Transactions on Computational Social Systems	35	20	30	85

Responsible evaluators

The qualifications and experience of the responsible evaluators are provided in the T-015-003 Clinical Evaluation Report (CER) to demonstrate that the responsible persons fulfill the requirements to evaluate the product clinical evaluation.

Name	Position	Company	Role in the evaluation	Declaration of interest	CV
Alberto Sabater	Medical Data Scientist	AI Labs Group SL	Editor	Yes	Yes
María Belén Hirigoity	Dermatologist and Medical Advisor to Legit.Health	Hospital 9 de Octubre and Alta Estética clinic	Reviewer	Yes	Yes
Constanza Balboni	Professor of Medicine (Dermatology) and Medical Advisor at Legit.Health	UBA School of Medicine	Reviewer	Yes	Yes
María Diez	Quality Manager and PRRC	AI Labs Group SL	Reviewer	Yes	Yes
Taig Mac Carthy	Design and development manager	AI Labs Group SL	Reviewer	Yes	Yes
Alfonso Medela	Technical Responsible and PRRC	AI Labs Group SL	Approver	Yes	Yes

Record signature meaning

• Author: JD-009
• Reviewer: JD-004 or JD-003
• Approver: JD-005