Seeing Beneath the Surface: AI Spots Skin Cancer with Greater Accuracy

by

Author: Denis Avetisyan

A new machine vision system, built using a custom convolutional neural network, is showing promise in the early detection of skin lesions.

Researchers demonstrate superior performance over traditional methods and transfer learning for preliminary skin lesion assessment using a deep learning approach trained from scratch.

Early detection remains crucial for improving outcomes in aggressive skin cancers, yet translating clinical expertise into automated diagnostic tools presents significant challenges. This is addressed in ‘A Machine Vision Approach to Preliminary Skin Lesion Assessments’, which explores a machine vision system combining the established ABCD rule with machine learning classification. Results demonstrate that a custom three-layer convolutional neural network, trained from scratch, outperforms both traditional methods and transfer learning approaches-achieving 78.5% accuracy and 86.5% recall-suggesting direct pixel-level learning captures critical diagnostic patterns. Could purpose-built, lightweight architectures unlock more effective computer-aided diagnosis for limited, domain-specific medical imaging datasets?

The Challenge of Early Skin Cancer Detection

The prognosis for melanoma, the most dangerous form of skin cancer, is overwhelmingly positive when detected and treated early; however, current reliance on visual inspection by clinicians introduces significant limitations. This method, while widely accessible, is inherently subjective-subtle variations in lesion appearance can be easily overlooked, and distinguishing between benign moles and early-stage melanomas often proves challenging even for experienced dermatologists. Inter-observer variability-differences in assessment between different doctors-is common, leading to potential delays in diagnosis or, conversely, unnecessary biopsies. Consequently, the critical window for effective intervention can be missed, underscoring the urgent need for more objective and reliable diagnostic approaches to improve early detection rates and patient outcomes.

Current techniques for identifying early skin cancers frequently struggle with the nuances of subtle changes in lesions. Dermatologists rely heavily on visual inspection, a process inherently susceptible to individual interpretation and limited in its ability to detect microscopic differences between healthy and cancerous tissue. Biopsies, while definitive, are invasive and often reserved for suspicious lesions, meaning very early cancers – those without pronounced visual characteristics – may be overlooked. This limitation stems from the fact that early-stage melanomas, and some basal cell carcinomas, can present as remarkably similar to benign moles or simply appear as slight irregularities in skin texture or color. Consequently, these subtle presentations can easily evade detection by the naked eye, highlighting a critical need for more sensitive diagnostic tools capable of revealing these early warning signs before the cancer has a chance to progress.

The rising global incidence of skin cancer, fueled by factors like increased ultraviolet radiation exposure and an aging population, presents a significant public health challenge demanding more effective diagnostic strategies. Current screening practices, often reliant on visual examination by dermatologists, are hampered by subjectivity and limited accessibility, particularly in underserved communities. This escalating need is driving research into innovative tools – from advanced imaging techniques and artificial intelligence-powered diagnostic systems to point-of-care devices – designed to facilitate earlier and more accurate detection. The goal is to move beyond reactive treatment of advanced cancers towards proactive, widespread screening capable of identifying lesions at their earliest, most treatable stages, ultimately reducing morbidity and mortality associated with this increasingly common disease.

Current diagnostic techniques for skin cancer often fall short because they struggle to objectively assess the intricate visual characteristics that distinguish harmless moles from potentially dangerous lesions. Human visual inspection, while commonly employed, is inherently subjective and susceptible to individual interpretation, leading to inconsistencies and missed diagnoses. While tools like dermoscopy enhance visualization, they still rely heavily on the clinician’s expertise to interpret subtle patterns – variations in color, asymmetry, border irregularity, and textural changes. The challenge lies in translating these complex visual cues into quantifiable data; current methods often fail to capture the full spectrum of these features or to weight them appropriately, hindering accurate differentiation and necessitating further, often invasive, diagnostic procedures like biopsies for confirmation. This inability to effectively quantify visual complexity limits the potential for early and non-invasive detection of skin cancer, impacting treatment outcomes and patient prognosis.

Building a Foundation for Assessment: Handcrafted Features

The ABCD rule – encompassing Asymmetry, Border irregularity, Color diversity, and Dermoscopic structures – provides a foundational framework for the quantitative analysis of skin lesion characteristics. Asymmetry assesses the symmetry of the lesion’s shape; Border irregularity evaluates the smoothness and definition of the lesion’s edges; Color diversity quantifies the range of colors present within the lesion; and Dermoscopic structures identifies and characterizes specific features visible under dermoscopy, such as globules, streaks, and networks. These four characteristics, when objectively measured, allow for the creation of quantifiable metrics used in lesion assessment and potentially differentiate benign nevi from malignant melanomas. The rule serves as the basis for developing automated diagnostic tools and standardized evaluation criteria in dermatology.

The Asymmetry and Border Irregularity Scores are calculated using quantifiable metrics to assess these features objectively. Asymmetry Score is determined by comparing the lesion’s shape to an ellipse; deviation from elliptical symmetry yields a higher score. Border Irregularity Score utilizes techniques like Intersection over Union (IoU) to measure the degree to which the lesion’s border deviates from a smooth curve. IoU calculates the overlap between the actual border and a smoothed approximation; a lower IoU value indicates greater irregularity and thus a higher Border Irregularity Score. These scores provide numerical representations of traditionally subjective visual assessments, enabling more consistent and reproducible analysis.

The Color Diversity Score quantifies the complexity of pigmentation within a skin lesion by leveraging the K-Means clustering algorithm. This process involves grouping pixels based on their color values, with the number of clusters – k – representing distinct color groups within the lesion. A higher value of k, determined through optimization techniques, indicates a greater variety of colors present. The score is then calculated based on the distribution of pixels across these clusters; a more uniform distribution – indicating a broad range of colors – results in a higher Color Diversity Score. This metric provides an objective assessment of pigmentation heterogeneity, which is a recognized characteristic of certain dermatological conditions.

The Dermoscopic Structures Score quantifies the presence of key morphological features within a skin lesion using blob detection algorithms. A common method employed is the Laplacian of Gaussian (LoG) filter, which identifies structures by convolving the dermoscopic image with a Gaussian kernel and then applying the Laplacian operator to highlight areas of high contrast corresponding to potential structures like globules, streaks, or dots. The resulting image is then analyzed to count and characterize these blobs, with parameters such as blob size, intensity, and spacing contributing to the overall Dermoscopic Structures Score. This score provides a quantifiable assessment of structural complexity, aiding in the differentiation of benign and malignant lesions.

Refining Assessment: Enhancing Feature Extraction and Classification

Image preprocessing, utilizing techniques like Gaussian, Median, and Flat Average Filtering, is a critical initial step in dermatological image analysis. Gaussian filtering employs a weighted average of neighboring pixels, effectively blurring the image and reducing high-frequency noise while preserving edges to a degree. Median filtering replaces each pixel value with the median of its neighbors, proving particularly effective at removing salt-and-pepper noise and impulsive noise without blurring edges significantly. Flat Average Filtering calculates the average intensity of all pixels within a defined kernel and applies this average to the central pixel, resulting in noise reduction but potentially reducing image sharpness. The application of these filters prior to feature extraction enhances the clarity of relevant image characteristics, improving the accuracy and reliability of subsequent analysis stages by minimizing the impact of image artifacts.

Principal Component Analysis (PCA) is employed as a dimensionality reduction technique to address challenges associated with high-dimensional feature spaces. By transforming the original feature set into a new coordinate system defined by principal components – which are orthogonal linear combinations of the original variables – PCA identifies the directions of maximum variance in the data. This allows for the selection of a reduced subset of these components while retaining most of the original data’s variance, effectively minimizing data redundancy and computational complexity. Consequently, classification algorithms experience improved efficiency and, potentially, enhanced generalization performance due to the reduction in overfitting and the alleviation of the ‘curse of dimensionality’.

Supervised machine learning algorithms are employed for lesion classification using the extracted handcrafted features as input. Logistic Regression, Random Forest, and Support Vector Machines (SVM) are commonly utilized models; each leverages these features to learn patterns indicative of different lesion types. During training, these algorithms adjust internal parameters to minimize prediction error on a labeled dataset of lesions. Once trained, the models can predict the lesion class (e.g., benign or malignant) for new, unseen data based on the identified feature values. Model performance is evaluated using metrics such as accuracy, precision, recall, and F1-score, and hyperparameter tuning is often performed to optimize classification results.

Evaluations of traditional machine learning methodologies utilizing handcrafted features demonstrated a peak accuracy of 59.5% in lesion classification. Concurrent with this accuracy, the recall rate reached 62.0%. These performance metrics were established through rigorous testing of models such as Logistic Regression, representing the upper bounds achievable with feature sets derived from manual design rather than automated learning. It is important to note that these results served as a baseline for comparison with more advanced techniques, highlighting the potential for improvement through alternative approaches.

The Total Dermoscopy Score (TDS) is a validated scoring system designed for malignancy assessment in dermoscopic images. It integrates seven established dermoscopic features – atypical pigmented network, streaks, globules, hypopigmentation, gray-blue veil, regression structures, and vascular patterns – each scored from 0 to 3, resulting in a total score ranging from 0 to 21. Higher TDS scores correlate with increased likelihood of melanoma; a score of 5 or greater is generally considered a threshold for potential malignancy, prompting further investigation such as biopsy. The TDS provides a standardized, reproducible method for evaluating dermoscopic features, aiding clinicians in differential diagnosis and risk stratification.

The Rise of Deep Learning for Automated Diagnosis

Traditionally, medical image diagnosis relied on painstakingly designed features – characteristics manually identified and programmed into algorithms to detect patterns. Convolutional Neural Networks (CNNs) represent a paradigm shift, offering a method that bypasses this need for explicit feature engineering. Instead of relying on human expertise to define what constitutes a relevant indicator of disease, CNNs learn these features directly from the raw image data itself. This is achieved through a hierarchical system of convolutional layers that automatically detect edges, textures, and more complex patterns, building increasingly abstract representations of the image. By learning directly from the data, CNNs can often identify subtle and nuanced indicators that might be missed by handcrafted features, leading to improved diagnostic accuracy and the potential to uncover previously unknown biomarkers. This ability to autonomously extract relevant information makes CNNs a particularly powerful tool in the complex field of medical image analysis.

Despite the power of Convolutional Neural Networks (CNNs) to autonomously learn features from raw image data, effective preprocessing remains a vital step in achieving optimal performance. Techniques such as Otsu’s thresholding, which automatically determines an optimal cutoff value for image segmentation, and various filtering methods – including Gaussian, median, and flat averaging – significantly enhance image quality and reduce noise. These methods prepare the images by improving contrast, smoothing irregularities, and highlighting relevant structures, thereby easing the task for the CNN. While CNNs minimize the need for handcrafted features, they do not eliminate it entirely; preprocessing acts as a crucial initial stage, ensuring that the network receives clean, well-defined input, ultimately leading to improved accuracy and reliability in automated diagnosis.

The efficacy of Convolutional Neural Networks in medical image diagnosis is profoundly linked to the scale of the datasets used for training. Large, well-curated datasets, exemplified by the HAM10000 Dataset – a collection of over 10,000 dermatoscopic images – allow these networks to autonomously learn the intricate features indicative of various conditions. Unlike traditional machine learning methods that rely on manually engineered features, CNNs discern these patterns directly from the image data itself. This automated feature extraction process, fueled by substantial data volume, significantly enhances classification accuracy and enables the identification of subtle visual cues often missed by human observation. Consequently, the availability of expansive datasets is not merely beneficial, but fundamentally crucial for realizing the full potential of deep learning in automated diagnosis, leading to more reliable and precise results.

Recent investigations into automated diagnosis have demonstrated the efficacy of convolutional neural networks, specifically a custom three-layer CNN, in discerning complex patterns within medical imaging. When tested on a 1,000-image subset of the extensive HAM10000 dataset, this network achieved an accuracy of 78.5% and, crucially, a recall rate of 86.5%. This performance represents a substantial improvement over traditional machine learning methods employed for the same task, indicating the CNN’s ability to not only correctly identify positive cases, but also to minimize false negatives-a critical factor in medical diagnosis. The success highlights the potential of deep learning to automatically extract and utilize relevant features from image data, surpassing the limitations of manually engineered feature sets and paving the way for more accurate and efficient diagnostic tools.

Evaluations demonstrated a substantial performance advantage for the custom convolutional neural network when paired with flat average preprocessing. Specifically, the model achieved a Receiver Operating Characteristic Area Under the Curve (ROC-AUC) score of 0.833, a metric reflecting its ability to distinguish between different diagnostic categories. This result represents a marked improvement over the 0.619 ROC-AUC score obtained by a Support Vector Machine (SVM) utilizing traditional machine learning techniques and handcrafted features. The higher ROC-AUC indicates that the CNN, aided by the flat average preprocessing, exhibits significantly enhanced discriminatory power, suggesting a more reliable and accurate diagnostic capability compared to the conventional approach.

The application of transfer learning, exemplified by models like EfficientNet-B0, represents a significant advancement in automated diagnosis. Rather than requiring extensive training from scratch, these models leverage knowledge gained from massive datasets – often unrelated to the specific diagnostic task – and adapt it to the new problem with relatively little additional data. While an initial accuracy of 4% may seem modest, it highlights the potential for rapid prototyping and effective performance even with limited resources. This approach drastically reduces the computational demands and data requirements traditionally associated with deep learning, opening avenues for deployment in resource-constrained settings and accelerating the development of diagnostic tools across various medical domains.

The pursuit of enhanced accuracy in machine vision, as demonstrated by this research into skin lesion analysis, echoes a fundamental challenge: what exactly are we optimizing for, and at what cost? This study highlights the potential of custom-built convolutional neural networks to surpass established methods, yet the limited dataset underscores the fragility of such advancements. As Karl Popper observed, “The more we learn, the more we realize how little we know.” This sentiment resonates deeply; the improvement in diagnostic capabilities, while promising, must be viewed within the context of inherent uncertainty and the constant need for rigorous testing and validation. Algorithmic bias is a mirror of our values, and even sophisticated algorithms remain susceptible to the limitations of the data upon which they are trained.

What Lies Ahead?

The demonstrated efficacy of a custom-built convolutional neural network, while promising, merely highlights the ethical scaffolding inherent in even the most technically proficient algorithms. Achieving higher accuracy on a limited dataset is not a triumph of objectivity, but a reflection of the choices made during network architecture, data labeling, and performance metric selection. Each decision encodes a particular worldview regarding what constitutes a ‘significant’ lesion – a worldview that demands rigorous scrutiny. Scalability without such ethical grounding risks automating bias at an unprecedented rate.

Future work must move beyond the pursuit of incremental gains in diagnostic accuracy. The field should prioritize explainability – understanding why a network classifies a lesion as it does – and actively address the potential for disparate impact. The ABCD rule, while foundational, is itself a simplification of complex dermatological knowledge. Automation of this simplification, without acknowledging its inherent limitations, represents a dangerous path.

Ultimately, the true challenge lies not in building more accurate classifiers, but in designing systems that respect patient privacy as a fundamental design principle, not a post-hoc checkbox. The focus must shift from simply detecting lesions to providing genuinely supportive and equitable healthcare, acknowledging that algorithms, however sophisticated, are merely tools within a larger moral landscape.

Original article: https://arxiv.org/pdf/2601.15539.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-01-25 22:06