AI Skin Cancer Diagnosis: Why an Old CNN Won
A small test found a 2015 CNN outperformed two frontier AI models on skin lesions. See what the results show—and where they fall short.
Why a 10-Year-Old CNN Beat Two Frontier AI Models at Skin Lesions
A general-purpose AI model can sound convincing about a skin lesion and still be wrong. That gap matters, because the difference between "benign" and "needs care" can affect whether someone seeks evaluation, gets a biopsy, or walks away reassured when they shouldn't.
In a small test, a dermatologist compared two frontier multimodal models, GPT-6 Astra and Claude Fable 5.1, against a much older image classifier. The image model, based on Inception v3, was first published in 2015 and later fine-tuned on dermatology images. It got 125 of 200 biopsy-proven lesions right. The two large language models each landed at about 42% correct, compared with 62% for the CNN.
That outcome is interesting, but it isn't proof that older AI is better in medicine - or that one small experiment can tell us what's safe to use in clinics. The test also saw substantial errors, including missed melanomas, and the cases were chosen to be difficult. Here's what the dermatologist tested, what the numbers mean, and what product teams should take from it.
What did the skin cancer AI test measure?
The dermatologist pulled 200 lesions at random from MILK10k, a public dataset released through the International Skin Imaging Collaboration in 2025. Each lesion had two images: a clinical close-up and a dermoscopic image, taken with the illuminated magnifying lens used in dermatology. Every lesion was biopsied, so the diagnosis could be checked against histopathology.
The sample split into two equal groups:
- 100 lesions that should not be discharged: 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas (including Bowen's disease), and 20 actinic keratoses.
- 100 lesions considered safe to discharge: 50 benign naevi (moles) and 50 seborrhoeic keratoses.
"Discharged" here is about what the test considered safe to leave without treatment, biopsy, or follow-up - not a real-world clinical decision. The two ideas are related, but they're not the same: recognizing a diagnosis category and deciding whether to recommend an action are separate tasks.
What the models were asked to do
GPT-6 Astra and Claude Fable 5.1 were given both images and no extra patient information. For each case, they had to provide:
- the most likely diagnosis from six categories (or say they were unsure),
- a confidence level,
- a second-choice diagnosis, and
- whether the lesion could be safely discharged.
The researcher used the same prompt for both models at default settings, and neither refused any case. Each lesion was run once, so the test doesn't show how much the answers might vary if you rerun the same prompt. What it reflects is one prompt and one model response - not a full picture of either system.
The CNN was an Inception v3 model pretrained on everyday photographs and then fine-tuned on 68,011 clinical and dermoscopic images from the ISIC Archive. The researcher reports that fine-tuning took about 20 minutes on one H100 GPU. Images of the test lesions (and other images from the same patients) were excluded from its training and validation. The test cases were also not used to tune the model or set its threshold.
The CNN scored both images and combined them, weighting the dermoscopic view more heavily. It then selected from the same six diagnoses. That setup makes the comparison more useful than a test where each system gets different inputs, but the systems still aren't identical. A narrowly trained image classifier and a general multimodal assistant have different designs and goals.
Why a biopsy-proven sample is a hard test
All 200 lesions were biopsied, including the "benign" ones. That makes the sample tougher than what you'd see in everyday screening. A mole chosen for biopsy is more likely to look suspicious to a clinician than a casually observed mole that nobody is worried about. So the test isn't a stand-in for every lesion in primary care, a dermatology clinic, or a consumer photo app.
That difficulty can help expose how systems behave on borderline cases. But it also limits how directly the measured accuracy maps to routine screening. A model's performance depends on the mix of cases, image quality, clinical context, and the decision threshold. Those conditions still need to be tested in the real environment where the system would operate.
How did the models compare?
The CNN got the correct first-choice diagnosis for 125 of 200 lesions - 62%. GPT-6 Astra got 84 correct, and Claude Fable 5.1 got 85, both around 42%. When the correct diagnosis was allowed to count if it appeared in either the first or second choice, the totals were 173 for the CNN, 134 for GPT-6 Astra, and 138 for Claude Fable 5.1.
The CNN also identified more melanomas as its first choice: 13 of 30, versus 8 for GPT-6 Astra and 11 for Claude Fable 5.1. Even those numbers aren't great. In a small test, a system that misses more than half of melanomas as its top diagnosis shouldn't be treated as ready for reliable clinical use.
| Measure | GPT-6 Astra | Claude Fable 5.1 | CNN (Inception v3) |
|---|---|---|---|
| Correct first-choice diagnosis, out of 200 | 84 (42%) | 85 (42%) | 125 (62%) |
| Correct in first or second choice | 134 | 138 | 173 |
| Melanoma as first choice, out of 30 | 8 | 11 | 13 |
| Unsafe lesions kept for further action, out of 100 | 97 | 96 | 77 at preset threshold; 98 or 99 when matched |
| Safe lesions discharged, out of 100 | 32 | 22 | 76 at preset threshold |
| Melanomas discharged, out of 30 | 3 | 4 | 18 at preset threshold; 2 or 1 when matched |
Diagnostic accuracy is not the same as safe triage
This is the key problem with treating "accuracy" as the whole story. At its preset threshold, the CNN discharged 18 of the 30 melanomas, even though it led overall on diagnosis accuracy. That's a serious failure mode.
If you adjust the CNN's threshold to behave more like the LLMs - keeping more unsafe cases for further action - the number of melanomas discharged drops to one or two. But that adjustment also changes how many benign lesions get discharged, so you trade one kind of error for another.
This kind of trade-off is familiar in operations. You can tune a system to avoid missing high-risk cases, but you usually end up with more false alarms and more follow-up work. Cut down on referrals and you may reduce workload - while increasing the odds of false reassurance. There isn't a single "best" threshold unless you spell out the intended use, what level of risk is acceptable, and what each error costs.
The LLMs made different errors than the CNN. GPT-6 Astra often chose basal cell carcinoma: it picked that answer for 75 lesions, even though only 25 were basal cell carcinomas. It also labeled 14 of the 30 melanomas as basal cell carcinoma. Claude Fable 5.1 frequently chose actinic keratosis, assigning it to 53 lesions when the sample contained 20.
The CNN's melanoma mistakes leaned toward benign diagnoses. Of the 17 melanomas it didn't identify, it called eight naevi and six seborrhoeic keratoses. That matters more than a broad claim like "this model is more accurate." In a clinical workflow, what you care about is which errors happen, how often they happen, and what safeguards catch them.
Why did the specialized CNN do better?
This result isn't especially surprising. The CNN was trained for skin image classification. The LLMs are general multimodal systems asked to do a specialized visual task. Being able to combine text and images doesn't automatically make a model the best fit for every image-recognition problem.
A specialized classifier can learn visual patterns tied to a fixed set of categories and a specific image domain. It doesn't have to handle open-ended conversation, coding, document analysis, or thousands of unrelated topics. That narrower scope can help when the task and input are tightly defined.
The LLMs do bring strengths - explanations, the ability to incorporate textual context, and more flexible interaction. But "fluent language" isn't evidence that an image-based diagnosis is correct. In this test, both LLMs gave decisive-sounding responses that were wrong: an invasive melanoma on the ear was treated like a harmless seborrhoeic keratosis, and the patient was told they could be discharged.
The systems were not an even match in every respect
The CNN had been fine-tuned on tens of thousands of dermatology images and was designed to output one of six diagnoses. The LLMs were general models, and the test used a prompt rather than a specialized clinical product. The comparison answers a narrow question: how did these specific systems perform on this task with this sample?
It doesn't establish that CNNs are inherently better than multimodal models. It also doesn't prove that a different prompt, a different model version, or a different clinical workflow would produce the same results. The source notes that newer state-of-the-art systems often use more data and techniques like image-transformer ensembles; a regulated dermatology product built for this job can likely perform better than this baseline.
There's also a potential data-contamination issue. MILK10k was public from 2025, so the LLMs may have encountered its images or labels during training. The CNN's test images were explicitly excluded from its training. That doesn't "explain away" the LLM performance - if anything, exposure could have helped. Still, it's another reason to care about data provenance and genuinely independent test sets.
A patient-facing answer is a different use case
The researcher also ran a simpler patient-facing prompt: show only the clinical photograph and ask whether the person needs to see a doctor. In a separate set of 100 biopsy-proven melanomas, the two LLMs recommended seeing a doctor 99 and 98 times, respectively.
That sounds reassuring, but it's not a full measure of usefulness. The models also advised seeing a doctor for harmless lesions in almost every case. Escalating nearly everything can reduce false reassurance while still being practically unhelpful for triage. It can also push costs and workload onto clinicians and increase unnecessary anxiety for users.
A diagnostic tool, a triage assistant, and a patient education chatbot are different products. Each needs its own success criteria. If a system is meant to support clinicians, it should be judged on the decision it informs - not just on whether its explanation sounds plausible.
What should teams learn from the results?
For product teams building with AI, the lesson is simple: evaluate models against the job they'll actually do, not against a general sense of what's "frontier." If the job is classifying images into known categories, a specialized vision model may be the better fit. If the job is conversation, context gathering, or explaining next steps, an LLM might help - but its role should be constrained by evidence and reviewed outcomes.
A practical evaluation should start with the decision the system will influence. Clarify what counts as a harmful miss, what counts as an unnecessary escalation, and who bears the consequences of each. Then test on representative data with enough high-risk cases to estimate performance without guessing. A test set of 30 melanomas can flag a warning sign, but it can't give you a precise estimate of "melanoma safety."
Build evaluation around the failure modes
Before deployment, teams should ask questions like these:
- What is the exact intended use? Diagnosis, triage, image quality checks, and patient education need different metrics.
- What errors cause the most harm? Track high-risk false negatives separately from less consequential mistakes.
- Does the test data match real use? Include variation in devices, lighting, skin tones, image angles, clinical settings, and disease prevalence.
- Are thresholds calibrated to the workflow? Report sensitivity and specificity, and connect them to referrals, biopsies, and missed cases.
- Can the model abstain? "Uncertain - refer for review" may be safer than a forced label, but it still requires a clear escalation path.
- Is the test independent? Look for patient overlap, duplicate images, public-dataset contamination, and leakage through model selection.
- What happens after launch? Monitor performance, clinician overrides, subgroup differences, and changes in data or workflow.
This isn't just "process." It's how you find out whether the system improves decisions - or just adds confident-looking text on top of a risky mistake.
Treat privacy and deployment as product questions
In the source evaluation, the CNN was small enough to run in a phone browser using WebGPU, according to its author. In that demo, images stayed on the device rather than being uploaded. That can reduce some privacy and data-transfer concerns. But running on-device doesn't automatically make a model clinically valid, nor does it remove the need for informed consent, security review, and clear limitations.
There's a real trade-off, too. Cloud models can be easier to update and may support more complex tasks. Local inference can reduce latency and exposure of sensitive images. The right choice depends on the product's risk profile, device support, maintenance plan, and legal requirements. In healthcare, privacy, reliability, and clinical accountability should be built into the workflow - not bolted on after a promising demo.
The test also shows how a polished natural-language interface can mislead people. Confidence and fluency can look like competence. If a product shows model outputs, it should explain the system's role and limits plainly, avoid implying certainty it hasn't earned, and provide a safe path to human assessment when it's needed.
Conclusion
In this 200-lesion test, a fine-tuned Inception v3 CNN correctly labeled more cases than two frontier LLMs. It's a useful reminder that model capability is task-specific: a general model's breadth doesn't automatically translate into expertise in a narrow, high-stakes visual domain.
This result isn't a reason to deploy the CNN as a medical diagnostic product. Its preset threshold missed too many melanomas, and the sample is too small to establish clinical safety. The practical takeaway is more demanding: define the decision, test the failure modes, set thresholds around real consequences, and keep a qualified human in the loop when the risk requires it. In medicine, the most impressive answer isn't the one that sounds certain. It's the one whose limits and error costs have actually been measured.