Assessing Dataset Integrity in Brain Tumor MRI Classification

Paper: Scores That Hold, Benchmarks That Leak: Measuring Dataset Contamination in Public Brain-Tumor MRI Classification

Authors: Bhanu Prakash Vangala, Sowmya Guda, Latha Peddi, Navya Vangala

Executive Summary

Recent research has highlighted significant concerns regarding the integrity of datasets used in automated brain tumor classification from MRI scans, a field that has seen deep learning applications achieving reported accuracies above 98% on public benchmarks. While high accuracy figures are often celebrated, this study emphasizes that they do not account for critical factors affecting the reliability of these benchmarks, specifically the independence of training and testing data.

The researchers introduced a framework to evaluate dataset integrity, identifying three layers of potential contamination: duplicate images, patient overlap, and source-label leakage. They conducted an audit of the three most commonly used datasets in this field, comparing them against a control set of chest radiographs that do not involve tumors. Their analysis revealed alarming levels of contamination across all datasets. For instance, 28.8% of the test images in the primary dataset had close duplicates present in the training set. In another dataset, 22.3% of test images were found to be byte-identical to those in the training set. Additionally, 95.5% of the test images could be traced back to patients already represented in the training data. Notably, the study found that even when they removed the identified contaminated images, the overall accuracy of the models remained largely unchanged.

These findings suggest that high accuracy rates can be misleading. They indicate that models may not be effectively learning to identify tumors but could instead be leveraging dataset-specific characteristics that do not correlate with genuine diagnostic capabilities. This raises concerns for biomedical research, as reliance on these benchmarks without considering dataset integrity may lead to the deployment of models that are not truly reliable in clinical settings.

The researchers have made their findings transparent by releasing lists of the contaminated files, patient identifiers, and deduplicated dataset splits. This openness allows other researchers and organizations to better assess the quality of the datasets they use and strive for improved model validation.

The implications of this study are significant for healthcare organizations and technology developers working in medical imaging. It stresses the importance of scrutinizing dataset integrity before drawing conclusions from benchmark performances. While the research does not offer direct applications, it may prompt companies to adopt more rigorous evaluation processes for their AI models, ensuring that true diagnostic capabilities are prioritized over misleading accuracy metrics derived from flawed datasets.

Academic Abstract

Automated classification of brain tumors from MRI is a heavily published application of deep learning in medical imaging, with reported accuracies on public benchmarks routinely exceeding 98%. However, accuracy does not capture a critical dimension of benchmark quality: dataset integrity, defined as the independence of test from training data at the image, patient, and acquisition-source levels. We introduce a three-layer contamination framework comprising duplicate, patient, and source-label leakage to assess the public corpora on which this literature rests. We audit the three most widely used corpora against a chest-radiograph negative control and quantify each layer's effect on measured performance across nine architectures and three evaluation conditions. Contamination is severe at every layer: 28.8% of the dominant corpus's official test split has a near-twin in its own training split, a second corpus leaks 22.3% of its test images byte-identically, 95.5% of traceable test images share a patient with training, and file-header features containing no anatomy separate tumor from no-tumor at 0.959 balanced accuracy, at parity with fine-tuned ResNet backbones. The unexpected result is that removing every identified leaked test image leaves balanced accuracy essentially unchanged: stable performance after deduplication does not establish benchmark integrity. Our findings establish dataset integrity as a distinct, measurable axis of benchmark quality that a stable leaderboard cannot certify. For biomedical research, reported accuracy on these corpora alone does not establish that a model has learned to recognize tumors rather than exploit dataset-specific cues. We release the contaminated-file lists, recovered patient identifiers, and deduplicated splits.