Machine Learning Cytology Tool Improves Cancer Cell Identification

Cytological screening remains central to early cancer detection, but its accuracy depends heavily on expert interpretation of stained cells. Under conventional microscopy, malignant and reactive cells can appear nearly identical because some of their distinguishing features occur below the diffraction limit. Submicron changes in cytoskeletal structure, however, can alter how cells scatter light and provide an additional objective signal for classification. A new study now shows that machine learning analysis of light-scattering spectra can distinguish difficult-to-classify cancer cell types in cytology specimens.

Nara Institute of Science and Technology (NAIST; Nara, Japan) researchers developed a workflow that combines dark-field microscopy with machine learning to classify cells in routine cytology samples. The system captures white-light scattering from each cell across wavelengths of 420–720 nanometers, generating spectra that reflect nanometric structural features. These spectra are reduced using principal component analysis (PCA) and then classified with a support vector machine (SVM).

Image Credit: Adobe Stock

The team evaluated cytology specimens containing cancerous mesothelioma cells and morphologically similar reactive mesothelial cells. After training, the model differentiated these cell types with about 91% accuracy in patient-based validation. The approach also showed potential to distinguish gastric and urothelial cancer cells, as well as several lung and thoracic tumor cell types, although performance varied with tumor type and validation method.

Findings were published in Scientific Reports on August 3, 2026. Collaborators included Kindai University Faculty of Medicine, Nara Hospital, Hyogo Medical University Hospital, Hyogo Medical University School of Medicine, and Osaka Institute of Technology. The study is presented as proof of concept for augmenting conventional cytology with light-scattering information to support diagnostic decision-making. The team is optimizing optical settings and operating protocols and applying advanced machine learning methods to pursue further accuracy gains.

“The result implies that the light scattering spectrum, which contains information at the submicron scale, is very sensitive in detecting differences between cell types compared to visual inspection or image analysis,” said Yuka Tsuri, assistant professor at Nara Institute of Science and Technology.

“Integrating this spectroscopic system with a microscope could serve as a powerful aid for pathologists and potentially improve the accuracy of their diagnoses,” added Tsuri.

Related Links
Nara Institute of Science and Technology