New Training Approach Helps AI Distinguish Neighboring Cells in Tissue

Artificial intelligence (AI) can assist with identifying cells in digital pathology images, but densely packed tissue remains difficult to analyze. Neighboring cells may be merged, counted twice, or missed, affecting automated cell counts and spatial measurements. Manual counting can also be time-consuming and subject to variation between observers. Researchers now describe a training strategy that improves AI cell detection by teaching models to consider cellular arrangement within tissue.

Koç University (Istanbul, Turkey) has developed topology-guided hard example mining (TG-HEM), a method designed to improve cell detection in pathology images. The approach trains AI models to focus on image regions where predictions do not capture the spatial organization of cells. Rather than relying only on pixel-level errors, TG-HEM incorporates information about surrounding cell arrangements. The strategy changes model training without requiring a larger or more complex network.

During training, TG-HEM compares predicted cell distributions with annotated reference distributions in sampled image regions. It uses topological data analysis to quantify differences in the structure and organization of these distributions. Regions with larger structural discrepancies receive greater weight during learning, while difficult pixels within those regions are also emphasized.

The researchers evaluated TG-HEM across eight AI models using the KUCell dataset and the publicly available BRCA-M2C breast cancer dataset. TG-HEM improved cell counting and localization compared with baseline models and existing hard example mining approaches. These gains were achieved without additional computational overhead when trained models analyzed new images, and with only a negligible effect on training time.

The study was published in Medical Image Analysis. The research involved Koç University’s Department of Computer Engineering and KUIS AI Center, with collaborators in mathematics and pathology. The team also released the KUCell dataset and implementation code to support further evaluation and development of the approach.

More accurate cell detection could strengthen automated measurements used in digital pathology and support more consistent tissue assessment. The study demonstrated improvements in cell detection on research datasets. The experiments did not establish improved diagnostic accuracy or patient outcomes.

Related Links
Koç University