AI Tool Helps See How Cells Work Together Inside Diseased Tissue

Microscopes have long been central to diagnosing disease by allowing doctors to examine stained tissue samples. However, modern medical research now generates vast amounts of additional data, including detailed maps of gene and protein activity within cells. These diverse data types are difficult to interpret together, limiting a full understanding of how diseases develop and progress. A new artificial intelligence (AI) system now brings these data streams into a unified framework, offering a more comprehensive picture of tissue biology and disease.

Researchers at Yale University (New Haven, CT, USA); have developed a computational platform called spEMO, short for spatial multi-modal embeddings, designed to integrate tissue images with molecular and biological information. The system uses Pathology Foundation Models trained on large datasets to interpret images, language-based biological knowledge, and molecular signals. By merging these models into a shared analytical space, spEMO enables coordinated analysis of tissue structure, gene expression, and protein activity.

Image: The spEMO platform combines microscope images with gene and protein data to provide a unified view of disease (Photo courtesy of Adobe Stock)

Researchers demonstrated that spEMO could more accurately distinguish distinct regions within tissues and predict disease states compared with methods relying on a single data type. The system also identified communication patterns between cells and helped generate draft medical reports that integrated visual and genetic information. In evaluations by expert pathologists, AI-generated reports were considered more complete and accurate than those based solely on images. The findings, published in Nature Biomedical Engineering, highlight the potential of multimodal AI in biomedical research.

Using cancer datasets as a case example, the platform identified potential interactions between immune cells within tumors by combining tissue images with predicted gene activity. Such insights may improve understanding of tumor biology and treatment responses. Although still under refinement, the researchers suggest the system could accelerate research, support diagnostic workflows, and contribute to personalized medicine. By integrating multiple biological signals, the technology offers a more holistic approach to disease analysis.

“This approach moves us closer to a more holistic view of disease,” said Dr. Hongyu Zhao, PhD, the study’s senior author and a professor of biostatistics at the Yale School of Public Health. “By bringing together molecular data and tissue structure, we can gain insights that would otherwise be missed.”

Related Links:
Yale University