New Method Determines Optimal Number of Pathologists Needed for Correct Disease Diagnosis

A new method, named the Observers Needed for Evaluation of Subjective Tests (ONEST), has been developed to determine the optimal number of pathologists needed for a correct diagnosis.

The method developed by researchers at the Texas A&M University School of Public Health (College Station, TX, USA) could improve accuracy of disease diagnosis as pathology tests are mostly subjective, and even trained experts can disagree on results. The researchers developed a statistical framework to assess the performance of a diagnostic test with multiple observers. The proposed method includes an exploratory analysis, a statistical test of whether the observers’ agreement percentage will plateau to a non-zero value, and a statistical model to estimate the agreement percentage and the number of observers for reaching the plateau.

Illustration

The method was applied in a non-small cell lung cancer example and a triple negative breast cancer example using reads of the immunohistochemical tests with SP142 and SP263 assays for expression of Programmed death-ligand 1 (PD-L1) to determine the number of observers needed for evaluation of the subjective tests. The proposed method can indicate whether adding more observers to a test causes the proportion of agreement to plateau. Cases where the curve does not plateau could indicate an unreliable test. In cases where the curve does flatten, the method indicates at least how many observers are needed to reach a stable and reliable estimation of their agreement.

Better understanding of how many observers are needed for optimal accuracy on a diagnostic test will help improve correct diagnosis, the right level of care and disease treatment. The researchers believe this method could be utilized by test creators and regulatory agencies to evaluate newly proposed subjective laboratory tests at different numbers of pathologists, which can ensure that the test will perform reliably in real-world settings.

Related Links:
Texas A&M University School of Public Health