Influence of Number of Surfaces and Observers on Statistical Power in a Multi-observer ROC Radiographic Caries Detection Study

“Objective:” To evaluate the influence of the number of surfaces (Nsurf) and the number of observers (Nobs) on the statistical power of a study comparing the diagnostic accuracy of radiographic systems used for approximal caries lesion detection. “Method:” A data set consisting of 338 surfaces examined by 10 observers using four radiographic systems was available. The presence of a caries lesion was assessed from a 5-point confidence scale. The true diagnosis was established by histological validation. ROC curve areas (Az) were used to express the diagnostic accuracy of the observers with the radiographic systems. Assuming that the Azs were tested by a two-way analysis of variance we performed a simulation study in order to evaluate how the power of this statistical analysis depended on Nsurf and Nobs. As a measure of the statistical power we used the standard error of the difference (sed) between the expected Az of two systems. The simulations were made with Nsurf in the range from 25 to 338 surfaces and Nobs from 2 to 10 observers. “Results:” The simulations showed that the power increased as a function of the total number of evaluations per system (Nsurf ×Nobs). E.g. a design with 2 observers and 200 surfaces (400 evaluations) gave an average sed of 0.028 whereas a design with 4 observers and 100 surfaces (also 400 evaluations) gave an average sed of 0.027. How the total number was attained in relation to the number of surfaces and observers had only marginal influence on the power. “Conclusion:” The statistical power, using a two-way analysis of variance, was dependent on the total number of evaluations. Hence, studies comparing the diagnostic accuracy of several systems can be composed freely in relation to the number of surfaces and observers as long as the total numbers of evaluations are sufficient.