Methods: An experienced reference examiner trained three hygienists in correct procedures for obtaining periodontal measures. We obtained duplicate measures of probed pocket depth (PPD) (mm) and the free gingival margin to the cemento-enamel junction (CEJ-GM) (mm) from nine subjects in a pilot study to assess the impact of within subject correlation of binary indices of agreement on the variance of the estimates of exact agreement and agreement within 1 mm. We used these results to design a larger calibration study, where sample sizes were adjusted for the effect of within subject clustering.
Results: We estimated that the within subject clustering of binary indices of agreement resulted in approximately a four-fold increase in the variance of the resulting estimates of percent agreement with the reference examiner for both PPD and CEJ-GM. After adjusting for variance inflation, we estimated that 972 and 988 surfaces were required to obtain estimates of agreement such that the half-widths of the corresponding 95% CIs were fixed at 0.06 and 0.054 for exact agreement and agreement within 1 mm, respectively.
Conclusion: The precision associated with measures of agreement derived from clustered periodontal measures is less than that one would observe from an equal number of independent observations. Failure to account for the dependence among surface-level agreement indices results in a false sense of precision in the resulting reliability estimates.
This work was partially supported by NIH grants P20 RR-017696 and U24 DE-016508.