Evaluation of diagnostic term extraction from dental cases: preliminary results

In order to construct a diagnostic terminology in dentistry, first we need to extract diagnostic terms adequately from dental records. Adequate extraction is likely to improve the quality of terms and provide better coverage when structuring a dental terminology. Objective: determine the level of confidence in our extraction process. Methods: we used agreement between experts as an initial measurement in the process of developing a new terminology. Two experts (a dentist and a librarian) extracted terms from dental records and a third dentist served as a judge to measure the agreement between these experts. Having experts come from a different domain increased the variability in extraction of the terms thus increasing the coverage of subsequent terminology. Based on grounded theory, the terms were later analyzed to identify categories or classes of terms that would guide the development of higher level concepts. We calculated the F-measure on a case-by-case basis. F-measure is used to identify the reliability of a gold standard for information retrieval studies. Results: the average and medium F-Measure for 40 cases was 0.815108. The range per case was from 0.582279 to 1. We found that the F-Measure was acceptable. Conclusion: these measurements increase our confidence in the extraction process since the our results show an acceptable level of agreement.