{Svetlana Kiritchenko Xiaodan Zhu Joel Martin Berry de Bruijn Colin Cherry}

Abstract

Objective: As clinical text mining continues to mature, itspotential as an enabling technology for innovations inpatient care and clinical research is becoming a reality. Acritical part of that process is rigid benchmark testing ofnatural language processing methods on realistic clinicalnarrative. In this paper, the authors describe the designand performance of three state-of-the-art text-miningapplications from the National Research Council ofCanada on evaluations within the 2010 i2b2 challenge.Design: The three systems perform three key steps inclinical information extraction: (1) extraction of medicalproblems, tests, and treatments, from dischargesummaries and progress notes; (2) classification ofassertions made on the medical problems; (3)classification of relations between medical concepts.Machine learning systems performed these tasks usinglarge-dimensional bags of features, as derived from boththe text itself and from external sources: UMLS, cTAKES,and Medline.Measurements: Performance was measured persubtask, using micro-averaged F-scores, as calculated bycomparing system annotations with ground-truthannotations on a test set.Results: The systems ranked high among all submittedsystems in the competition, with the following F-scores:concept extraction 0.8523 (ranked first); assertiondetection 0.9362 (ranked first); relationship detection0.7313 (ranked second).Conclusion: For all tasks, we found that the introductionof a wide range of features was crucial to success.Importantly, our choice of machine learning algorithmsallowed us to be versatile in our feature design, and tointroduce a large number of features without overfittingand without encountering computing-resourcebottlenecks.