1{2 "document_id": "lb_qasper_10f50fd914b227d12f503b3ef4ac5fe772e70f44557c17b5",3 "family": "public_qasper",4 "split": "dev",5 "text": "Introduction\nCancer is one of the leading causes of death in the world, with over 80,000 deaths registered in Canada in 2017 (Canadian Cancer Statistics 2017). A computer-aided system for cancer diagnosis usually involves a pathologist rendering a descriptive report after examining the tissue glass slides obtained from the biopsy of a patient. A pathology report contains specific analysis of cells and tissues, and other histopathological indicators that are crucial for diagnosing malignancies. An average sized laboratory may produces a large quantity of pathology reports annually (e.g., in excess of 50,000), but these reports are written in mostly unstructured text and with no direct link to the tissue sample. Furthermore, the report for each patient is a personalized document and offers very high variability in terminology due to lack of standards and may even include misspellings and missing punctuation, clinical diagnoses interspersed with complex explanations, different terminology to label the same malignancy, and information about multiple carcinoma appearances included in a single report BIBREF0 .\nIn Canada, each Provincial and Territorial Cancer Registry (PTCR) is responsible for collecting the data about cancer diseases and reporting them to Statistics Canada (StatCan). Every year, Canadian Cancer Registry (CCR) uses the information sources of StatCan to compile an annual report on cancer and tumor diseases. Many countries have their own cancer registry programs. These programs rely on the acquisition of diagnostic, treatment, and outcome information through manual processing and interpretation from various unstructured sources (e.g., pathology reports, autopsy/laboratory reports, medical billing summaries). The manual classification of cancer pathology reports is a challenging, time-consuming task and requires extensive training BIBREF0 .\nWith the continued growth in the number of cancer patients, and the increase in treatment complexity, cancer registries face a significant challenge in manually reviewing the large quantity of reports BIBREF1 , BIBREF0 . In this situation, Natural Language Processing (NLP) systems can offer a unique opportunity to automatically encode the unstructured reports into structured data. Since, the registries already have access to the large quantity of historically labeled and encoded reports, a supervised machine learning approach of feature extraction and classification is a compelling direction for making their workflow more effective and streamlined. If successful, such a solution would enable processing reports in much lesser time allowing trained personnel to focus on their research and analysis. However, developing an automated solution with high accuracy and consistency across wide variety of reports is a challenging problem.\nFor cancer registries, an important piece of information in a pathology report is the associated ICD-O code which describes the patient's histological diagnosis, as described by the World Health Organization's (WHO) International Classification of Diseases for Oncology BIBREF2 . Prediction of the primary diagnosis from a pathology report provides a valuable starting point for exploration of machine learning techniques for automated cancer surveillance. A major application for this purpose would be “auto-reporting” based on analysis of whole slide images, the digitization of the biopsy glass slides. Structured, summarized and categorized reports can be associated with the image content when searching in large archives. Such as system would be able to drastically increase the efficiency of diagnostic processes for the majority of cases where in spite of obvious primary diagnosis, still time and effort is required from the pathologists to write a descriptive report.\nThe primary objective of our study is to analyze the efficacy of existing machine learning approaches for the automated classification of pathology reports into different diagnosis categories. We demonstrate that TF-IDF feature vectors combined with linear SVM or XGBoost classifier can be an effective method for classification of the reports, achieving up to 83% accuracy. We also show that TF-IDF features are capable of identifying important keywords within a pathology report. Furthermore, we have created a new dataset consisting of 1,949 pathology reports across 37 primary diagnoses. Taken together, our exploratory experiments with a newly introduced dataset on pathology reports opens many new opportunities for researchers to develop a scalable and automatic information extraction from unstructured pathology reports.\nBackground\nNLP approaches for information extraction within the biomedical research areas range from rule-based systems BIBREF3 , to domain-specific systems using feature-based classification BIBREF1 , to the recent deep networks for end-to-end feature extraction and classification BIBREF0 . NLP has had varied degree of success with free-text pathology reports BIBREF4 . Various studies have acknowledge the success of NLP in interpreting pathology reports, especially for classification tasks or extracting a single attribute from a report BIBREF4 , BIBREF5 .\nThe Cancer Text Information Extraction System (caTIES) BIBREF6 is a framework developed in a caBIG project focuses on information extraction from pathology reports. Specifically, caTIES extracts information from surgical pathology reports (SPR) with good precision as well as recall.\nAnother system known as Open Registry BIBREF7 is capable of filtering the reports with disease codes containing cancer. In BIBREF8 , an approach called Automated Retrieval Console (ARC) is proposed which uses machine learning models to predict the degree of association of a given pathology or radiology with the cancer. The performance ranges from an F-measure of 0.75 for lung cancer to 0.94 for colorectal cancer. However, ARC uses domain-specific rules which hiders with the generalization of the approach to variety of pathology reports.\nThis research work is inspired by themes emerging in many of the above studies. Specifically, we are evaluating the task of predicting the primary diagnosis from the pathology report. Unlike previous approaches, the system does not rely on custom rule-based knowledge, domain specific features, balanced dataset with fewer number of classes.\nMaterials and Methods\nWe assembled a dataset of 1,949 cleaned pathology reports. Each report is associated with one of the 37 different primary diagnoses based on IDC-O codes. The reports are collected from four different body parts or primary sites from multiple patients. The distribution of reports across different primary diagnoses and primary sites is reported in tab:report-distribution. The dataset was developed in three steps as follows.\nCollecting pathology reports: The total of 11,112 pathology reports were downloaded from NCI's Genomic Data Commons (GDC) dataset in PDF format BIBREF9 . Out of all PDF files, 1,949 reports were selected across multiple patients from four specific primary sites—thymus, testis, lung, and kidney. The selection was primarily made based on the quality of PDF files.\nCleaning reports: The next step was to extract the text content from these reports. Due to the significant time expense of manually re-typing all the pathology reports, we developed a new strategy to prepare our dataset. We applied an Optical Character Recognition (OCR) software to convert the PDF reports to text files. Then, we manually inspected all generated text files to fix any grammar/spelling issues and irrelevant characters as an artefact produced by the OCR system.\nSplitting into training-testing data: We split the cleaned reports into 70% and 30% for training and testing, respectively. This split resulted in 1,364 training, and 585 testing reports.\nPre-Processing of Reports\nWe pre-processed the reports by setting their text content to lowercase and filtering out any non-alphanumeric characters. We used NLTK library to remove stopping words, e.g., `the', `an', `was', `if' and so on BIBREF10 . We then analyzed the reports to find common bigrams, such as “lung parenchyma”, “microscopic examination”, “lymph node” etc. We joined the biagrams with a hyphen, converting them into a single word. We further removed the words that occur less than 2% in each of the diagnostic category. As well, we removed the words that occur more than 90% across all the categories. We stored each pre-processed report in a separate text file.\nTF-IDF features\nTF-IDF stands for Term Frequency-Inverse Document Frequency, and it is a useful weighting scheme in information retrieval and text mining. TF-IDF signifies the importance of a term in a document within a corpus. It is important to note that a document here refers to a pathology report, a corpus refers to the collection of reports, and a term refers to a single word in a report. The TF-IDF weight for a term INLINEFORM0 in a document INLINEFORM1 is given by DISPLAYFORM0\nWe performed the following steps to transform a pathology report into a feature vector:\nCreate a set of vocabulary containing all unique words from all the pre-processed training reports.\nCreate a zero vector INLINEFORM0 of the same length as the vocabulary.\nFor each word INLINEFORM0 in a report INLINEFORM1 , set the corresponding index in INLINEFORM2 to INLINEFORM3 .\nThe resultant INLINEFORM0 is a feature vector for the report INLINEFORM1 and it is a highly sparse vector.\nKeyword extraction and topic modelling\nThe keyword extraction involves identifying important words within reports that summarizes its content. In contrast, the topic modelling allows grouping these keywords using an intelligent scheme, enabling users to further focus on certain aspects of a document. All the words in a pathology report are sorted according to their TF-IDF weights. The top INLINEFORM0 sorted words constitute the top INLINEFORM1 keywords for the report. The INLINEFORM2 is empirically set to 50 within this research. The extracted keywords are further grouped into different topics by using latent Dirichlet allocation (LDA) BIBREF11 . The keywords in a report are highlighted using the color theme based on their topics.\nEvaluation metrics\nEach model is evaluated using two standard NLP metrics—micro and macro averaged F-scores, the harmonic mean of related metrics precision and recall. For each diagnostic category INLINEFORM0 from a set of 37 different classes INLINEFORM1 , the number of true positives INLINEFORM2 , false positives INLINEFORM3 , and false negatives INLINEFORM4 , the micro F-score is defined as DISPLAYFORM0\nwhereas macro F-score is given by DISPLAYFORM0\nIn summary, micro-averaged metrics have class representation roughly proportional to their test set representation (same as accuracy for classification problem with a single label per data point), whereas macro-averaged metrics are averaged by class without weighting by class prevalence BIBREF12 .\nExperimental setting\nIn this study, we performed two different series of experiments: i) evaluating the performance of TF-IDF features and various machine learning classifiers on the task of predicting primary diagnosis from the text content of a given report, and ii) using TF-IDF and LDA techniques to highlight the important keywords within a report. For the first experiment series, training reports are pre-processed, then their TF-IDF features are extracted. The TF-IDF features and the training labels are used to train different classification models. These different classification models and their hyper-parameters are reported in tab:classifier. The performance of classifiers is measured quantitatively on the test dataset using the evaluation metrics discussed in the previous section. For the second experiment series, a random report is selected and its top 50 keywords are extracted using TF-IDF weights. These 50 keywords are highlighted using different colors based on their associated topic, which are extracted through LDA. A non-expert based qualitative inspection is performed on the extracted keywords and their corresponding topics.\nExperiment Series 1\nA classification model is trained to predict the primary diagnosis given the content of the cancer pathology report. The performance results on this task are reported in tab:results. We can observe that the XGBoost classifier outperformed all other models for both the micro F-score metric, with a score of 0.92, and the macro F-score metric, with a score of 0.31. This was an improvement of 7% for the micro F-score over the next best model, SVM-L, and a marginal improvement of 5% for macro F-score. It is interesting to note that SVM with linear kernels performs much better than SVM with RBF kernel, scoring 9% on the macro F-score and 12% more on the micro F-score. It is suspected that since words used in primary diagnosis itself occur in some reports, thus enabling the linear models to outperform complex models.\nExperiment Series 2\nfig:keywords shows the top 50 keywords highlighted using TF-IDF and LDA. The proposed approach has performed well in highlighting the important regions, for example the topic highlighted with a red color containing “presence range tumor necrosis” provides useful biomarker information to readers.\nConclusions\nWe proposed a simple yet efficient TF-IDF method to extract and corroborate useful keywords from pathology cancer reports. Encoding a pathology report for cancer and tumor surveillance is a laborious task, and sometimes it is subjected to human errors and variability in the interpretation. One of the most important aspects of encoding a pathology report involves extracting the primary diagnosis. This may be very useful for content-based image retrieval to combine with visual information. We used existing classification model and TF-IDF features to predict the primary diagnosis. We achieved up to 92% accuracy using XGBoost classifier. The prediction accuracy empowers the adoption of machine learning methods for automated information extraction from pathology reports.",6 "char_len": 14089,7 "document_hash": "52c6cee9b9c6445ce6f46dc921ced7c8cc205d9ef129381f060162ede8edd8b0",8 "questions": [9 {10 "question_id": "q00",11 "qtype": "qasper",12 "question": "What features are used?"13 }14 ]15}