Abstract
The problem of binary classification is of great interest across many fields, including data mining, satellite imaging, and medical diagnostics. The performance of a classifier is, mostly, measured in terms of the error rate, i.e., the total probability of misclassification. A more general approach is to use the Receiver Operating Characteristic (ROC). The ROC is a plot of all the possible values of one type of errors versus the other one. Very practical and easy-to-interpret summary measures can be derived from such a curve, e.g., the Area Under the Curve (AUC), and the Partial Area Under the Curve (PAUC).
This dissertation studies the assessment of classification rules using the entire ROC space with no parametric assumptions. That is, the present approach requires having no knowledge about the distribution of the data. In addition, when data are scarce the classification rule should be designed and assessed from the single available data set. In the present regulatory setting for public-policy making, e.g., in the area of medical diagnostics, the available data set is required to be split into two disjoint sets, one for design and the other for assessment. In this dissertation, both strategies are studied. Moreover, the techniques developed in this dissertation assume no particular form for the classification rule to be assessed: the methodology is general across classical as well as novel modern architectures. The linear and quadratic discriminants used in the dissertation were selected, simply, for demonstration purposes.
The contemporary use of the expression Computational Intelligence refers to a number of rapidly maturing branches of the general field of artificial intelligence, including neural networks, fuzzy logic, evolutionary algorithms ...etc. Algorithms developed in these subfields to solve classification problems are, from a statistical point of view, nonparametric classification rules. This dissertation may, therefore, provide critical assessment tools for such algorithms when they must be developed within a setting in which data are scarce.
Author’s original abstract, transcribed from the thesis; line wrapping normalised.
Research overview
Assessing classifiers without assuming a particular data distribution.
The dissertation develops a general framework for evaluating binary classification rules through receiver operating characteristic analysis. It considers full and partial area under the ROC curve, performance at a chosen operating point, and the uncertainty introduced when training and assessment must share a limited dataset.
The methods apply across classical and modern classifier architectures, with particular relevance to medical diagnostics and other high-stakes decision systems.
Suggested citation: Yousef, Waleed Ahmed. “Assessment of Statistical Classification Rules: Implications for Computational Intelligence.” Doctoral dissertation, The George Washington University, 2006.