One of the main problems when machine-learning classifiers are employed in practice is to determine whether classifications assigned to new instances are reliable. The meta-classifier approach is one of the simplest approaches to this problem. Given a base classifier, the approach is to learn a meta classifier that predicts the correctness of each instance classification of the base classifier. The source of the meta training data are the training instances. The meta label of an instance indicates reliable classification, if the instance is classified correctly by the base classifier; otherwise, the meta label indicates unreliable classification. The meta classifier plus the base classifier form one combined classifier. The classification rule of the combined classifier is to assign a class predicted by the base classifier to an instance if the meta classifier decides that the classification is reliable.

While the meta-classifier approach is a relatively simple approach to reliable classification, there is no a complete study of the applicability of the approach. Because of that we propose a research program that aims at systematically studying the meta-classifier approach and its modifications. The main research questions in this program are:

(Q1)          What type of classifiers do we have to learn for meta classifiers, for what type of base classifiers and what type of data?

(Q2)          What is the role of the accuracy of the meta classifiers in the whole scheme?

(Q3)          How do we have to represent meta data?

(Q4)          How can we have to generate meta data?

(Q5)          Do we have to learn meta² classifiers; i.e., meta classifiers of the meta classifiers? And if so, do we have to learn metaⁿ classifiers for n greater than 2? For what n do we have to stop the process of meta-classifier generation?

The research program studying the meta-classifier approach consists of a series of research projects described below. Each of these projects can be a base for a Bachelor project with a possibility to be continued later as a Master project.

The aim of this research project is to study the meta-classifier approach to reliable classification when the representation of the meta training instances coincide with that of the original training instances. The project is experimental. First, a system of the meta-classifier approach has to be implemented in the WEKA data-miming environment. The system has to use the original training-instance representation for the meta instances. It has to allow different classifiers to be chosen for the base classifier and meta classifier. Once the system is implemented, a series of experiments on standard datasets has to be performed. The experiments have to show the behavior of the system on standard data sets when different classifiers are chosen for the base classifier and meta classifier. The results of the experiments have to be used in order to answer the research questions Q1-Q4 (see above).

The aim of this research project is to study the meta-classifier approach to reliable classification when the meta training instances are represented by the probability distributions over the classes in the original data. The project is experimental. First, a system of the meta-classifier approach has to be implemented in the WEKA data-miming environment. The system has to use the class-probability representation of the meta instances. It has to allow different classifiers to be chosen for the base classifier and meta classifier. Once the system is implemented, a series of experiments on standard datasets has to be performed. The experiments have to show the behavior of the system on standard data sets when different classifiers are chosen for the base classifier and meta classifier. The results of the experiments have to be used in order to answer the research questions Q1-Q4 (see above).

The aim of this Bachelor project is to study the pre-class meta-classifier approach to reliable classification when the representation of the meta training instances coincide with that of the original training instances. The pre-class meta-classifier approach differs from the meta-classifier approach in the way the meta labels are formed. In the pre-class meta-classifier approach the meta labels are formed for each class involved (assume that an instance has a true class C and it  receives a classification X from the base classifier; Then, the meta label of the instance indicates reliable classification  for the class C if and only if C=X, otherwise the meta label of the instance indicates unreliable classification  for the class C). The new labeling scheme means that for each class there is one separate meta classifier. The meta classifiers plus the base classifier are able to form one combined classifier. The classification rule of the combined classifier is to assign a class predicted by the base classifier to an instance if the meta classifier of that class decides that the classification is reliable.

The project is experimental. First, a system of the pre-class meta-classifier approach has to be implemented in the WEKA data-miming environment. The system has to use the original training-instance representation for the meta instances. It has to allow different classifiers to be chosen for the base classifier and meta classifiers. Once the system is implemented, a series of experiments on standard datasets has to be performed. The experiments have to show the behavior of the system on standard data sets when different classifiers are chosen for the base classifier and meta classifiers. The results of the experiments have to be used in order to answer the research questions Q1-Q4 (see above).

The aim of this Bachelor project is to study the pre-class meta-classifier approach to reliable classification when the meta training instances are represented by the probability distributions over the classes in the original data. The pre-class meta-classifier approach differs from the meta-classifier approach in the way the meta labels are formed. In the pre-class meta-classifier approach the meta labels are formed for each class (assume that an instance has a true class C and it  receives a classification X from the base classifier. Then, the meta label of the instance indicates reliable classification  for the class C if and only if C=X, otherwise the meta label of the instance indicates unreliable classification  for the class C). The new labeling scheme means that for each class there is one separate meta classifier. The meta classifiers plus the base classifier are able to form one combined classifier. The classification rule of the combined classifier is to assign a class predicted by the base classifier to an instance if the meta classifier of that class decides that the classification is reliable.

The project is experimental. First, a system of the meta-classifier approach has to be implemented in the WEKA data-miming environment. The system has to use the class-probability representation of the meta instances. It has to allow different classifiers to be chosen for the base classifier and meta classifiers. Once the system is implemented, a series of experiments on standard datasets has to be performed. The experiments have to show the behavior of the system on standard data sets when different classifiers are chosen for the base classifier and meta classifiers. The results of the experiments have to be used in order to answer the research questions Q1-Q4 (see above).