# Black-Box Classifier Interpretation Using Decision Tree and Fuzzy Logic-Based Classifier Implementation

## Article information

## Abstract

Black-box classifiers, such as artificial neural network and support vector machine, are a popular classifier because of its remarkable performance. They are applied in various fields such as inductive inferences, classifications, or regressions. However, by its characteristics, they cannot provide appropriate explanations how the classification results are derived. Therefore, there are plenty of actively discussed researches about interpreting trained black-box classifiers. In this paper, we propose a method to make a fuzzy logic-based classifier using extracted rules from the artificial neural network and support vector machine in order to interpret internal structures. As an object of classification, an anomalous propagation echo is selected which occurs frequently in radar data and becomes the problem in a precipitation estimation process. After applying a clustering method, learning dataset is generated from clusters. Using the learning dataset, artificial neural network and support vector machine are implemented. After that, decision trees for each classifier are generated. And they are used to implement simplified fuzzy logic-based classifiers by rule extraction and input selection. Finally, we can verify and compare performances. With actual occurrence cased of the anomalous propagation echo, we can determine the inner structures of the black-box classifiers.

## 1. Introduction

The successful applications of black-box models in various fields such as engineering, science, marketing and medicine have provided an evidence of the black-box models over past years [1]. Among the black-box models, artificial neural networks (ANNs) and support vector machines (SVMs) are representative models and they have been proven their usefulness with remarkable performances. The artificial neural networks, which are inspired by microscopic biological models, have been widely used as solving many regression and classification problems [2, 3]. The support vector machines, which are based on the idea of structural risk minimization instead of empirical risk minimization, has been introduced as power equipments for solving regression and classification problems [4, 5].

However, as always, everything has two sides, and ANNs and SVMs are not an exemption. Even the ANNs and the SVMs attain remarkable performances in regression and classification problems, these models cannot provide delineation in a comprehensible form the process through which a given output generated by the models have been reached [1, 6]. It leads to prevent to apply the black-box models in some problems which is essential that a system user should be able to validate the output of the models under all possible input conditions such as airlines, power plants, medical care, and so on. Therefore, researches and techniques for rule extraction from ANNS and SVMs have been introduced to improve this problem and aid in elucidation of their decisions [1, 6].

Rule extraction is the process of developing natural language-like syntax that describes the behavior of black-box models, and changes the models into a white-box systems by translating the internal knowledge of the models into a set of symbolic rules [7]. Many rule extraction algorithms have been designed to reveal the information concealed in the black-box models. In case of the rule extraction algorithms for the ANNs, there are NeuroRule [8], RX [9], GLARE [10], OSRE [11], etc. And in case of the rule extraction algorithms for the SVMs, there are SQRex-SVM [12], RulExSVM [13], SVM DT [14], and so on.

In this paper, we propose a method to implement a fuzzy logic-based classifier implementation using a decision tree from the black-box classifiers. First, we choose a decision tree-based indirect interpretive method in order to delieate the black-box models. Second, we convert the derived decision trees to fuzzy logic-based classifiers for providing a linguistic variable-based classificaton method to users which is easily comprehensible. We choose both artifiical neural network and support vector machine. Further, an anomalous propagation echo is selected in this paper as an object of classification which occurs frequently in radar data and becomes the problem in a precipitation estimation process.

The rest of the paper is organized as follows. In Section 2, we explain the entire proposed system which consists of ANN, SVM, decision tree, rule extraction, and fuzzy logic-based classifier. And in Section 3, we describe anomalous propagation echo as a target of classification methods. After that, the experimental results with actual anomalous propagation echo occurrence cases are presented in Section 4. Finally, conclusion and future work are shown in Section 5.

## 2. Interpretation of Black-Box Classifiers

### 2.1 ANN and SVM

An ANN is a biologically inspired computational model formed from lots of artificial neurons, connected with weights which constitute the neural structure. Although a single neuron can perform certain simple information processing functions, known as a perceptron model, the power of neural computations comes from connecting neurons in a network. ANNs are capable of processing extensive amounts of data, and making accurate regression and classification results [3].

A SVM is a binary classification method that divides the given data into two groups in the best possible way by using hyperplanes. This method is based on a structural risk minimization method to reduce the error rather than the empirical risk minimization method used in traditional statistical learning theory. In other words, after the division of an entire group into subgroups, a decision function is selected. This function can minimize the empirical risk for the subgroups. Thus, the SVM method has the advantage of achieving great performance in classification, regression, and estimation processes by using a relatively low amount of the given learning data. Without any knowledge of the mapping, the SVM finds the optimal hyperplane by using the dot product functions in feature space using kernel functions. The solution of the optimal hyperplane can be written as a combination of a few input points that are called support vectors [4, 5].

The similarities between ANN and SVM is as follows. First, both ANN and SVM produce black-box model, which was the main motivation behind the rule extraction studies. Second, they can deal with nonlinear models using their own properties. In case of ANN, hidden layers and nonlinear activation functions make the ANN handle nonlinear systems. And in case of SVM, mapping to higher dimensional feature space and nonlinear kernel function make the SVM handle nonlinear systems. Further, their decision function form looks similar as following equations.

where *lhs* is last hidden layer, *ω** ^{i}* is the weight from last hidden layer to the output layer in equation (1), and where

*sv*is the model support vectors,

*α*

*is Lagrange multiplier,*

_{s}*K*(

*χ*

_{s}*; χ*) is a kernel function in equation (2), respectively.

### 2.2 Rule Extraction Using Decision Tree and Fuzzy Logic-Based Classifier Implementation

Rule extraction methods search through inner structure of given classifiers and analyze their operating principle. It is important to figure out how the classifier derives results using given data especially the case which both derivation process and result are crucial such as power plant, airlines, medical care, and so on. There are lots of ongoing researches about analyzing the ANNs and SVMs. In case of the rule extraction algorithms for the ANNs, there are NeuroRule [8], RX [9], GLARE [10], OSRE [11], etc. And in case of the rule extraction algorithms for the SVMs, there are SQRex-SVM [12], RulExSVM [13], SVM DT [14], and so on.

Among these rule extraction methods, we select a method which uses a decision tree. This method is originated from the rule extraction method for SVM [15] using decision tree and artificially labelled dataset. The idea for creating artificially labelled datasets where the given class is replaced by the classified results by the SVM and ANN without even looking at their individual inner structures. The artificially generated dataset can be collaborated with another classification methods which are comprehensible their entire principles.

The rule extraction method using decision tree consists of the following steps. The steps are summarized in Figure 1.

Step 1: Normalize given learning data.

Step 2: Divide the normalized data into four pieces, and name the divided data as A, B, C, and D, respectively.

Step 3: Implement SVM and ANN using the dataset A.

Step 4: Apply dataset B to implemented SVM and ANN, described as B

_{1}and B_{2}. The results can be considered as representations of each classifier.Step 5: Implement decision tree using B

_{1}and B_{2}.Step 6: Verify performances of decision trees.

Step 7: Derive a series of crisp rules from each decision tree.

Step 8: Convert crisp rules into fuzzy rules for constructing fuzzy logic-based classifiers.

Step 9: Set input and output membership parameters and build fuzzy logic-based classifiers.

Figure 2 shows an example of decision tree for deriving a fuzzy inference system. We applied 5 properties as inputs of the SVM and ANN: a centroid altitude of the cluster (x_{1}), an average reflectivity data (x_{2}), a maximum reflectivity data (x_{3}), an average Doppler velocity (x_{4}), and a minimum Doppler velocity (x_{5}). In Section 3, we describe how the input variables are generated. And the decision tree indicates there is only 3 important inputs for separating anomalous propagation echo. Also, we can construct fuzzy rules using the decision tree as follows. The first fuzzy rules are derived from Figure 2, and the second fuzzy rules are derived from Figure 4.

Rule 1: If

**x**_{1}is*small*, then**y**is*NOTAP*.Rule 2: If

**x**_{1}is*large*and**x**_{5}is*small*, then**y**is*NOTAP*.Rule 3: If

**x**_{1}is*large*and**x**_{5}is*large*and**x**_{2}is*small*, then**y**is*AP.*Rule 4: If

**x**_{1}is*large*and**x**_{5}is*large*and**x**_{2}is*large*, then**y**is*NOTAP*.Rule 1: If

**x**_{5}is*large*, then**y**is*NOTAP*.Rule 2: If

**x**_{5}is*small*and**x**_{3}is*small*, then**y**is*NOTAP*.Rule 3: If

**x**_{5}is*small*and**x**_{3}is*large*and**x**_{1}is*small*, then**y**is*NOTAP*.Rule 4: If

**x**_{5}is*small*and**x**_{3}is*large*and**x**_{1}is*large*, then**y**is*AP*.

From these rules, the SVM and the ANN consider different input as most significant variable according to the roots of the decision trees and the induced fuzzy rules. In case of the SVM, the centroid altitude of the cluster is most important variable. On the other hands, the minimum Doppler velocity is most important in case of the ANN. The common input variables are **x**_{1} and **x**_{5}, and the different input variables are **x**_{2} and **x**_{3}. Further, **x**_{4} seems not significant influence because it is not shown in the trees and rules. Their input and output membership functions are generated as shown in Figures 4 and 5, respectively. The functions are trapezoidal shaped function.

## 3. Anomalous Propagation Echo

Due to properties of remote sensing device, the radar observation efficiency depends on the atmospheric condition. In other words, a beam path of weather radar can be changed by temperature, humidity, etc. The changed beam paths can be categorized as sub-refraction, normal refraction, super-refraction, and ducting according to its refractive index [16]. The sub-refraction phenomenon occurs when the radar beam is refracted toward opposite way of surface more than the normal refraction. It causes relatively low influence to weather forecasting. However, when the radar beam is deviated toward the surface by super-refraction or ducting, the resultant echo represents reflection of the ground or the sea surface which is not a meteorological target. It is called as an anomalous propagation echo. The weather radar computes altitude of observation targets consider as the normal refraction of the radar beam. Therefore, unexpected echoes could appear in the observation region of the weather radar by a surface scattering when the super-refraction or the ducting occurs.

It is one of the representative contamination source in the weather forecasting process because it induces a severe problem in quantitative precipitation estimation. The anomalous propagation echo should be removed from radar data because the echo originating from the surfaces can be misinterpreted as heavy precipitation in low altitude. In short, the refracted signals may lead to large overestimates of precipitation by the radar beam seeing surface instead of the atmosphere. Also, its location is difficult to predict. Furthermore, when the radar beam refracts toward the surface more severely, the intensity and extension of clutter areas can also change [17].

The entire proposed system is shown in Figure 6. The detailed sequence is described below. First of all, we need to clarify why we select corrected reflectivity (CZ) and Doppler velocity data (VR). There are several kinds of useful information in raw radar data such as spectrum width (SW) and uncorrected reflectivity (DZ). According to recent research for the anomalous propagation echo classification, the echo has following properties [18]: a near-zero radial velocity, a low spectrum width, a high texture of the reflectivity field, and so on. In this reason, we select the corrected reflectivity and a Doppler velocity data as input features.

Due to observation principles of the weather radar, the raw radar data follows the spherical coordinate. Therefore, in order to analyze the radar data intuitively, a coordinate conversion process should be applied from spherical to Cartesian. This process makes to apply a clustering algorithm for grouping individual point data. Also, in order to find same locations, the coordinate conversion process is applied to both the corrected reflectivity and Doppler velocity data.

After the coordinate conversion, a spatial clustering method [19] which is one of hierarchical clustering methods is applied for grouping the reflectivity data. The clustered data is easier to deal with than the raw data because the radar data includes millions of data points. The proposed system uses statistical features derived from the clusters such as mean, minimum, maximum and its centroid position.

In feature extraction process, five properties are derived and used as inputs: centroid altitude of the cluster (x_{1}), average reflectivity data (x_{2}), maximum reflectivity data (x_{3}), average Doppler velocity (x_{4}), and minimum Doppler velocity (x_{5}). The reason why we select the centroid altitude of the cluster is that the anomalous propagation echo appears in low altitude by its own properties.

After the feature extraction process, a classification method is applied. The detailed sequences are described in next section. Using the classifiers, each cluster is classified +1 or −1, which indicates that the selected cluster is anomalous propagation echo or others, respectively. The classified clusters which determined as an anomalous propagation echo are removed from the corrected reflectivity data.

After the removal process is done, the reverse coordinate conversion process is applied: from Cartesian to spherical coordinate. Finally, the processed radar data without the anomalous propagation echo is generated.

## 4. Experimental Results

For the purpose of verifying the proposed system, we selected actual appearance cases in this paper for training and testing. Figures 7 and 8 show the classification results using the implemented fuzzy inference system with actual appearance cases of the anomalous propagation echo.

The case shown in Figure 7 indicates a case with small amount of precipitation echo in left lower side. As shown in Figure 7(a), a squared mark on the center region represents as the anomalous propagation echo. Figure 7(b) shows the radar image without the classified anomalous propagation echo by the proposed system. And Figure 7(c) describes the separated anomalous propagation echo.

The case shown in Figure 8 indicates an independent appearance case of the anomalous propagation echo. As shown in Figure 8(a), the entire region in the squared marks represent as the anomalous propagation echo. Figure 8(b) shows the radar image without the classified anomalous propagation echo by the proposed method. And Figure 8(c) shows the anomalous propagation echo only.

From Figures 7 and 8, it is confirmed that the most of the regions of the anomalous propagation echo are removed in both cases: with and without precipitation echoes. In conclusion, the induced rule-based fuzzy inference system from the black-box models can be evaluated well according to these experiment results and accuracy comparison results.

There are several performance indexes for verifying classification methods such as accuracy, precision, and so on. In this paper, a confusion matrix is applied for calculating the accuracy as shown in equation (3).

Each parameter in equation (3) indicates as follow: TP for true positive, TN for true negative, FP for false positive, and FN for false negative. The true parameter indicates the anomalous propagation echo. And the false parameter is the other echoes. In this paper, we select three radar sites for experiments and evaluate accuracies.

The average accuracy of the SVM and the ANN classifier is shown in Tables 1 and 2, respectively. Due to the indirect approach using decision tree, the derived decision trees have slightly lower accuracy. But the induced fuzzy logic-based classifiers show better than the trees. The results indicate that flexible decision boundaries have beneficial effects on the accuracy, which the fuzzy logic-based classifier have.

## 5. Conclusions

In weather forecasting process, it is important to analyze the radar data accurately. Among the non-precipitation echoes, the anomalous propagation echo is one of the representative non-precipitation echo. This paper proposes the fuzzy inference system with induced rules from the SVM and the ANN. The five different properties derived by clustering algorithm are applied as inputs of the classifiers. We can conclude that the fuzzy inference system from the SVM can detect the anomalous propagation echo well.

Further proposed work is to improve accuracy of detecting the anomalous propagation echo. The induced membership function parameters should be optimized for improving accuracy. Also, the other classification method could be applied such as artificial neural network, naive Bayesian classifier, and so forth. The empirical study is needed to select most appropriate algorithm for the anomalous propagation echo. Finally, the proposed system could be applied to other non-precipitation echoes.

## Acknowledgements

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2014R1A1A2056958) and was supported by Global PhD Fellowship Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2013-034596).

## Notes

**Conflict of Interest**

No potential conflict of interest relevant to this article was reported.

## References

## Biography

**Hansoo Lee** received the B.S. and M.S. degrees in Department of Electrical and Computer Engineering from Pusan National University, Busan, Korea, in 2010 and 2013, respectively, and is currently pursuing the Ph.D. degree in Electrical and Computer engineering at Pusan National University, Busan, Korea. His present interests include intelligent system and data mining.

E-mail: hansoo@pusan.ac.kr

**Sungshin Kim** received his B.S. and M.S. degrees in Electrical Engineering from Yonsei University, Korea, in 1984 and 1986, respectively, and his Ph.D. degree in Electrical Engineering from the Georgia Institute of Technology, USA, in 1996. He is currently a professor at the Department of Electrical and Computer Engineering, Pusan National University. His research interests include fuzzy logic controls, neuro fuzzy systems, neural networks, robotics, signal analysis, and intelligent systems.

E-mail: sskim@pusan.ac.kr