An Ensemble Method for Supervised Learning for Intrusion Detection and Network Forensics

Abstract

With the rapid expansion of Internet usage, cyber-attacks are becoming a point of interest for network security researchers. While companies are interested in providing their services over the Internet, their services do not enjoy appropriate security protection. Consequently, they are costing millions of dollars with network threats. Accordingly, current research suggest that ensemble methods may have advantages over individual base learners in intrusion detection problem. In this research work, we propose a new intrusion detection model combining an ensemble learning method and the expert system. In this study, first, we explore the three popular ensemble learning methods (namely boosting, bagging, and random subspace) with three individual base learners (random forest, Bayes net, and SMO) for network intrusion detection. Moreover, the NSL-KDD intrusion dataset was experimented to evaluate the effectiveness and efficiency of ensemble learning methods for intrusion detection. The performance of ensemble classifiers on the NSL-KDD dataset is tested in terms of average prediction accuracy. On the basis of experiments, empirical results show that general learning methods considerably improve the performance of base learners. Among the three ensemble methods, AdaBoostM1 boosting ensemble method based on the random forest base learner achieves the better comparative results. These results show that ensemble methods are appropriate for intrusion detection and further combined with the expert system for a knowledge-rich intrusion detection model.

1. Introduction

The rapid expansion of the Internet and Internet usage has been accompanied by different network threats targeting the services and applications rest on it by organizations. As stated in Ref. [1, 2], in today’s human life, the Internet is becoming highly prevailing and popular as well because of these countering cyber threats, mainly detecting network intrusions have been continued as an interesting discipline of research in the area of cyber security. Various security approaches have been deployed and implemented to prevent networks against security threats. Among all, firewall is a popular and most commonly deployed security appliance used as a basic packet filter in many organizations [3, 4]. However, in the ever-evolving network and technology introduction, the firewall is becoming insufficient to provide a required security level for the given network environment. Therefore, the rolling out of technologies and network spaces network threats make the network security arena more vulnerable. As a result, another security mechanism called intrusion detection has been introduced as a second line of defense in addition to the firewall to provide a better and more secure network environment, to eliminate these threats, and intrusion detection system is gaining the attention of the researcher and becoming a major concern of network security. Therefore, IDSs are put forth to detect and prevent breaching authenticity, availability, confidentiality, and integrity of enterprise systems from different attacks.

Specifically, IDSs are widely deployed in distributed enterprise systems, to analyze and monitor the outgoing and incoming network traffic and identify abnormal connections that may compromise the system, for signing network intrusion. Therefore, it analyses and monitors traffic continuously for occurrence of network intrusions [5]. Whenever it finds any kind of network intrusions disrupting the enterprise system, it automatically triggers an alarm to the network administrator indicating the occurrence of abnormal traffic in the network and admin will take fast counteraction to prevent malicious traffic before spreading and damaging the enterprise system.

As described in Ref. [6], based on the approach used for detection, IDSs are classified into two major classes; misuse detection. Misuse detection, legitimate network connections are identified from abnormal ones based on the known pattern or signature stored in the knowledge base, whereas anomaly detection distinguishes legitimate network connections from abnormal ones by identifying deviations from threshold created based on the normal behavior of the enterprise network. The hybrid intrusion detection, on the other hand, combines intrusion detections based on both abuse and abnormality. The misuse detection technique is good for identifying known network attacks with a low false-positive rate (FP), but it cannot detect novel attacks. Without the drawback of generating a high false-positive rate (FP), anomaly detection techniques can detect unknown network attacks better than misuse detection [7]. Hence, the key challenge in intrusion detection research is to come up with a model that can have a low false-positive rate and better prediction accuracy at the same time.

As stated in Ref. [7], new threats and vulnerabilities have evolved rapidly with the increasing sophistication of the attackers. IDS are highly required to detect and deal especially with unknown attacks, with the significant growth risk in the compromised critical infrastructures of enterprises.

As observed in a variety of literature like [2, 5, 6, 8, 9, 10, 11, 12], several paradigms have been researched and developed to come up with a better-performing IDSs. One of the paradigms among them is machine learning (ML) [7]. Several machine learning approaches have been researched to design IDS, including neural networks, support vector machines, fuzzy inference systems for both signatures based, and anomaly detection models. Recently, various types of raw data have been transformed into knowledge with the evolving advancement of knowledge discovery techniques. However, these individual techniques have their limitation on their associated classification results. As a result, machine learning researchers acknowledged combining different single classification techniques for solving the problem. Hence, to overcome these limitations hybrid models have been suggested over single supervised learning approaches. Therefore, this hybrid model consists of various single-base learners to improve the overall performance such as classification accuracy and false-positive rate. This hybrid model is called an ensemble model. By reducing the combined effect of errors and variance, the ensemble model can improve classification performance [13]. To address IDS problems, a new ensemble approach of machine learning has been introduced and widely used to alleviate intrusion detection problems, that is, the combinations of various base machine learning algorithms.

In recent years, these kinds of data mining and machine learning techniques have made wonderful progress [14]. However, the results provided by the data mining system and the action taken based on these results are still a significant gap. This gap greatly limits the efficiency of the overall process and applicability of machine learning results. Therefore, the gap that existed in previous works must be covered by human work, but it may be tedious to analyze manually and act accordingly. Not only this, the decision-making process about the action taken on the data mining result involves a chain of inferences, and these become the system bottleneck.

Moreover, network forensics is also a technique for gathering, archiving, and examining data about network activity in order to identify the cause of a security breach or other systemic information security problems. The basic goal of network forensics is to pinpoint all potential sources of security breaches and create systems for loss minimization through intrusion detection and prevention [15].

In order to identify the origin of security attacks or other problematic instances, network forensics involve the capture, recording, and analysis of network events. A branch of digital forensics known as network forensics deals with the observation and examination of computer network traffic for the goal of data collection, establishing credibility, or intrusion detection.

In this paper, we propose knowledge-rich learning for network intrusion detection systems to detect and prevent network attacks with high efficiency, low false-positive, and high accuracy. The proposed solution employs an ensemble classifier blended with an expert system, a machine learning system, to improve performance and accuracy of intrusion detection. On the other hand, in the machine learning system, ensemble learners automatically uncover knowledge from data, which needs expert intervention on the decisions made in each ensemble learning result. In this approach, machine learning discovered knowledge and expert knowledge can cooperate to fill the knowledge gap of the machine learning system, in the meantime, expert knowledge is employed to intervene in the results forwarded by the ensemble learning process. The expert system interprets and presents the resulting preventive recommendations in textual forms in an interactive way, thereby improving the overall functionality of the overall system. Machine learning techniques were applied to the KDD intrusion datasets containing 41 attributes and 40,580 records. The special of this work is presented as follows:

1. We propose a new approach that combines ensemble learning and expert system, which aims to improve performance and accuracy of intrusion detection.

2. An ensemble classifier using ensemble learning methods (boosting, bagging, and random subspace) with base learners (random forest, SMO, and Bayes net).

3. The expert knowledge is elicited from human experts and developed, which can be used to fill the knowledge gap of machine learning results.

The special contribution of this article is to prove the possibility of combining ensemble learning methods and expert systems and verifying the effectiveness of various ensemble learning methods for intrusion detection and network forensics. The remaining part of the article is organized as follows. Some related works about intrusion detection using ensemble methods are discussed in Section 2. Section 3 introduces the ensemble methods deployed in experiments. The experiments and results are analyzed in Sections 4 and 5. The combination of the ensemble methods and expert systems is presented in Section 6. Section 7 discusses conclusions and future research directions.

2. Related works

The ensemble learning methods are well suited for today’s IDS, and they are being used for implementing intrusion detection systems. In this section of the article, related works about IDS using ensemble methods are discussed in short. Authors have proposed a greedy boost hybrid approach for intrusion detection based on the KDD99 intrusion dataset [15]. In this study, greedy boost has been compared with C4.5 and AdaBoost based using KDD99. Experimental results show that the greedy boost hybrid approach outperforms C4.5 and AdaBoost. A new ensemble IDS has been presented by Amin Aburoman et al. [16] based on PSO, SVM, and KNN. The proposed approach is verified based on the KDD99 intrusion dataset. Based on the experiments PSO-SVM-KNN outperforms the weighted majority algorithm.

A random forest (RF) based intrusion detection system was proposed by the author [17]. The best features are selected using symmetrical uncertainty measures. Preprocessing techniques such as filling in missing values and discretization are applied before the classification phase. The experimental result suggests that the random forest is better in all four types of attack classification. As presented by Pham et al. [18], an ensemble classifier based on decision tree for implementing IDS was presented. The proposed work aimed to come up with IDS using various ensemble learning methods. An ensemble of four different decision trees was proposed for ensembles. From the experiment, we can conclude that the ensemble method improves the false positive (FP) and false-negative rate (FN) of the IDS. Ensemble classifiers based on decision trees and rule learners are formulated for IDS [19]. Authors stated that the proposed approach, which is formulated based on decision tree and rule learners, shows higher performance in terms of prediction accuracy compared to individual base learners. Ensemble learning methods using different individual learning methods are tested in Ref. [20]. In this study, the authors first investigated the performances of individual learning methods like SVM, ANN, and MARS. Then they observed the performance of the ensemble method based on SVM, ANN, and MARS. Experimental results show that ensembles have registered better performance than base learning methods.

A new ensemble classifier was proposed for the intrusion detection system and network forensic purposes [21]. The authors presented an ensemble based on Naïve Bayes and AD Tree using the NSL-KDD intrusion data set. IQR and discretization preprocessing techniques are used. Kumar and selvaumar [22] proposed a fuzzy-based ensemble classifier for IDS using DARPA and CAIDA data sets. The authors have used preprocessing techniques on DARPA and CAIDA data sets for experimental analysis.

Although numerous researchers have conducted various studies utilizing various ensemble approaches to enhance intrusion detection systems. However, much of the existing work on network intrusion detection uses only ensemble methods as a solution to the IDS problem. Furthermore, the lack of knowledge existed in the results of the ensemble methods; and the difficulty of understanding the result of these methods needs further investigation. This research proposes a new intrusion detection system that incorporates an ensemble approach and an expert system, which was inspired by the departure from earlier work on the intrusion detection system.

3. Ensemble learning methods

As mentioned in Ref. [23], comprehensive learning is a sort of machine learning approach where multiple basic learners are used to solve a single problem. As stated by Wang et.al, a single machine learning method tries to learn one thing from a given training data whereas the ensemble method tries to learn and combine multiple hypotheses on the training data for use. According to Wang et al., two necessary conditions should be considered; accuracy and diversity to achieve good ensemble. In practice, there are two categories of ensemble learning methods: bagging and boosting learning methods.

3.1 Bagging

As mentioned in Refs. [24, 25, 26], bagging is one of the earliest, simplest, and most intuitive ensemble learning algorithms to implement with a better performance. In bagging, diversity is gained using a bootstrapped replica of the training dataset. The data subsets are taken from the entire training data with replacement [25, 26, 27, 28]. Thus, various basic learners are trained on the training dataset.

The majority vote is used to combine different base learners in the bagging learning method. Variance can be reduced with this combining strategy of base learners. The pseudo-code and learning process for the bagging learning algorithm is shown in Figure 1 and Algorithm 1, respectively.

Figure 1.

Learning process of bagging.

Algorithm 1.

Pseudocode for bagging.

3.2 The random subspace

Random subspace method is one of an ensemble learning method [29, 30]. The training dataset is modified in random subspace as in the bagging ensemble learning method. The modification is not in the instance space instead it is in the feature space. The pseudocode and the working procedure for the random subspace learning method are presented in Figure 2 and Algorithm 2, respectively. Random subspace is benefited by constructing and aggregating the base learners using random subspaces. It uses the better base classifier in the random subspace instead of the original feature space when the dataset has redundant features.

Figure 2.

Learning process of random space.

Algorithm 2.

Pseudocode for random space.

3.3 Boosting

Boosting learning method is ensemble learning method that consists family of methods [24, 25, 26, 27, 28, 31]. In boosting learning methods, different base learners are created by continuously reweighting the training datasets. Based on this, a case that was previously misclassified and misclassified by a basic learner will then be given more weight in the subsequent training. For a predefined number of iterations, the base learner in the boosting algorithm repeatedly applies to the modified version of the training data. First of all, a uniform weight is assigned for all instances in the training dataset; and each boosting iteration is a base learner. Finally, a linear combination of multiple base learners with their performance is obtained from the boosting algorithm. AdaBoost is the most widely used boosting algorithm, however, there are several boosting algorithms that are available. Therefore, the AdaBoost algorithm is used for this study. The learning process and pseudocode for AdaBoost are presented in Figure 3 and Algorithm 3, respectively.

Figure 3.

Boosting working process.

Algorithm 3.

Pseudocode for Boosting.

4. Experimental design

4.1 Experimental dataset

The publicly available NSL-KDD intrusion dataset is collected to verify the efficiency and effectiveness of ensemble learning methods for intrusion detection. The intrusion dataset was collected from the well-known repository. It consisted of 49 million collections network connections labeled as normal and abnormal, and the abnormal connection of the data is also labeled with statuses (DoS, Probe, U2R, and R2L).

4.2 Performance evaluation

To evaluate the performance of the proposed approach, we adopted average accuracy and the false positive rate, which are widely considered established standard measures. Therfore, the average accuracy and false-positive rates can be used to evaluate the effectiveness of the proposed approach.

To reduce the variability effect of the training dataset, 10-fold cross-validation evaluation techniques were deployed on the NSL-KDD dataset. The average 10-fold cross-validation was treated as a test result, and the average results of various learning methods were recorded.

Ensemble classifiers are built using a combined result of multiple classifiers with their learning methods. From previous literature, we used three commonly deployed base learners namely; Bayes Net (BN), SMO, and random forest (RF) with their ensemble methods for our experiments.

Random forest is both classification and regression-supervised learning algorithm [8, 32]. However, mostly, it is used as a classification-supervised learning algorithm. According to Ref. [31], random forest trains and creates a decision tree using training data, receives a prediction from the tree, then uses voting to choose the best option. Hence, it is an ensemble method because it uses the best solution employing voting by decreasing overfitting. The following processes are working steps of the RF algorithm:

Step 1. First, RF starts by selecting the random sample from the training dataset.

Step 2. Next, RF constructs a decision tree for the selected sample and gives prediction results for every decision tree.

Step 3. Here, RF performs voting for every predicted result.

Step 4. Finally, choose the most voted prediction result as the final prediction result.

Figure 4 depicts the working procedure of random forest [31].

Figure 4.

Working of random forest.

Bayes net is a simple probabilistic method that follows Bayes’ theorem for classification problems [32, 33]. It has strong independence assumptions, simple to construct model, and not requiring any difficult iteration.

SVM is a contemporary data mining technique that has many applications with its best performance [34]. It has a strong theoretical base with its ability to model complex problems. Sequential minimal optimization (SMO) is one of the efficient techniques for training Support Vector Machine (SVM) now [35, 36, 37, 38]. In SMO, very large quadratic problems are segregated into several smallest possible quadratic problems. As a result, it is possible to avoid a time-consuming numerical quadratic problem optimization by analytically solving the quadratic problems; and allows handling a very large training set with less amount of memory requirement.

Table 1 shows the performance of the three base learners based on the selected measurement parameters. As for individual base learners, Table 1 shows that the average detection accuracy achieved is 99.87%, 98.4%, and 99.81% for SMO, Bayes net, and random forest, respectively. Similarly, the false-positive rate achieved is 0.1% for SMO; 0.3% for Bayes net, and 0% for random forest.

Performance metrics	SMO	Bayes net	Random forest
TP Rate	99.9%	98.4%	100%
FP rate	0.1%	0.3%	0%
Precision	99.9%	99.2%	100%
Recall	99.9%	98.4%	9%
F-Measure	99.9%	98.7%	9%
Accuracy	99.87%	98.4%	99.81%

Table 1.

Summary results of individual base learners based on various evaluation criteria.

The three most prominent ensemble learning methods, that is, bagging, boosting, and random subspace were selected and employed with the three individual base learners. A total of six experiments were conducted on the NSL-KDD intrusion dataset to evaluate and verify the effectiveness of base learners and ensemble learning methods for network intrusion detection.

Tables 3–5 reports on the effectiveness of ensemble approaches; for example, the average precision and false positive rate of AdaBoost, bagging, and random subspace using SMO, Bayes net, and random forest are shown, respectively. The average accuracy of ensemble methods on the provided measures is shown in Table 2.

	SMO	Bayes net	Random forest
Boosting (AdaBoostM1)	99.98%	99.94%	99.99%
Bagging	99.87%	98.56%	99.98%
Random subspace	99.68%	98.78%	99.98%

Table 2.

Average accuracy of three ensemble methods.

Performance metrics	AdaBoost_SMO	AdaBoost_BN	AdaBoost_RF
TP rate	100%	99.9%	100%
FP rate	0%	0%	0%
Precision	100%	99.9%	100%
Recall	100%	99.9%	100%
F-Measure	100%	99.9%	100%

Table 3.

AdaBoost ensemble learning result.

Performance metrics	Bagging_SMO	Bagging_BN	Bagging_RF
TP rate	99.9%	98.6%	100%
FP rate	0.1%	0.2%	0%
Precision	99.9%	99.2%	100%
Recall	99.9%	98.6%	100%
F-Measure	99.9%	98.8%	100%

Table 4.

Bagging ensemble learning result.

Performance metrics	RS_SMO	RS_BN	RS_RF
TP rate	99.7%	98.8%	100%
FP rate	0.4%	0.6%	0%
Precision	99.7%	99%	100%
Recall	99.7%	98.8%	100%
F-Measure	99.7%	98.9%	100%

Table 5.

Random subspace ensemble learning result.

5. Experimental results and analysis

The experiments were performed on a personal computer with a 2.90 GHz Intel(R) Core(TM) i7-7500U CPU and 8 GB RAM, using Windows 10 operating system. We used WEKA (Waikato Environment for Knowledge Analysis) version 3.7.0 data mining tool [39]. This open-source tool, which is WEKA, consists of a collection of various machine learning algorithms used for solving data mining-related problems.

In this study, we have compared the performances of three different classification methods, including Bayes net, SMO, and random forest, and their equivalent ensemble methods of bagging, random subspace, and boosting. Among these methods, the Bayes net algorithm, SMO algorithm, and random forest algorithms were implemented by using the Naive Bayes module, SMO module, and J48 module of the module of WEKA, respectively. The AdaBoost M1 module and the bagging module of WEKA were employed to implement corresponding algorithms.

5.1 Experimental results and discussions from the base learners’ perspective

Figure 5 summarizes the experimental results of base learners on the NSL-KDD intrusion dataset for intrusion detection. The false-positive rate and average accuracy of various classification methods on the NSL-KDD datasets from the base learner are shown in Figure 2. Firstly, as shown in Figure 2, random forest has achieved the best average accuracy, that is, 99.81% and 0% false-positive rate; SMO has the best average accuracy, that is, 99.87% and 0.1% false-positive rate; and Bayes net has achieved an average accuracy of 98.4% and 0.3% false-positive rate using all features sets. These results indicate that among three base classification methods have a different comparative advantages for network intrusion detection. This experimental result is more likely consistent with previous research [40, 41] In network intrusion detection domain with the NSL-KDD training set, random forest can be used as individual classifier when the data is linear and with a large number of attributes [42].

Figure 5.

Summary of three base learners.

5.2 Experimental result and discussion from ensemble learning methods perspectives

As we have seen in all Figures 5–8 and Tables 1–5, for all of the three groups of the ensemble methods on the given training set, ensemble methods have better comparative results with a slight difference than individual base learners. As a result, we can conclude that ensemble learning methods are suitable for network intrusion detection. Besides this, some variations were observed during the experiments.

Figure 6.

Experimental result of boosting with three base learners.

Figure 7.

Experimental result of bagging with three base learners.

Figure 8.

Experimental result of random space with three base learners.

Figure 9 shows the comparative result of the ensemble methods with the three base learners based on the detection accuracy. In general, empirical results have shown that boosting the ensemble learning method achieves 99.98%, 99.94%, and 99.99% for SMO, Bayes net, and random forests, respectively. Similarly, experimental results using the bagging ensemble learning method achieve 99.87%, 98.56%, and 99.98% with its base learners namely; SMO, Bayes net, and random forest, respectively. The third empirical result is with the random subspace ensemble learning method. This ensemble learning method also achieves 99.68%, 98.78%, and 99.98% with base learners SMO, Bayes net, and random forest, respectively. Also, some interesting variations were observed in the performed experiments. Among the three experimental results, bagging had slightly lower accuracy when it ensemble Bayes net as the base classifier. The second one is that the ensemble method using random forest as a base classifier has better comparative experimental results. The third one is that random subspace has lower comparative experimental results when SMO and random forest are used as the base classifier, but random subspace has scored the worst experimental result when Bayes net is used as the base learner. These empirical results are more likely consistent with the previous research like [3, 8]. These results clarify why most prior research used decision tree algorithms as base learners in ensemble learning methods.

Figure 9.

A summary result of three ensemble methods.

6. Developing expert system

Extracting, structuring, and organizing knowledge from human experts and other sources—such as books, databases, the Internet, research papers, documents, and one’s own experience—then adding it to the knowledge base [22]. Interviewing subject-matter experts and analyzing pertinent papers helped to gather the expertise for this study, which was then refined. Conceptual modeling is used to model the knowledge once it has been obtained. To fully comprehend the decision-making process during network intrusion prevention, domain experts’ and manuals’ information is modeled using decision trees for this study. After knowledge has been gathered and modeled, the next step is to represent it using the proper format. For this study, reasoning based on rules is used to describe and express knowledge, and it is in line with the decision tree for knowledge modeling.

After knowledge has been gathered and modeled, the next step is to represent it using the proper format. For this study, reasoning based on rules is used to describe and express knowledge, and it is in line with the decision tree for knowledge modeling. This is one of the most often utilized methods for representing information as IF condition-THEN action pairings. In order to employ the information as a course of action for the identified intrusion, the knowledge is finally added to the knowledge base.

7. Incorporating ensemble classifier result with expert system

In this study, we have proposed an approach that can combine both ensemble classifiers with the expert system to make our intrusion detection effective and efficient. The proposed knowledge-rich intrusion detection technique consists of both ensemble and expert systems. The knowledge gap that existed in the ensemble classifier has to be filled with knowledge database elicited and developed by human experts using a knowledge engineering approach. Therefore, an expert system that is acquired and developed by domain experts should be incorporated with the ensemble classifier results to improve the interpretability and the action to be taken by the security experts in the enterprise network. As a result, the proposed intrusion detection model can predict the network intrusions using ensemble classifiers and interpret the ensemble results and forward the appropriate countermeasure for the ensemble classifier result. The combined result of the two sub-modules has come true with the help of different tools and techniques. For this study, Weka data mining tools consist of several machine learning algorithms, Prolog declarative logic programming language, Java Prolog interface library, and SWI-WEKA package as an interface to integrate ensemble classifier result and the expert system implemented in Prolog. Finally, the combined result of the ensemble classifier and expert system can predict the intrusion in the training set and provide the countermeasure for the predicted result.

8. Conclusion and future works

The rise of network intrusion has driven an interest in network intrusion detection. Promptly and correctly identifying abnormal network traffic from normal traffic has become a serious issue for individuals and companies’ security experts. In this work, we evaluated both individual base classifiers and ensemble learning methods (boosting, bagging, and random subspace) for performing intrusion detection tasks. Publicly available NSL-KDD datasets were used to evaluate the effectiveness and efficiency of ensemble learning methods for intrusion detection. Experimental results presented that ensemble learning methods scored better results than individual base classifiers. Among the three ensemble methods, boosting (AdaBoostM1) random forest had the best prediction accuracy. All these empirical results show that ensemble learning methods are an appropriate method for network intrusion detection. There can be plenty of future research directions recommended in this study. Firstly, as the intrusion datasets are sometimes class imbalanced, the balanced dataset should be prepared to come up with a sound conclusion for this work. Secondly, the number of attributes used can be an important factor for predicting network intrusions; hence, we used all attributes in this research. Thirdly, as ensemble learning methods for intrusion detection need to fill the gap of knowledge in interpreting the result of prediction and the countermeasure to be taken should be further explored to address these gaps; the hidden knowledge learned by ensemble methods is difficult to understand for human experts. Therefore, developing a self-learning knowledge database for improving the usability and interpretability of ensemble results can be an important further research direction.