A Host-based Anomaly Detection Approach by Representing System Calls as States of Kernel Modules


  1. Murtaza, S.S.
  2. Hmou-Lhadj, A.
  3. Couture, M.
Corporate Authors
Defence Research and Development Canada, Valcartier Research Centre, Quebec QC (CAN);Concordia Univ, Montreal QUE (CAN) Dept of Electrical and Computer Engineering
Despite over two decades of research, high false alarm rates, large trace sizes and high processing times remain among the key issues in host-based anomaly intrusion detection systems. In an attempt to reduce the false alarm rate and processing time while increasing the detection rate, this paper presents a novel anomaly detection technique based on semantic interactions of system calls. The key concept is to represent system calls as states of kernel modules, analyze the state interactions, and identify anomalies by comparing the probabilities of occurrences of states in normal and anomalous traces. In addition, the proposed technique allows a visual understanding of system behaviour, and hence a more informed decision making. We evaluated this technique on Linux based programs of UNM datasets and a new modern Firefox dataset. We created the Firefox dataset on Linux using contemporary test suites and hacking techniques. The results show that our technique yields fewer false alarms and can handle large traces with smaller (or comparable) processing times compared against the existing techniques for the host based anomaly intrusion detection systems.
anomaly detection;cyber-threat;execution trace;system call;kernel module
Report Number
DRDC-RDDC-2016-N021 — External Literature
Date of publication
04 Oct 2016
Number of Pages
Electronic Document(PDF)

Permanent link

Document 1 of 1

Date modified: