Network Intrusion Detection using Machine Learning & Deep Learning

Overview

This project implements a Network Intrusion Detection System (NIDS) using the UNSW-NB15 dataset. It explores both traditional Machine Learning and Deep Learning techniques to detect malicious network traffic and classify different attack categories.

The notebook includes:

• Exploratory Data Analysis (EDA)
• Data preprocessing and feature engineering
• Binary intrusion detection (Normal vs Attack)
• Multi-class attack classification
• Random Forest baseline model
• LSTM-based Deep Learning model
• Enhanced LSTM with Attention Mechanism
• Performance evaluation and comparison

---

Dataset

Dataset: UNSW-NB15

The dataset contains modern network traffic with both normal and malicious activities.

Target variables:

• label

  • 0 → Normal
  • 1 → Attack

• attack_cat

  • Analysis
  • Backdoor
  • DoS
  • Exploits
  • Fuzzers
  • Generic
  • Reconnaissance
  • Shellcode
  • Worms
  • Normal

---

Project Workflow

1. Data Loading

• Extract dataset from ZIP archive
• Load training and testing Parquet files
• Merge datasets
• Inspect data structure

---

2. Exploratory Data Analysis

The notebook performs:

• Dataset inspection
• Missing value checking
• Feature exploration
• Protocol analysis
• Service analysis
• Attack category distribution
• Binary class distribution

---

3. Data Preprocessing

Preprocessing steps include:

• Handling categorical features
• Renaming suspicious service values
• Label Encoding
• Feature Scaling using MinMaxScaler
• Train/Test split
• SMOTE balancing for imbalanced classes

---

4. Machine Learning Model

Random Forest

Used for:

• Binary Classification
• Multi-class Classification

Evaluation metrics:

• Accuracy
• Precision
• Recall
• F1-score
• Classification Report

---

5. Deep Learning Models

Binary LSTM

Architecture:

• LSTM Layer
• Dropout
• Dense (Sigmoid)

Task:

• Detect whether traffic is Normal or Attack.

---

Multi-class LSTM

Architecture:

• LSTM
• Dropout
• Dense (Softmax)

Task:

• Predict attack category.

---

LSTM with Attention Mechanism

This model introduces a self-attention layer on top of the LSTM outputs.

Advantages:

• Better feature importance learning
• Improved interpretability
• Higher detection capability
• Reduced overfitting

---

Evaluation Metrics

Models are evaluated using:

• Accuracy
• Precision
• Recall
• F1-score
• RMSE
• MAE
• Training Accuracy
• Validation Accuracy
• Overfitting Gap
• Inference Time

---

Visualizations

The notebook includes:

• Dataset exploration
• Target distribution
• Training Accuracy curves
• Validation Accuracy curves
• Training Loss curves
• Confusion Matrices
• Actual vs Predicted plots
• Attention model visualization
• Model comparison tables

---

Technologies Used

• Python
• Pandas
• NumPy
• Matplotlib
• Scikit-learn
• TensorFlow / Keras
• Imbalanced-learn (SMOTE)

---

Project Structure

Network Intrusion Detection.ipynb
README.md
requirements.txt

---

Installation

Clone the repository:

git clone https://github.com/yourusername/network-intrusion-detection.git

Install dependencies:

pip install -r requirements.txt

Run Jupyter Notebook:

jupyter notebook

Open:

Network Intrusion Detection.ipynb

---

Models Compared

Model	Task
Random Forest	Binary Classification
Random Forest	Multi-class Classification
LSTM	Binary Classification
LSTM	Multi-class Classification
LSTM + Attention	Binary Classification

---

Future Improvements

• Transformer-based intrusion detection
• Explainable AI (XAI)
• Real-time packet monitoring
• Hyperparameter optimization
• Model deployment using FastAPI
• Docker containerization
• Streaming detection using Kafka

---

Learning Outcomes

This project demonstrates:

• Network traffic preprocessing
• Intrusion detection using Machine Learning
• Deep Learning with LSTM
• Attention mechanisms for cybersecurity
• Performance comparison between classical ML and Deep Learning
• Data visualization and model evaluation

---

Author

Hamid

Electrical Engineer | AI & Machine Learning Enthusiast | Cybersecurity Research

---