Policy Gradient Deep Reinforcement Learning Based Vampire Attack Detection

Description

##### Code for the paper: [Juneja, V., Dinkar, S. K., & Gupta, D. V. (2022). An anomalous co‐operative trust & PG‐DRL based vampire attack detection & routing. Concurrency and Computation: Practice and Experience, 34(3), e6557.](https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.6557)
**Disclaimer: Results Images are subject to copyright of the paper itself.**
## Introduction
Wireless Sensor Networks (WSN) have become an essential part of the modern, smart, and technological world. These include areas like smart cities, industrial automation, agriculture, healthcare, etc. These networks consist of small sensor nodes that transfer information by communicating with each other.

However, malicious attacks can threaten these nodes, damaging the network structure and increasing energy consumption for data transfer. Vampire Attack is a type of this attack, which drains the battery of the nodes by making the data travel longer routes.

Here a PG-DRL (Policy Gradient Deep Reinforcement Learning) is proposed for vampire attack detection, and mitigation by rerouting the sensor node transfer path to avoid malicious nodes. Fig.1 depicts a category of Vampire attack called Carousel attack. It can be seen from the figure that the frequency of node hops increases significantly when WSN is under attack.
{.alignnone}

## Problem Statement
To identify these malicious attacks trust thresholds are used by the traditional methods which typically lack in effectiveness because these threshold values vary for each different network and require manual tuning based on knowledge. Furthermore, safe routing in WSNs usually demands the whole route to be shared across nodes, therefore increasing the danger of route manipulation by malicious nodes and hence wasting more energy.

## Contributions
1. **Novel Approach of Detection for Vampire Attacks**:
* Since vampire attacks quickly deplete the energy of sensor nodes in a network, they are very damaging.
* The authors devised a novel approach using cooperative trust values and spectral clustering that finds and separates hostile nodes, hence enhancing network security and performance.

2. **Routing Energy-Efficiently to Reduce Vampire Attacks**:
* A reinforcement learning-based routing algorithm is developed that chooses the next hop for data transfer in real time to stop vampire attacks.
* This concept guarantees the network stays efficient even when under attack by using the minimum of energy for data transfer. Effective across numerous network situations, the model is versatile, can adapt to changes, and does not need specified paths.

## Methodology
The methodology involves four steps which are,
1. **Calculating cooperative trust scores to identify reliable nodes for data transmission**:
In this paper, each sensor node has one-to-many connections in the network and a cooperative trust score is calculated based on the energy levels of
its neighboring nodes. The eigenvalues of the cooperative trust matrix open the door to identifying the malicious vampire nodes.
2. **Detecting the malicious vampire nodes by Gaussian Mixture Model (GMM)**:
Using relative eigenvalues of the cooperative trust matrix, a GMM model with Expectation- maximization (EM) clustering is used to cluster the nodes
into benign and attack categories.
3. **Training of the PG-DRL model**:
* Policy Gradient Deep Reinforcement Learning (PG-DRL) is an advanced method that enables a system to learn how to make decisions by continuously
improving its strategy through trial and error. PG-DRL trains an agent to maximize rewards by avoiding vampire nodes. The algorithm involves
initializing the actor-critic method where the critic network optimizes the action or state value while the actor network works on the update policy
parameters guided by the critic network. This results in a final greater cumulative reward i.e., the best tuning parameters.
* The PG-DRL algorithm is implemented with specific parameters and tested over multiple episodes to ensure the agent learns to avoid vampire nodes and
select energy-efficient routes.
4. **And a rerouting model based on the PG policy**:
The rerouting process for the efficient node route selection in WSN after detecting the Vampire nodes is:
* If the system (agent) selects a good next node that isn’t a vampire node and still has energy left, it earns a reward. This reward is calculated
based on how much energy the chosen node has and how far away it is. The closer the node and the more energy it has, the better the reward.
* If the system accidentally chooses a vampire node, it gets a big penalty of -100. This is to discourage it from making that mistake again.
* If the system reaches a point where it can’t make any more moves, either because there’s no energy left or no new nodes to select, it gets a small penalty of -1.

## Implementation discussion with code snippets
First, the trust score calculation is done with the help of energy consumption and distribution in the network.

“`matlab
conn = nodes{1}.inrange;

E_temp=zeros(size(conn)); E_temp_norm = zeros(size(conn));
for rr=1:size(conn,1)
distt=[];temp=[];Ee=[];
distt=nodes{1}.distance(1,find(conn(rr,:)));
for ii=1:numel(find(conn(rr,:)))
Ee(ii)=(alpha1*datarate(ff)*pktsize*8)+(alpha2*datarate(ff)*pktsize*8)*(distt(ii))^alpha;
end
E_temp(rr,find(conn(rr,:))) = E(find(conn(rr,:)));
E_temp(rr,find(conn(rr,:))) = E(find(conn(rr,:)))-Ee;
E_temp_norm(rr,:) = E_temp(rr,:)./sum(E_temp(rr,:));
E_temp_norm(rr,E_temp_norm(rr,:) 30;
this.IsDone = IsDone;

“`
* `nxtNbr` is used to decide what packet should be routed to based on the action `Action` chosen by the PG-DRL agent.
* `E`computes the energy needed to send the packet from the current node to the next node `nxtNbr` based on their distance. The residual energy of the selected node is then updated by subtracting this energy from its current value via `rsE(nxtNbr)`.
* The updated node is then saved as the current using `this.state=nxtNbr`
* Finally the `IsDone` function checks whether the current routing episode should terminate.
The reinforcement reward function is given by
“`matlab
function Reward = getReward(this,nxtNbr)
if ~this.IsDone && (~ismember(nxtNbr,this.VampireNid))
Reward = (this.resE(nxtNbr)/this.nodes.distance(this.crntNode,nxtNbr)^2);
elseif ~this.IsDone && (ismember(nxtNbr,this.VampireNid))
Reward=-100;
else
Reward = -1;
end
end
“`
* The `Reward` function trains the PG-DRL agent by giving it a positive reward for the correct routing path and reinforces this behavior. And heavily penalizes if the model selects the Vampire node `this.VampireNid`.
## Results
A WSN environment with a predefined number of nodes is established, using Dynamic Source Routing (DSR) as the baseline protocol. As it is explained in the methodology the vampire nodes are detected by the energy consumption of these malicious nodes compared to benign nodes. The figure below shows energy consumption of the two types of vampire nodes: Carousel and Stretch attacks.
{.alignnone}

* It can be seen from the figure that stretch attacks consume more energy due to higher hop counts.
* The current algorithm is evaluated in a simulated WSN scenario with 80% vampire nodes in the network. The figure below shows the detected vampire nodes and true vampire nodes labeled in the DSR path for both carousel and stretch attacks.
{.alignnone}

* The proposed detection algorithm demonstrated 100% accuracy in detecting vampire nodes under stretch and carousel attacks in some trials, though this accuracy decreases with higher densities of vampire nodes.
* The statistical analysis shows that detection accuracy remains high, especially in networks with fewer vampire nodes.
* After the detection of these vampire nodes, it is necessary to find a secure route for data transfer among the nodes with the least energy consumption. The PG-DRL algorithm is used to secure routing. The training results show that the PG-DRL algorithm effectively learns to avoid vampire nodes over time, improving the network’s average reward and ensuring secure routing.
* The PG-DRL routing is compared to DSR routing under both benign and attack scenarios, the figure below shows that PG-DRL can reduce the number of hops and energy consumption under attack conditions. The network lifetime is also improved by approximately 3% when using PG-DRL compared to DSR.
{.alignnone}

* The proposed detection algorithm outperforms existing methods like Theil Index and fuzzy trust score-based approaches, with higher detection accuracy even under high vampire node densities. The PG-DRL algorithm provides secure and efficient routing, leading to an increased network lifetime.

## Conclusion
The proposed two-fold approach combining cooperative trust calculation and PG-DRL for vampire attack detection and node path rerouting shows significant improvements in attack detection accuracy and creating the shortest and most secure routing to avoid the detected Vampire nodes.

By avoiding routes with Vampire nodes and selecting paths that use the least amount of energy, the proposed approach reduces energy consumption. This results in increased operational time of the nodes and increased network lifetime.

## References
1. [Juneja, V., Dinkar, S. K., & Gupta, D. V. (2022). An anomalous co‐operative trust & PG‐DRL based vampire attack detection & routing. Concurrency and Computation: Practice and Experience, 34(3), e6557.](https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.6557)
2. [Pu, Cong, Jacqueline Brown, and Logan Carpenter. “A Theil Index-Based Countermeasure Against Advanced Vampire Attack in Internet of Things.” In 2020 IEEE 21st International Conference on High Performance Switching and Routing (HPSR), pp. 1-6. IEEE, 2020.](https://congpu.github.io/document/paper/ieee_hpsr_2020.pdf)
3. [Srikaanth, P. Balaji, and V. Nagarajan. “A Fuzzy Trust Relationship Perspective-Based Prevention Mechanism for Vampire Attack in MANETs.” Wireless Personal Communications 101, no. 1 (2018): 339357.](https://link.springer.com/article/10.1007/s11277-018-5691-8)
4. [Isaac Sajan, R., and J. Jasper. “Trust‐based secure routing and the prevention of vampire attack in wireless ad hoc sensor network.” International Journal of Communication Systems 33, no. 8 (2020): e4341.](https://www.scihorizon.com/cdn/pdf/1611988598_262d174883336e0f3813.pdf)
5. [Srikaanth, P. Balaji, and V. Nagarajan. “Semi-Markov chain-based grey prediction-based mitigation scheme for vampire attacks in MANETs.” Cluster Computing 22, no. 6 (2019): 15541-15549.](https://www.academia.edu/download/87794256/s10586-018-2698-620220620-1-1xscfqy.pdf)

Dr. Abhishek Gupta

ScholarsColab.com is an innovative and first of its kind platform created by Vidhilekha Soft Solutions Pvt Ltd, a Startup recognized by the Department For Promotion Of Industry And Internal Trade, Ministry of Commerce and Industry, Government of India recognised innovative research startup.