Skip to content

Commit

Permalink
🔀 Merge Pull Request: Remove Redis and socket connection bugs (#18)
Browse files Browse the repository at this point in the history
  • Loading branch information
AleksaMCode authored Dec 22, 2023
2 parents 6c6d1a0 + 2113fe5 commit bc2023a
Show file tree
Hide file tree
Showing 12 changed files with 152 additions and 49 deletions.
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ class="center"
> All content in this project is intended for security research purpose only.
> [!NOTE]
> - <p align="justify">I'm currently writting a tehnical paper which will thoroughly explain theory that makes this tool possible. It will be published by the end of this year.</p>
> - <p align="justify">I'm currently writing a tehnical paper which will thoroughly explain what that makes this tool possible. It should be published in the coming weeks.</p>
> - <p align="justify">To monitor the ongoing work on the PNLS, see <a href="https://github.com/users/AleksaMCode/projects/1">project's board</a>.</p>
## Table of contents
Expand All @@ -34,7 +34,7 @@ class="center"
- [Using Docker](#using-docker)
- [Using Prebuild Docker Image](#using-prebuild-docker-image)
- [Without Docker](#without-docker)
- [Probe Request](#probe-request)
- [Probe Requests](#probe-requests)
- [Architecture](#architecture)
- [Why Asynchronous Server Gateway Interface?](#why-asynchronous-server-gateway-interface)
- [Why WebSockets?](#why-websockets)
Expand Down Expand Up @@ -62,7 +62,7 @@ class="center"
- Needed in order to use monitoring mode and [aircrack-ng](https://github.com/aircrack-ng/aircrack-ng) tool. You can download Kali Linux ARM image from [here](https://www.kali.org/get-kali/#kali-arm).
- Alternatively, you could use another OS, but you will need to patch[^3] the kernel using the [nexmon](https://github.com/seemoo-lab/nexmon)[^4] or use a wireless adapter that supports monitoring mode. Here is a [link](https://elinux.org/RPi_USB_Wi-Fi_Adapters) for supported USB adapters by Raspberry Pi.
- You will also have to install the *aircrack-ng* tool, as it only comes preinstalled on the Kali Linux.
- Start your network interface in a monitoring mode with: `airmon-ng start wlan0` [2].
- Start your network interface in a monitoring mode with: `sudo airmon-ng start wlan0` [2].

> [!NOTE]
> <p align="justify">The Kali image uses <a href="https://re4son-kernel.com/">Re4son</a>'s kernel, which includes the drivers for external Wi-Fi cards and the nexmon firmware for the built-in wireless card on the RPi 3 and 4 [3].</p>
Expand Down Expand Up @@ -122,7 +122,7 @@ Here is a screenshot when I ran everything "manually":

- Top Left: Redis server
- Top Right: ASGI server
- Bottom Left: sniffer service
- Bottom Left: Sniffer service
- Bottom Right: React server

<p align="center">
Expand All @@ -136,7 +136,7 @@ class="center"
</p>
</p>

## Probe Request
## Probe Requests

<p align="justify">Probe Requests are management 802.11 frames which are used to connect devices to the previously associated wireless Access Points (AP). Whenever a device has enabled Wi-Fi, but it isn't connected to a network, it is periodically sending a burst of Probe Requests containing SSIDs from it's PNL. These frames are sent unencrypted, and anyone who is Radio Frequency (RF) monitoring can capture and read them. Probes are sent to the broadcast DA address (<code>ff:ff:ff:ff:ff:ff</code>). Once they are sent, the device starts the Probe Timer. At the end of the timer, the device processes the received answer. If the device hasn't received an answer, it will go to the next channel and repeat the process. There are two types of Probe Requests:</p>
<ul>
Expand Down Expand Up @@ -166,13 +166,13 @@ class="center"

### Why Asynchronous Server Gateway Interface?

<p align="justify">Asynchronous Server Gateway Interface (ASGI) provides standardizes interface between async-capable Python web servers and services [4]. The ASGI was chosen due to the project's need for a long-lived WebSocket connection in order to facilitate async communications between different clients. In addition, it also allows for utilization of the background coroutines during API calls. The PNLS uses <a href="https://github.com/encode/uvicorn">uvicorn</a> implementation for Python in order to use the ASGI web server.</p>
<p align="justify">Asynchronous Server Gateway Interface (ASGI) provides standardized interface between async-capable Python web servers and services [4]. The ASGI was chosen due to the project's need for a long-lived WebSocket connection in order to facilitate async communications between different clients. In addition, it also allows for utilization of the background coroutines during API calls. The PNLS uses <a href="https://github.com/encode/uvicorn">uvicorn</a> implementation for Python in order to use the ASGI web server.</p>

### Why WebSockets?
<p align="justify">Through utilization of WebSocket communication protocol, we are able to facilitate full-duplex, two-way communication. While this project doesn't have the need for two-way communication, it does have a need for real-time interaction between the system components. This way, the sniffed data will be available to the end-user as soon as they are captured.</p>

### Pub-Sub Model
<p align="justify">Project's MOM is realized through the Message Broker using Redis. In the publish-subscribe (pub-sub) model, <i>sniffer</i> is responsible for producing messages, while the web application (subscriber) registers for the specific Topic (Redis channel). When a sniffer sends a message to a Topic, it is distributed to all subscribed consumers, allowing for the asynchronous and scalable communication. PNLS uses lightweight messaging protocol <i>Redis Pub/Sub</i> for message broadcasting in order to propagate short-lived messages with low latency and large throughput [5][6]. In this way, overheads of encoding data structures in a form that can be written to a disk have been avoided and in doing this solution will have potential better performance [7]. Figure below displays the simplified system activity through the event-driven workflow.</p>
<p align="justify">Project's MOM is realized through the Message Broker using Redis. In the publish-subscribe (pub-sub) model, <i>sniffer</i> is responsible for producing messages, while the web application (subscriber) registers for the specific Topic (Redis channel). When a <i>sniffer</i> sends a message to a Topic, it is distributed to all subscribed consumers, allowing for the asynchronous and scalable communication. PNLS uses lightweight messaging protocol <i>Redis Pub/Sub</i> for message broadcasting in order to propagate short-lived messages with low latency and large throughput [5][6]. In this way, overheads of encoding data structures in a form that can be written to a disk have been avoided. In doing so, this solution will have potentially better performance [7]. Figure below displays the simplified system activity through the event-driven workflow.</p>

<p align="center">
<img
Expand All @@ -191,7 +191,7 @@ class="center"
## Acronyms
<table>
<tr> <td>PNL</td> <td>Preferred Network List</td> </tr>
<tr> <td>PNLS</td> <td>Preferred Network List Sniffer</td> </tr>
<tr> <td>PNLS</td> <td>Preferred Network List Sniffer</td> </tr>
<tr> <td>SSID</td> <td>Service Set Identifier</td> </tr>
<tr> <td>UI</td> <td>User Interface</td> </tr>
<tr> <td>RPi</td> <td>Raspberry Pi</td> </tr>
Expand Down
2 changes: 1 addition & 1 deletion resources/pnls-system-diagram.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
15 changes: 13 additions & 2 deletions sniffer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,15 @@
## Run

After cloning the project move to the backend root folder.
First start the `Redis` server by running the following command in the terminal:

```shell
redis-server
```

This will start the Redis server in the default configuration at port 6379.

After, clone the project and move to the backend root folder.

```shell
cd Preferred-Network-List-Sniffer/sniffer
Expand All @@ -28,7 +36,10 @@ pip3 install -r requirements.txt
```shell
python3 pnls.py
```
Serve with hot reload will be available on `localhost:3001/`.
Serve without hot reload will be available on `localhost:3001/`.

> [!WARNING]
> In order to have ASGI server running, you first need to run a Redis server, otherwise the server will not start.
To start the sniffer microservice, run the following:

Expand Down
1 change: 0 additions & 1 deletion sniffer/logger.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ def create_logger(file_name: str):
)


# TODO: Rename methods.
async def log_info_async(message: str):
logger.info(message)

Expand Down
77 changes: 60 additions & 17 deletions sniffer/message_broker/websocket_broker.py
Original file line number Diff line number Diff line change
@@ -1,49 +1,92 @@
import asyncio
import json

import redis
from fastapi import WebSocket
from redis import asyncio as aioredis
from starlette.websockets import WebSocketState

from message_broker.message_broker import MessageBroker
from settings import CHANNEL_ID


class WebSocketBroker:
def __init__(self):
self.channel_id = None
self.sockets: list = []
def __init__(self, channel_id: str):
self.channel_id = channel_id
self.sockets: list[WebSocket] = []
self.pubsub_client = MessageBroker()

async def add_user_to_channel(self, channel_id: str, websocket: WebSocket) -> None:
async def accept(self) -> None:
"""
Adds a user's WebSocket connection to a channel.
:param channel_id: Channel ID to add user to.
:param websocket: WebSocket connection object.
Connects to Redis server and establish channel.
"""
await websocket.accept()
self.sockets.append(websocket)

if self.channel_id is None:
self.channel_id = channel_id
if not self.sockets:
await self.pubsub_client.connect()
ps_subscriber = await self.pubsub_client.subscribe(channel_id)
ps_subscriber = await self.pubsub_client.subscribe(self.channel_id)
asyncio.create_task(self._pubsub_data_reader(ps_subscriber))

async def add_client_to_channel(self, websocket: WebSocket) -> None:
"""
Adds a client's WebSocket connection to a channel.
:param websocket: WebSocket connection object.
"""
self.sockets.append(websocket)

async def broadcast_to_channel(self, channel_id: str, message: str) -> None:
"""
Broadcasts a message to all connected WebSockets in a channel.
:param channel_id: Channel ID to publish to.
:param message: Message to be broadcast.
"""
await self.pubsub_client.publish(channel_id, message)
if self.sockets:
await self.pubsub_client.publish(channel_id, message)

async def _pubsub_data_reader(self, ps_subscriber: aioredis.Redis):
"""
Reads and broadcasts messages received from Redis PubSub.
:param ps_subscriber: PubSub object for the subscribed channel.
"""
while True:
message = await ps_subscriber.get_message(ignore_subscribe_messages=True)
message = None
try:
message = await ps_subscriber.get_message(
ignore_subscribe_messages=True
)
except redis.exceptions.ConnectionError:
# TODO: Add logging.
# TODO: Replace return handle when Redis is closed when server is running with a better
# approach, perhaps a back-off algorithm could be added.
return
except Exception as e:
# TODO: Implement handle of other Exceptions.
pass

if message:
for socket in self.sockets:
data = message["data"].decode("utf-8")
await socket.send_json(json.loads(data))
if (
socket.application_state == WebSocketState.CONNECTED
and socket.client_state == WebSocketState.CONNECTED
):
data = message["data"].decode("utf-8")
await socket.send_json(json.loads(data))

async def remove_client_from_channel(self, websocket: WebSocket) -> None:
"""
Removes a client's WebSocket connection from a channel.
:param websocket: WebSocket connection object.
"""
self.sockets.remove(websocket)

async def close_sockets(self):
"""
Closes client sockets.
"""
for socket in self.sockets:
if (
socket.application_state == WebSocketState.CONNECTED
and socket.client_state == WebSocketState.CONNECTED
):
await socket.close()


socket_broker = WebSocketBroker(CHANNEL_ID)
2 changes: 1 addition & 1 deletion sniffer/parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def parse_ip_packet(packet):
"ssid": ssid,
"timestamp": datetime.utcfromtimestamp(
float(packet.time)
).strftime(TIMESTAMP_FORMAT),
).strftime(TIMESTAMP_FORMAT)[:-3],
}
)
)
Expand Down
5 changes: 3 additions & 2 deletions sniffer/pnls.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,13 @@
from logger import create_logger
from services import pub_sub
from settings import SERVER
from utils.pnls_util import lifespan

origins = ["*"]

# Create a logger.
create_logger(f"{Path(__file__).stem}.log")
app = FastAPI()
app = FastAPI(lifespan=lifespan)

# Add router.
app.include_router(pub_sub.router)
Expand All @@ -31,5 +32,5 @@
f"{Path(__file__).stem}:app",
host=SERVER["localhost"],
port=SERVER["port"],
reload=True,
reload=False,
)
43 changes: 32 additions & 11 deletions sniffer/services/pub_sub.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import asyncio
import json

import redis
from fastapi import APIRouter, WebSocket, WebSocketDisconnect, WebSocketException
from starlette import status

Expand All @@ -10,11 +11,11 @@
log_info_async,
log_warning_async,
)
from message_broker.websocket_broker import WebSocketBroker
from message_broker.websocket_broker import socket_broker
from settings import CHANNEL_ID
from utils.pnls_util import shutdown

router = APIRouter(prefix="/ws")
socket_manager = WebSocketBroker()


@router.websocket("/pub/{channel_id}")
Expand All @@ -29,26 +30,34 @@ async def publish(websocket: WebSocket, channel_id: str):
)
raise WebSocketException(code=status.WS_1003_UNSUPPORTED_DATA)

await websocket.accept()
asyncio.create_task(
log_info_async(
f"Publisher ({websocket.client.host}:{websocket.client.port}) established socket connection successfully."
try:
await socket_broker.accept()

await websocket.accept()
asyncio.create_task(
log_info_async(
f"Publisher ({websocket.client.host}:{websocket.client.port}) established socket connection successfully."
)
)
)

try:
while True:
data = await websocket.receive_text()
if data:
_ = await socket_manager.broadcast_to_channel(
_ = await socket_broker.broadcast_to_channel(
channel_id, json.dumps(data)
)
except WebSocketDisconnect:
await log_warning_async(
f"Publisher ({websocket.client.host}:{websocket.client.port}) disconnected from the channel `{channel_id}`."
)
except redis.exceptions.ConnectionError as e:
await log_exception_async(
f"Failed to establish connection with Redis server due to an Exception: {str(e)}"
)
shutdown()
except Exception as e:
await log_exception_async(f"Exception occurred: {str(e)}.")
shutdown()


@router.websocket("/sub/{channel_id}")
Expand All @@ -64,17 +73,29 @@ async def subscribe(websocket: WebSocket, channel_id: str):
raise WebSocketException(code=status.WS_1003_UNSUPPORTED_DATA)

try:
await socket_manager.add_user_to_channel(channel_id, websocket)
await socket_broker.accept()

await websocket.accept()
asyncio.create_task(
log_info_async(
f"Client ({websocket.client.host}:{websocket.client.port}) subscribed successfully to the channel."
f"Subscriber ({websocket.client.host}:{websocket.client.port}) established socket connection successfully."
)
)

await socket_broker.add_client_to_channel(websocket)

while True:
_ = await websocket.receive_json()
except WebSocketDisconnect:
await log_warning_async(
f"Client ({websocket.client.host}:{websocket.client.port}) disconnected from the channel `{channel_id}`."
)
await socket_broker.remove_client_from_channel(websocket)
except redis.exceptions.ConnectionError as e:
await log_exception_async(
f"Failed to establish connection with Redis server due to an Exception: {str(e)}"
)
shutdown()
except Exception as e:
await log_exception_async(f"Exception occurred: {str(e)}.")
shutdown()
2 changes: 1 addition & 1 deletion sniffer/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
DELAY = 30

# Format of the timestamp that will be stored alongside SSID.
TIMESTAMP_FORMAT = "%Y-%m-%d %H:%M:%S"
TIMESTAMP_FORMAT = "%Y-%m-%d %H:%M:%S.%f"

# Timestamp is used to name the channel id for storing data.
# Format is 'year + month + day', e.q. 20231202.
Expand Down
4 changes: 1 addition & 3 deletions sniffer/sniffer.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
import sys
import threading
from http.client import HTTPException

from websocket import WebSocketApp

from parser import parse_ip_packet_wrapper
from pathlib import Path
from urllib.error import HTTPError

from scapy.sendrecv import AsyncSniffer
from websocket import WebSocketApp
from yaspin import yaspin

from logger import create_logger, log_exception, log_info, log_warning
Expand Down
24 changes: 24 additions & 0 deletions sniffer/utils/pnls_util.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
import os
import signal
from contextlib import asynccontextmanager

from fastapi import FastAPI

from logger import log_error, log_info_async
from message_broker.websocket_broker import socket_broker


@asynccontextmanager
async def lifespan(app: FastAPI):
await log_info_async("Server starting.")
yield
await log_info_async("Server shutting down.")
await socket_broker.close_sockets()


def shutdown():
"""
Used to shut down the ASGI server.
"""
log_error(f"Server shut down forcefully.")
os.kill(os.getpid(), signal.SIGTERM)
Loading

0 comments on commit bc2023a

Please sign in to comment.