Docs, tech specs and README

Update technical specifications, README and user manual documentation

CHANGELOG.md

@@ -1,3 +1,19 @@
+# 0.1.2 (2024-09-07)
+
+## Added
+
+- new system/software requirement specification (SRS) items
+
+## Fixed
+
+- broken relative links in the user manual and README
+
+## Modified
+
+- technical specifications and traceability graph: revised content and certain high-level architecture (HARC) design artifacts moved to the low-level architecture (LARC) and others to software design (SWD)
+- README to provide further usage recommendations and to highlight a few technical points related to the underlying machine learning algorithm
+- user manual "overview" page renamed to "README"
+
 # 0.1.1 (2024-09-01)
 
 ## Added

README.md

@@ -88,7 +88,7 @@ See our [Instructions for IDPS and SIEM integrated deployment](./deployment/READ
 
 ## User manual
 
-Please see the [Instructions for IDPS and SIEM integrated deployment](./deployment/README.md) and the [ADBox user manual](./docs/manual/overview.md) page to learn more about the installation, setup requirements, overall usage and specific modules of the ADBox. 
+Please see the [Instructions for IDPS and SIEM integrated deployment](./deployment/README.md) and the [ADBox user manual](./docs/manual/README.md) page to learn more about the installation, setup requirements, overall usage and specific modules of the ADBox. 
 
 ## Documentation and technical specifications
 
@@ -233,7 +233,7 @@ The output of the all the detections performed through the console are stored in
 ./adbox.sh
 ```
 
-In this mode, the ADBox will train a detector using the default arguments and then also perform detection based on default arguments, with the detector trained using the previously mentioned default arguments. To know more about the input arguments used in default mode, visit the [user manual](./docs/manual/overview.md) page.
+In this mode, the ADBox will train a detector using the default arguments and then also perform detection based on default arguments, with the detector trained using the previously mentioned default arguments. To know more about the input arguments used in default mode, visit the [user manual](./docs/manual/README.md) page.
 
 The output of the default detection is also stored in `./siem_mtad_gat/assets/detector_models/{detector_id}/prediction` folder.
 
@@ -250,7 +250,7 @@ I have deployed all the components as explained in the [guide for IDPS and SIEM
 
 I want a detector which correlates resource usage and rules statistics. Once created, I want this detector to keep running on new data.
 
-Therefore, I prepare a dedicated [use case file](./use_case.md).
+Therefore, I prepare a dedicated [use case file](./docs/manual/use_case.md).
 
 Namely,
 
@@ -398,22 +398,36 @@ We traced the two anomalies to two real events that had happened in the correspo
 
 In both cases, the actions that (most probably) generated the anomalies had been carried out by a system administrator. In case not, while A1 would have been noticed by looking at single features and/or Wazuh; A2 would have not been as obvious to track.
 
-## Disclaimer: Use of alpha/experimental Software
+## Disclaimer: use of alpha/experimental software
 
 This software is currently in its alpha or experimental phase and is provided for testing and evaluation purposes only. It may contain errors, bugs, or other issues that could result in security vulnerabilities, data loss, or other unpredictable outcomes. As such, **this software is not intended for use in production environments** or for handling sensitive, confidential, or critical information.
 
-In particular, given the nature of security-related software, it is crucial to understand that the algorithms, protocols, and implementations within this software may not have undergone thorough security audits or peer review. **Do not rely on this software for critical system functions.**.
+In particular, given the nature of security-related software, it is crucial to understand that the algorithms, protocols, and implementations within this software may not have undergone thorough security audits or peer review. **Do not rely on this software for critical system functions.**
 
 The developers, contributors, and affiliated organizations **disclaim all warranties, express or implied,** including but not limited to the implied warranties of fitness for a particular purpose. **No guarantee is made regarding the correctness, completeness, or security** of the software, and you assume full responsibility for any risks associated with its use.
 
 By using this software, you acknowledge that you understand the risks and agree to use it **at your own risk.** You are strongly encouraged to conduct your own security assessments and tests before deploying this software in any environment.
 
+### Usage recommendations and remarks
+
+We advise the user __not__ to 
+- set the detector time granularity parameter to a value lower than 30s,
+- run prediction-only use cases if there are no corresponding detectors available in the [detectors folder](./siem_mtad_gat/assets/detector_models/). 
+
+Furthermore, we highlight the following points:
+
+- Anomalous timestamps should be considered more as period indicators rather than precise links to an event for two reasons: 
+  - (i) data points from events are aggregated, to which rounding is applied during preprocessing; 
+  - (ii) the anomaly is defined in terms of windows, hence consecutive sets of events.
+- Depending on the selected configurations, running the full stack of IDPS-ESCAPE may require up to 26 GB of persistent storage, while RAM usage for the default ADBox configuration remains close to 4 GB, the same as the recommended value for Wazuh, which is used as our source of data for training and prediction. Note that the various subsystems can be deployed on different nodes, e.g., the ADBox on one node and our customized Wazuh+Suricata setup on another, or all three on separate nodes (see the [integration](./deployment/README.md) and [remote monitoring](./deployment/remote_monitoring/remote_monitoring.md) pages).
+- Clearly, the MTAD-GAT hyperparameters (e.g., the number of GRU layers) require tuning when it comes to training machine learning models.
+
 ## Roadmap
 
 Some of the currently planned items include:
 
 - automated unit/integration/system test suites providing coverage for critical parts of the software;
-- a GUI frontend integrated into Wazuh or OpenSearch;
+- a web-based GUI frontend, either standalone or integrated into Wazuh or OpenSearch;
 - tailoring the underlying ADBox algorithms to specific SOC operations.
 
 For details on our roadmap and features planned for future releases, please see the [Wiki](https://github.com/AbstractionsLab/idps-escape/wiki) section of this repository.

deployment/README.md

@@ -54,4 +54,5 @@ Restart the agent and then the logs could be seen on the Wazuh dashboard.
 sudo systemctl restart wazuh-agent 
 ```
 To test the correct working of the integration, any online tool that generates some malicious traffic for testing purposes can be used, such as: 
+
 [GitHub - 3CORESec/testmynids.org: A website and framework for testing NIDS detection](https://github.com/3CORESec/testmynids.org)  

deployment/integration.md

@@ -3,7 +3,7 @@ Once Suricata and Wazuh are installed, proceed with the integration process outl
 
 
 ### Requirements 
-1. [A running instance of the Suricata service
+1. A running instance of the Suricata service
 2. A running instance of the Wazuh distribution
 3. An installation of a Wazuh agent
 

deployment/remote_monitoring/remote_monitoring.md

@@ -18,6 +18,8 @@ Notice that, dedicated Wazuh agents are installed on the endpoints that need to
 1. [Install Wazuh agent](./wazuh/wazuh_agents.md) using address of cental manager.
 1. Follow [integration procedure](integration.md) with local paths.
 
+![Integrated Model](./../../docs/specs/larc/assets/IDPS-ESCAPE-EP-integrated.png)
+
 ### Hybrid Model 
 
 1. [Install Wazuh agent](./wazuh/wazuh_agents.md) using address of cental manager. 
@@ -52,3 +54,5 @@ Then finally, the script can be executed as:
 ./idps-escape-port-mirroring.sh source_interface=enp9s0 source_ip=192.168.5.8 destination_interface=enp7s0 destination_ip=192.168.5.5 tunnel_ip=192.168.5.10/24 tunnel_name=tun0
 ```
  This creates a GRE tunnel from the source to destination mirroring all incoming and outgoing network traffic to the destination machine. Any malicious activity on the source machine would now be seen through the Wazuh dashboard on the destination machine i.e., the centralized monitoring machine hosting the Suricata and Wazuh integration. 
+
+![Integrated Model](./../../docs/specs/larc/assets/IDPS-ESCAPE-EP-hybrid.png)

docs/manual/overview.md → docs/manual/README.md

@@ -29,18 +29,22 @@ SIEM.
 - [Detector](./detector_data_structure.md)
 - [Front-end](./front_end.md)
 - [Data transformation](./data_transformation.md)
+- [Example](./example.md)
+- [Glossary](./glossary.md)
+
 
 ## Overview
 
 - **Install.** Instructions to install ADBox.
-- **Setup and prerequisites.** List of configurations file and prerequisites to complete the deployment of ADBox and be able to create and run detectors.
+- **Setup and prerequisites.** List of configurations file and prerequisites to complete the deployment of ADBox and to be able to create and run detectors.
 - **User guide**. 
-- **Use case definition guide**. Via ADBox it is possible to create, use and maintain detector which ingest data and analyzed them. The user can simply define the parameter via *use case* configuration files and feed them to ADBox entry point. This page contains the instruction for understanding and defining use case files.
-- **Anomaly detection engine.** The anomaly detection engine is the core component of ADBox. In fact, for every available anomaly detection method it orchestrates the interaction between the bulk functions of every algorithm, the data ingestion, data storage, user output, etc. In other words, the Engine determines the sequence of action to be performed to successfully go through the detection pipeline. This page gives also an overview of the train and predict pipelines.
-- **MTAD-GAT.** ADBox incorporates machine learning algorithms for AD. Currently, MTAD-GAT algorithm is supported. This page is a high level overview.
-- **Detector** The *detectors* are the "object"  used to perform detection. This page explained the notion and provide an overview of the ADBox pipelines' outcomes.
+- **Use case definition guide**. Via ADBox it is possible to create, use and maintain detectors which ingest data and analyze them. The user can simply define the parameters via *use case* configuration files and feed them to the ADBox entry point. This page contains the instructions for understanding and defining use case files.
+- **Anomaly detection engine.** The anomaly detection engine is the core component of ADBox. In fact, for every available anomaly detection method it orchestrates the interaction between the bulk functions of every algorithm, the data ingestion, data storage, user output, etc. In other words, the Engine determines the sequence of actions to be performed to successfully go through the detection pipeline. This page also gives an overview of the train and predict pipelines.
+- **MTAD-GAT.** ADBox incorporates machine learning algorithms for AD. Currently, the MTAD-GAT algorithm is supported. This page gives a high level overview.
+- **Detector**. The *detectors* are the "objects" used to perform detection. This page explains this notion and provides an overview of the ADBox pipelines' outcomes.
 - **Front-end.** Available front-end interfaces.
-- **Data transformation.** The row data ingested by ADBox from Wazuh, or any other source, must be cleaned and prepared to be fed to the machine learning model. This page provides an overview of the transformations, including the preprocessing.
-
+- **Data transformation.** The raw data ingested by ADBox from Wazuh, or any other source, must be cleaned and prepared to be fed to the machine learning model. This page provides an overview of the transformations, including the preprocessing.
+- **Example**. An example of ADBox usage from use case definition up to output analysis. This example used data from HIDS and includes [Monitoring Linux resource usage](./linux_resource.md).
+- **Glossary**. Summary of specific terminology used in this manual.
 
 ![ADBox ADBox high level architecture](../specs/harc/assets/ADBox-system_high_level_architecture.png "ADBox high level architecture")

docs/manual/installation.md

@@ -99,7 +99,7 @@ The output of the all the detections performed through the console are stored in
 ./adbox.sh
 ```
 
-In this mode, the ADBox will train a detector using the default arguments and then also perform detection based on default arguments, with the detector trained using the previously mentioned default arguments. To know more about the input arguments used in default mode, visit the [user manual](./docs/manual/overview.md) page.
+In this mode, the ADBox will train a detector using the default arguments and then also perform detection based on default arguments, with the detector trained using the previously mentioned default arguments. To know more about the input arguments used in default mode, visit the [user manual](./README.md) page.
 
 The output of the default detection is also stored in `./siem_mtad_gat/assets/detector_models/{detector_id}/prediction` folder.
 

docs/manual/time_series.md

@@ -18,5 +18,5 @@ $$
 \vdots & \ddots & \vdots\\
 {x}_{k,1} &\dots&{x}_{k,n}
 \end{bmatrix}} { features}
-$$ {#eq-ts-decomposition}
+$$
 

docs/manual/user_guide.md

@@ -1,20 +1,19 @@
 # User guide 
 
-Via ADBox one can create detectors and use them to detect anomalous behaviour in chosen data. 
-
-To do so, it is sufficient to compile a **use case** file summarizing the necessary information and run the following command via the terminal
+Via ADBox one can create detectors and use them to detect anomalous behaviors in chosen data. To do so, it is sufficient to compile a **use case** file summarizing the necessary information and run the following command via the terminal
 
 ```sh
 ./adbox.sh -u {number}
 ```
+
 where `{number}` is the number associated with the use case.
 
 A complete overview of how to define a use case is given in the [use case definition guide](./use_case.md).
 
 Via a use case one can
 
 - create a detector, by including the `training` settings.
-- use a detector for predict, by including the `prediction` settings.
+- use a detector for prediction, by including the `prediction` settings.
 - both, by including both `training`and `prediction` settings.
 
 Indeed, if the `training`/`prediction` key is present, the ADBox starts the corresponding pipeline in the [anomaly detection engine](./engine.md).

docs/specs/harc/HARC-001.yml

@@ -1,20 +1,21 @@
 active: true
 derived: false
-header: ''
+header: |
+  ADBox subsystem
 level: 1.0
 links:
-- MRS-002: null
-- MRS-004: null
-- MRS-032: null
-- MRS-033: null
-- MRS-039: null
-- MRS-041: null
+- MRS-002: PcjCPpAQg4L-F5WYHxXPLu4WLt030SwN9R24ecLAMdg=
+- MRS-004: xi45PNuE6R8VT5LWMegbdcrYRDwa3nxZATXBXcayriA=
+- MRS-032: 8PqSYwKBzcz2ebnoKiLMRb21-sFRTziyoKQ3hieM-yY=
+- MRS-033: Qo3tudn-i84fGKUF66LxEtGw3FdzfYcS9U1xwgQKYJo=
+- MRS-039: 4pepNOPXHsHmADQ2Ub21C5a1WEQO9zFwcEBZremD0HM=
+- MRS-041: y8mj4oHfwFhaVQebBfNa9p9DRhqS88bd8RS1i3nzQW0=
 normative: true
 ref: ''
-reviewed: null
+reviewed: Hzg5bd3rAuMArWh5hMiEIZ8gAMz4pFCyk8CHY2ncEjU=
 text: |
-  ## Diagram
+  ## ADBox subsystem diagram
 
-  The diagram below explicit the context of ADBox subsystem.
+  The diagram below depicts the context of the ADBox subsystem.
 
-  ![ADBox context diagram](assets/ADBox-context.png "ADBox context diagram")
+  ![ADBox context diagram](assets/ADBox-context.png "ADBox context diagram"){: width="80%"}

docs/specs/harc/HARC-002.yml

@@ -1,18 +1,21 @@
 active: true
 derived: false
-header: ''
-level: 1.1
+header: |
+  ADBox architecture
+level: 2.0
 links:
-- MRS-004: null
-- MRS-031: null
-- MRS-032: null
-- MRS-033: null
-- MRS-039: null
-- MRS-041: null
+- MRS-004: xi45PNuE6R8VT5LWMegbdcrYRDwa3nxZATXBXcayriA=
+- MRS-031: Qe4O1p-Ejjhx1iVzqvyOfQ2tmT40K3v6GYgQlFi72cc=
+- MRS-032: 8PqSYwKBzcz2ebnoKiLMRb21-sFRTziyoKQ3hieM-yY=
+- MRS-033: Qo3tudn-i84fGKUF66LxEtGw3FdzfYcS9U1xwgQKYJo=
+- MRS-039: 4pepNOPXHsHmADQ2Ub21C5a1WEQO9zFwcEBZremD0HM=
+- MRS-041: y8mj4oHfwFhaVQebBfNa9p9DRhqS88bd8RS1i3nzQW0=
 normative: true
 ref: ''
-reviewed: null
+reviewed: m-8WrFxsgTppkhedHH9jrjk2mpniEKijIpfaieSLWUQ=
 text: |
-  ## Diagram
-  The diagram below explicit the architecture of ADBox subsystem.
-  ![ADBox high level architecture](assets/ADBox-system_high_level_architecture.png "ADBox high level architecture")
+  ## ADBox architecture diagram
+  
+  The diagram below depicts the high-level architecture of the ADBox subsystem.
+
+  ![ADBox high level architecture](assets/ADBox-system_high_level_architecture.png "ADBox high level architecture"){: width="70%"}

docs/specs/harc/HARC-003.yml

@@ -1,16 +1,36 @@
 active: true
 derived: false
-header: ''
-level: 1.2
+header: |
+  IDPS-ESCAPE context
+level: 3.0
 links:
-- MRS-004: null
-- MRS-032: null
+- MRS-002: PcjCPpAQg4L-F5WYHxXPLu4WLt030SwN9R24ecLAMdg=
+- MRS-003: MbDfgrdzfC2Mk9ADobhu5OjCFeQsCaMiAjfLxd4LwMU=
+- MRS-004: xi45PNuE6R8VT5LWMegbdcrYRDwa3nxZATXBXcayriA=
+- MRS-005: WU-OQqjP5o1FghU8Hq0-_jp7o-kDIm-B969Ssgvg058=
+- MRS-006: _1kYIYSuIF1OxtdFRnPwTZQNZ1Jwadp4bEqGWuy5eBc=
+- MRS-007: MYGDh7hfSDYQoh_aLsw5nLs_RYInVm0opcD702A5fQI=
+- MRS-008: NIxP2kk-rkPA2dTJMoWs_cm5XQjXwuOpIPvuF6uLtkE=
+- MRS-011: mwmFXP_LWPCCuta_18kOVD8Cn5UGc6hSaqZf8L1TPnk=
+- MRS-012: C4CHVQL0aQetnxT0ulvjF_wEMJInUYdyZ1OXkdBWFfM=
+- MRS-013: FoVayZaf7wo3PIRmHpo3nQhlJAwCaeJilr_WYMlED6o=
+- MRS-017: eFUzqFk86Hzn5XIHC5Mh7vit4tbtyWX7f8PuMA9YFqE=
+- MRS-018: -YZZxjG1EIKk2Cj3rMXW-DbeLE15GVBx-fULfIJWpnI=
+- MRS-019: u5dQxXMsKxcdsP_2TIRtSZ2_htqogK5U5NtIBgzP8t4=
+- MRS-020: Y0EQ-_vRDGXOkkPcR-vf-PXnXWBPCg1bp-vp-1RNm_0=
+- MRS-022: lUOrsiYzrOZ7rjU8CQ6ZJhmoG-3JFUS6PRR5BswoPeM=
+- MRS-032: 8PqSYwKBzcz2ebnoKiLMRb21-sFRTziyoKQ3hieM-yY=
+- MRS-033: Qo3tudn-i84fGKUF66LxEtGw3FdzfYcS9U1xwgQKYJo=
+- MRS-037: qYyZcNrxvOvpVpfsQlpGAkUdhRKaGnSVCekMBzILydw=
+- MRS-039: 4pepNOPXHsHmADQ2Ub21C5a1WEQO9zFwcEBZremD0HM=
+- MRS-040: arnExMS6GVpYR1r-58fDJlROjnNgn51uRAodqXmSH4M=
+- MRS-041: y8mj4oHfwFhaVQebBfNa9p9DRhqS88bd8RS1i3nzQW0=
 normative: true
 ref: ''
-reviewed: null
+reviewed: TtJxl72Lls6qKVr41dHm-_x93Y2sv_vbL89Fbx-2Q_A=
 text: |
-  ## Diagram
+  ## IPDS-ESCAPE context diagram
 
-  The diagram below explicit ADEngine object of ADBox.
+  The diagram below illustrates the overall context of the IDPS-ESCAPE SOAR system.
 
-  ![ADBox Engine class diagram](assets/ADBox-engine-class.png "ADBox Engine class diagram")
+  ![IDPS-ESCAPE context diagram](assets/IDPS-ESCAPE-context-diagram.png "IDPS-ESCAPE context diagram"){: width="80%"}