From 71882491d95d3b0050fb7d67468991ce06dad76c Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 17:29:01 +0000 Subject: [PATCH 01/16] Tweaked index file. Changed title, improved target audience statement. --- .../snort3-multithreading/_index.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md index 7cddca063..75422d95e 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md @@ -1,16 +1,12 @@ --- -title: Scaling Snort3 - use multithreading for improved performance - -draft: true -cascade: - draft: true +title: Optimize performance of Snort 3 using multithreading minutes_to_complete: 45 -who_is_this_for: This blog is for engineers familiar with Snort who want to enhance its performance by leveraging the benefits of multithreading. +who_is_this_for: This Learning Path is for software developers familiar with Snort who want to optimize performance by leveraging the benefits of multithreading. learning_objectives: - - Install Snort with all of its dependencies. + - Install Snort and all of its dependencies. - Configure Snort Lua files to enable multithreading. - Use multithreading to process capture files and measure performance. From 462d3ddc13f63e2e1dcd538685968d48479f2909 Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 17:38:28 +0000 Subject: [PATCH 02/16] Some tweaks --- .../build-and-install.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md index 502f43755..3b9fdc2cd 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md @@ -1,5 +1,5 @@ --- -title: Install Snort3 and the required dependencies +title: Install Snort 3 and dependencies weight: 2 ### FIXED, DO NOT MODIFY @@ -10,15 +10,15 @@ Snort is an Open Source Intrusion Prevention System (IPS). Snort uses a series o Multithreading in Snort 3 refers to the ability to associate multiple threads with a single Snort instance enabling the concurrent processing of multiple packet files. This optimization frees up additional memory for further packet processing. -In order to enable multithreading in Snort3, specify the quantity of threads designated for processing network traffic using either the '--max-packet-threads' or '-z' option. +In order to enable multithreading in Snort 3, specify the quantity of threads designated for processing network traffic using either the '--max-packet-threads' or '-z' option. {{%notice Note%}} - The instructions provided have been tested on AWS EC2 Graviton4 instance, based on Neoverse V2. The examples are easiest to use if you have at least 16 cores in the system. + The instructions provided have been tested on AWS EC2 Graviton4 instance, based on Arm Neoverse V2. The examples work best if you have at least 16 cores in your system. {{%/notice%}} -## Compile and build Snort3 +## Compile and build Snort 3 -To install Snort3, use a text editor to save the script below on your Arm server in a file named `install-snort.sh`. +To install Snort 3, use a text editor to save the script below on your Arm server in a file named `install-snort.sh`. ``` bash @@ -193,21 +193,21 @@ echo 'make sure to source ~/.bashrc or set LD_LIBRARY_PATH using:"' echo ' export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"' ``` -The script takes 2 arguments: -- the directory used to build Snort3 and its dependencies -- the number of processors to use for the build +The script takes two arguments: +* The directory used to build Snort 3 and its dependencies. +* The number of processors to use for the build. -To build in a new directory named `build` with the number of processors in your system, run the script: +To create a new directory named `build` with the number of processors in your system listed, run the script: ```bash bash ./install-snort.sh build `nproc` ``` -You don't need to run the script as `root` but it assumes you are on Ubuntu 20.04 or 22.04 and have sudo permission. +You do not need to run the script as `root`, but it assumes you are on Ubuntu 20.04 or 22.04 and have sudo permission. -When the build completes you have the snort3 directory with all compiled software, and the `snort` executable is located in `/usr/local/bin`. +When the build completes, you will have the snort 3 directory with all compiled software, and the `snort` executable is located in `/usr/local/bin`. -To verify the installation is complete, run the command below and see the version printed: +To verify the installation is complete, run the command below and observe the version printed: ```bash { output_lines = "2-20" } snort -V @@ -228,6 +228,6 @@ To verify the installation is complete, run the command below and see the versio ``` -Don't delete the `build` directory as it will be used in the next step. +Do not delete the `build` directory as you will use it in the next step. -Proceed to learn how to test Snort3 multithreading. \ No newline at end of file +Now you can move on to learn how to test Snort 3 multithreading. From 5e971f99de42aa971dd33658694f1c05c1d2a55e Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 17:46:53 +0000 Subject: [PATCH 03/16] Update usecase.md --- .../snort3-multithreading/usecase.md | 32 +++++++++---------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md index 8d7507138..0ec4e83e4 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md @@ -1,17 +1,17 @@ --- -title: Test Snort3 multithreading +title: Test Snort 3 multithreading weight: 3 ### FIXED, DO NOT MODIFY layout: learningpathall --- -Before testing multithreading performance, perform the following steps to configure your system: +Before testing the Snort 3 multi-threading, configure your system by following these steps: -1. Configure Grub settings -2. Set up the Snort3 rule set -3. Download the PCAP files -4. Adjust Lua configurations +1. Configure Grub settings. +2. Set up the Snort3 rule set. +3. Download the PCAP files. +4. Adjust Lua configurations. ## Configure Grub settings @@ -39,8 +39,7 @@ After making this change, execute update-grub to apply the configuration: sudo update-grub ``` -Reboot the system to activate the settings. - +Reboot the system to activate the settings: ```bash sudo reboot ``` @@ -71,9 +70,9 @@ The output shows the isolated processors: 0-9 ``` -## Set up the Snort3 rule set +## Set up the Snort 3 rule set -Download the rule set from https://www.snort.org/ and extract it into your working directory. You should start in the `build` directory you used to build snort. +Download the rule set from https://www.snort.org/ and extract it into your working directory. Start in the `build` directory you used to build snort. ```bash cd $HOME/build @@ -95,7 +94,7 @@ Copy the `lua` folder from the `snort3` source directory into the rules director cp -r snort3/lua/ Test/snortrules/ ``` -## Download the packet capture (PCAP) files +## Download the Packet Capture (PCAP) files You can use any PCAP files that are relevant to your test scenario. @@ -115,10 +114,11 @@ cp maccdc2010_00000_20100310205651.pcap Test/Pcap/ ## Adjust Lua configurations There are two modifications to the Lau configurations: -- Pin each Snort thread to a unique core, ensuring that the cores match those isolated in the GRUB configuration -- Enable the desired ruleset and enabling profiling -### Pin snort threads to unique cpu core +* Pin each Snort thread to a unique core, ensuring that the cores match those isolated in the GRUB configuration. +* Enable the desired ruleset and enabling profiling. + +### Pin Snort Threads to Unique CPU Core Navigate to the `Test/snortrules/lua` directory. @@ -290,7 +290,7 @@ The output is similar to: 22:52:28 9 97.50 0.00 2.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ``` -## Test Snort3 multi-threading to process single pcap file +## Test Snort 3 multi-threading to process a single PCAP file The example usage demonstrates how multithreading increases the number of packets processed per second. @@ -307,4 +307,4 @@ Performance results | 1 | 940960 | 91.777964 | | 10 | 9406134 | 9.181182 | -The results demonstrate how increasing the thread count by ten times results in a ten times increase in packets processed per second, while reducing the execution time by ten times. \ No newline at end of file +The results demonstrate how increasing the thread count by ten times results in a ten times increase in packets processed per second, while reducing the execution time by ten times. From 28748febb2f4403bee99d32c4160adfde818df62 Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 17:53:05 +0000 Subject: [PATCH 04/16] Update _review.md --- .../snort3-multithreading/_review.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md index c439b6749..571a8df09 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md @@ -2,11 +2,11 @@ review: - questions: question: > - Which of the following is a key benefit of Snort3's multithreading support? + Which of the following is a key benefit of Snort 3's multithreading support? answers: - It allows Snort to detect encrypted traffic. - - It improves packet processing performance - - It enables Snort to be run on legacy hardware + - It improves packet processing performance. + - It enables Snort to be run on legacy hardware. - It support multiple rule sets at the same time. correct_answer: 2 explanation: > @@ -14,27 +14,27 @@ review: - questions: question: > - Which parameter is used to enable multithreading in Snort3? + Which parameter is used to enable multithreading in Snort 3? answers: - - --max-packet-threads - - --enable-threads - - --enable-multithreading - - --packet-loop + - --max-packet-threads. + - --enable-threads. + - --enable-multithreading. + - --packet-loop. correct_answer: 1 explanation: > --max-packet-threads parameter is used to enable and configure multithreading. - questions: question: > - In Snort 3, which DAQ (Data Acquisition) module is used to read capture files for packet processing? + In Snort 3, which Data Acquisition (DAQ) module is used to read capture files for packet processing? answers: - - afpacket - - vpp - - dump - - pcap + - afpacket. + - vpp. + - dump. + - pcap. correct_answer: 3 explanation: > - The dump module in Snort3 is used to read capture files (such as .pcap or .pcapng files) for offline packet analysis. + The dump module in Snort 3 is used to read capture files (such as .pcap or .pcapng files) for offline packet analysis. From 19a7b41330650d7ac5a34fe52a047fa34476e0c5 Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 17:57:22 +0000 Subject: [PATCH 05/16] Update usecase.md --- .../snort3-multithreading/usecase.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md index 0ec4e83e4..45d5baab3 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md @@ -9,7 +9,7 @@ layout: learningpathall Before testing the Snort 3 multi-threading, configure your system by following these steps: 1. Configure Grub settings. -2. Set up the Snort3 rule set. +2. Set up the Snort 3 rule set. 3. Download the PCAP files. 4. Adjust Lua configurations. @@ -72,7 +72,7 @@ The output shows the isolated processors: ## Set up the Snort 3 rule set -Download the rule set from https://www.snort.org/ and extract it into your working directory. Start in the `build` directory you used to build snort. +Download the rule set from https://www.snort.org/ and extract it into your working directory. Start in the `build` directory you used to build Snort. ```bash cd $HOME/build @@ -176,9 +176,9 @@ Continue to edit `snort.lua` and comment out the `profiler` and `latency` lines ### Modify the IPS policy -Snort3 allows you to fine-tune setups with the `--tweaks` parameter. This feature allows you to use one of Snort's policy files to enhance the detection engine for improved performance or increased security. +Snort 3 allows you to fine-tune setups with the `--tweaks` parameter. This feature allows you to use one of Snort's policy files to enhance the detection engine for improved performance or increased security. -Snort3 includes four preset policy files: max_detect, security, balanced, and connectivity. +Snort 3 includes four preset policy files: max_detect, security, balanced, and connectivity. The max_detect policy favors maximum security, whereas the connectivity policy focuses on performance and uptime, which may come at the expense of security. @@ -186,7 +186,7 @@ The max_detect policy favors maximum security, whereas the connectivity policy f Snort supports DAQ modules which serves as an abstraction layer for interfacing with data source such as network interface. -To see list of DAQ modules supported by snort use `--daq-list` command. +To see list of DAQ modules supported by Snort use `--daq-list` command. Return to the `build` directory: @@ -250,9 +250,9 @@ trace(v1): inline unpriv wrapper For testing, you can use `--daq dump` to analyze PCAP files. -## Spawn Snort3 process with multithreading +## Spawn Snort 3 process with multithreading -To run Snort3 with multithreading start from the `Test` directory. +To run Snort 3 with multithreading start from the `Test` directory. ```bash cd $HOME/build/Test From b640564baefc337e8eed7e70eb1a107329a942ee Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 18:00:26 +0000 Subject: [PATCH 06/16] Update build-and-install.md --- .../snort3-multithreading/build-and-install.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md index 3b9fdc2cd..737396f49 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md @@ -205,7 +205,7 @@ bash ./install-snort.sh build `nproc` You do not need to run the script as `root`, but it assumes you are on Ubuntu 20.04 or 22.04 and have sudo permission. -When the build completes, you will have the snort 3 directory with all compiled software, and the `snort` executable is located in `/usr/local/bin`. +When the build completes, you will have the Snort 3 directory with all compiled software, and the `snort` executable is located in `/usr/local/bin`. To verify the installation is complete, run the command below and observe the version printed: From 7397ba967b27ae985e4d3fa4f90721da1eb19063 Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 22:03:57 +0000 Subject: [PATCH 07/16] Added missing definite article in the title. --- .../snort3-multithreading/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md index 75422d95e..d0d007d26 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md @@ -1,5 +1,5 @@ --- -title: Optimize performance of Snort 3 using multithreading +title: Optimize the performance of Snort 3 using multithreading minutes_to_complete: 45 @@ -11,7 +11,7 @@ learning_objectives: - Use multithreading to process capture files and measure performance. prerequisites: - - An Arm-based instance from a cloud provider or an Arm server running Ubuntu 20.04 or 22.04. + - An Arm-based instance from a cloud provider, or an Arm server running Ubuntu 20.04 or 22.04. - A basic understanding of Snort's operation and configuration. From fde91ddc15fd8fd27ee31e4ee0a1ece33e553902 Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 22:06:14 +0000 Subject: [PATCH 08/16] Update _next-steps.md --- .../snort3-multithreading/_next-steps.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md index 5d7e1d691..311f62435 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md @@ -1,11 +1,11 @@ --- -next_step_guidance: To continue learning about enabling hyperscan on arm,please refer to the learning path provided below. +next_step_guidance: You can now try the Learning Path about enabling hyperscan on arm. See the link below. recommended_path: /learning-paths/servers-and-cloud-computing/vectorscan/ further_reading: - resource: - title: Snort3 Documentation + title: Snort 3 Documentation link: https://docs.snort.org/start/ type: documentation - resource: From b46c7285b585774681019181f74f3d5869db37e7 Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 22:07:59 +0000 Subject: [PATCH 09/16] Update _review.md --- .../snort3-multithreading/_review.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md index 571a8df09..9dcd37ae6 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md @@ -7,7 +7,7 @@ review: - It allows Snort to detect encrypted traffic. - It improves packet processing performance. - It enables Snort to be run on legacy hardware. - - It support multiple rule sets at the same time. + - It supports multiple rule sets at the same time. correct_answer: 2 explanation: > It improves packet processing performance by parallelizing tasks. @@ -34,7 +34,7 @@ review: - pcap. correct_answer: 3 explanation: > - The dump module in Snort 3 is used to read capture files (such as .pcap or .pcapng files) for offline packet analysis. + The dump module in Snort 3 is used to read capture files, such as .pcap or .pcapng files, for offline packet analysis. From 70b073f3115273255688a1719704f90a7c2166a0 Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 22:14:54 +0000 Subject: [PATCH 10/16] Update build-and-install.md --- .../snort3-multithreading/build-and-install.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md index 737396f49..c05024491 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md @@ -1,5 +1,5 @@ --- -title: Install Snort 3 and dependencies +title: Install Snort 3 and its Dependencies weight: 2 ### FIXED, DO NOT MODIFY @@ -8,7 +8,7 @@ layout: learningpathall Snort is an Open Source Intrusion Prevention System (IPS). Snort uses a series of rules to define malicious network activity. If malicious activity is found, Snort generates alerts. -Multithreading in Snort 3 refers to the ability to associate multiple threads with a single Snort instance enabling the concurrent processing of multiple packet files. This optimization frees up additional memory for further packet processing. +Multithreading in Snort 3 refers to the ability to associate multiple threads with a single Snort instance, which enables the concurrent processing of multiple packet files. This optimization frees up additional memory for further packet processing. In order to enable multithreading in Snort 3, specify the quantity of threads designated for processing network traffic using either the '--max-packet-threads' or '-z' option. @@ -18,7 +18,7 @@ In order to enable multithreading in Snort 3, specify the quantity of threads de ## Compile and build Snort 3 -To install Snort 3, use a text editor to save the script below on your Arm server in a file named `install-snort.sh`. +To install Snort 3, use a text editor to copy the text below and save the script on your Arm server in a file named `install-snort.sh`. ``` bash From 88b8060e87f69dd50e876378069d66d263bda938 Mon Sep 17 00:00:00 2001 From: Maddy Underwood <167196745+madeline-underwood@users.noreply.github.com> Date: Wed, 18 Dec 2024 22:29:24 +0000 Subject: [PATCH 11/16] Update usecase.md --- .../snort3-multithreading/usecase.md | 37 +++++++++++-------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md index 45d5baab3..97c4b74ff 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md @@ -6,7 +6,7 @@ weight: 3 layout: learningpathall --- -Before testing the Snort 3 multi-threading, configure your system by following these steps: +Before testing the Snort 3 multithreading, configure your system by following these steps: 1. Configure Grub settings. 2. Set up the Snort 3 rule set. @@ -19,7 +19,8 @@ To enable Transparent HugePages (THP) and configure CPU isolation and affinity, For the total available online CPUs ranging from 0 to 95, with CPUs 0 to 9 pinned to Snort, the grubfile configuration is shown below. -Feel free to modify the CPU numbers as needed. +You can modify the CPU numbers as needed: + ```bash CMDLINE="cma=128" HUGEPAGES="default_hugepagesz=1G hugepagesz=1G hugepages=300" @@ -98,12 +99,11 @@ cp -r snort3/lua/ Test/snortrules/ You can use any PCAP files that are relevant to your test scenario. -One place to get PCAP files is: -https://www.netresec.com/?page=MACCDC +You can obtain PCAP files at: https://www.netresec.com/?page=MACCDC. Visit https://share.netresec.com/s/wC4mqF2HNso4Ten and download a PCAP file. -Copy the file to your working directory and extract it, adjust the file name as needed if you downloaded a different PCAP file. +Copy the file to your working directory, and extract it. If you downloaded a different PCAP file, you might want to change the file name. ```bash gunzip maccdc2010_00000_20100310205651.pcap.gz @@ -113,7 +113,7 @@ cp maccdc2010_00000_20100310205651.pcap Test/Pcap/ ## Adjust Lua configurations -There are two modifications to the Lau configurations: +Now make two modifications to the Lau configurations: * Pin each Snort thread to a unique core, ensuring that the cores match those isolated in the GRUB configuration. * Enable the desired ruleset and enabling profiling. @@ -126,7 +126,7 @@ Navigate to the `Test/snortrules/lua` directory. cd Test/snortrules/lua ```` -Use an editor to create a file named `common.lua` with the contents below. +Use an editor to create a file named `common.lua`, and copy-and-paste in the contents below: ```bash ------------------------------------------------------------------------------- @@ -151,7 +151,7 @@ search_engine = { } snort_whitelist_append("threads") ``` -Include the above file in `snort.lua` by editing the file and adding the line below to the end of the file. +Edit `snort.lua` to include the contents above, and then add in the line below to the end of the file: ``` bash include('common.lua') @@ -178,15 +178,20 @@ Continue to edit `snort.lua` and comment out the `profiler` and `latency` lines Snort 3 allows you to fine-tune setups with the `--tweaks` parameter. This feature allows you to use one of Snort's policy files to enhance the detection engine for improved performance or increased security. -Snort 3 includes four preset policy files: max_detect, security, balanced, and connectivity. +Snort 3 includes four preset policy files: + +* Max_detect. +* Security. +* Balanced. +* Connectivity. -The max_detect policy favors maximum security, whereas the connectivity policy focuses on performance and uptime, which may come at the expense of security. +The max_detect policy focuses on maximum security, and the connectivity policy focuses on performance and uptime, which might come at the expense of security. ### Specify the data acquisition module Snort supports DAQ modules which serves as an abstraction layer for interfacing with data source such as network interface. -To see list of DAQ modules supported by Snort use `--daq-list` command. +To see list of DAQ modules supported by Snort use the `--daq-list` command. Return to the `build` directory: @@ -200,7 +205,7 @@ Run using the command: snort --daq-dir ./snort3/dependencies/libdaq/install/lib/daq --daq-list ``` -The output is: +The output should look like: ```output Available DAQ modules: @@ -248,17 +253,17 @@ trace(v1): inline unpriv wrapper file - Filename to write text traces to (default: inline-out.txt) ``` -For testing, you can use `--daq dump` to analyze PCAP files. +For testing, you can use `--daq dump` to analyze Pthe CAP files. ## Spawn Snort 3 process with multithreading -To run Snort 3 with multithreading start from the `Test` directory. +To run Snort 3 with multithreading, start from the `Test` directory. ```bash cd $HOME/build/Test ``` -The following example shows how to use multiple Snort threads to analyze PCAP files. +The following example shows you how to use multiple Snort threads to analyze PCAP files. ``` bash MPSE=hyperscan POLICY=./snortrules/lua/snort.lua TCMALLOC_MEMFS_MALLOC_PATH=/dev/hugepages/test snort -c ./snortrules/lua/snort.lua --lua detection.allow_missing_so_rules=true --pcap-filter maccdc2010_00000_20100310205651.pcap --pcap-loop 10 --snaplen 0 --max-packet-threads 10 --daq dump --daq-dir /usr/local/lib/daq --daq-var output=none -H --pcap-dir Pcap -Q --warn-conf-strict --tweaks security @@ -290,7 +295,7 @@ The output is similar to: 22:52:28 9 97.50 0.00 2.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ``` -## Test Snort 3 multi-threading to process a single PCAP file +## Test Snort 3 multithreading to process a single PCAP file The example usage demonstrates how multithreading increases the number of packets processed per second. From e1329726e1837c5d69bf3eb620629283e6a4d4e9 Mon Sep 17 00:00:00 2001 From: Maddy Underwood Date: Thu, 19 Dec 2024 05:17:00 +0000 Subject: [PATCH 12/16] Improvements to Snort 3 LP --- .../build-and-install.md | 31 ++++++++++------ .../snort3-multithreading/usecase.md | 35 ++++++++++--------- 2 files changed, 39 insertions(+), 27 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md index c05024491..8224ab15a 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md @@ -1,24 +1,31 @@ --- -title: Install Snort 3 and its Dependencies +title: Install Snort 3 and Dependencies weight: 2 ### FIXED, DO NOT MODIFY layout: learningpathall --- -Snort is an Open Source Intrusion Prevention System (IPS). Snort uses a series of rules to define malicious network activity. If malicious activity is found, Snort generates alerts. +#### Snort 3 -Multithreading in Snort 3 refers to the ability to associate multiple threads with a single Snort instance, which enables the concurrent processing of multiple packet files. This optimization frees up additional memory for further packet processing. +Snort is an Open Source Intrusion Prevention System (IPS). Snort uses a series of rules to define malicious network activity. If malicious activity is detected, Snort generates alerts. -In order to enable multithreading in Snort 3, specify the quantity of threads designated for processing network traffic using either the '--max-packet-threads' or '-z' option. +Snort 3 benefits from multithreading, which means that it enables the concurrent processing of multiple packet processing threads with a single Snort instance. This optimization frees up additional memory for further packet processing. + +#### Enable multithreading + +In order to enable multithreading in Snort 3, specify the quantity of threads designated for processing network traffic using either of these two options: + +* `--max-packet-threads` +* `-z` {{%notice Note%}} - The instructions provided have been tested on AWS EC2 Graviton4 instance, based on Arm Neoverse V2. The examples work best if you have at least 16 cores in your system. + These instructions have been tested on an AWS EC2 Graviton4 instance, based on Arm Neoverse V2. The examples work best if you have at least 16 cores in your system. {{%/notice%}} ## Compile and build Snort 3 -To install Snort 3, use a text editor to copy the text below and save the script on your Arm server in a file named `install-snort.sh`. +To install Snort 3, use a text editor to copy-and-paste the text below and save the script on your Arm server in a file named `install-snort.sh`. ``` bash @@ -197,17 +204,17 @@ The script takes two arguments: * The directory used to build Snort 3 and its dependencies. * The number of processors to use for the build. -To create a new directory named `build` with the number of processors in your system listed, run the script: +To create a new directory named `build` which lists the number of processors in your system, run the script: ```bash bash ./install-snort.sh build `nproc` ``` -You do not need to run the script as `root`, but it assumes you are on Ubuntu 20.04 or 22.04 and have sudo permission. +You do not need to run the script as `root`, but you do need to be running Ubuntu 20.04 or 22.04, and have sudo permission. -When the build completes, you will have the Snort 3 directory with all compiled software, and the `snort` executable is located in `/usr/local/bin`. +When the build completes, you will have the Snort 3 directory with all compiled software, and the `snort` executable will be located in `/usr/local/bin`. -To verify the installation is complete, run the command below and observe the version printed: +To verify completed installation, run the command below and look at the version that it prints to screen: ```bash { output_lines = "2-20" } snort -V @@ -228,6 +235,8 @@ To verify the installation is complete, run the command below and observe the ve ``` +{{% notice Note %}} Do not delete the `build` directory as you will use it in the next step. +{{% /notice %}} -Now you can move on to learn how to test Snort 3 multithreading. +Now you can move on to learn about how to test Snort 3 multithreading. diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md index 97c4b74ff..0f3791dfb 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md @@ -5,21 +5,23 @@ weight: 3 ### FIXED, DO NOT MODIFY layout: learningpathall --- +## System Configuration Before testing the Snort 3 multithreading, configure your system by following these steps: -1. Configure Grub settings. -2. Set up the Snort 3 rule set. -3. Download the PCAP files. -4. Adjust Lua configurations. +* Configure Grub settings. +* Set up the Snort 3 rule set. +* Download the PCAP files. +* Adjust Lua configurations. -## Configure Grub settings +### Configure Grub settings -To enable Transparent HugePages (THP) and configure CPU isolation and affinity, append the following line to the /etc/default/grub file: +To enable Transparent HugePages (THP) and configure CPU isolation and affinity, append the following line to the `/etc/default/grub file`, modifying the CPU numbers as required: +{{% notice Note %}} For the total available online CPUs ranging from 0 to 95, with CPUs 0 to 9 pinned to Snort, the grubfile configuration is shown below. +{{% /notice %}} -You can modify the CPU numbers as needed: ```bash CMDLINE="cma=128" @@ -34,13 +36,14 @@ THP="transparent_hugepage=madvise" GRUB_CMDLINE_LINUX="${CMDLINE} ${HUGEPAGES} ${ISOLCPUS} ${IRQAFFINITY} ${NOHZ} ${RCU} ${MAXCPUS} ${IOMMU} ${THP}" ``` -After making this change, execute update-grub to apply the configuration: +After making this change, execute `update-grub` to apply the configuration: ```bash sudo update-grub ``` Reboot the system to activate the settings: + ```bash sudo reboot ``` @@ -51,9 +54,7 @@ Confirm the new command line was used for the last boot: cat /proc/cmdline ``` -The output shows the additions to the kernel command line. - -It is similar to: +The output shows the additions to the kernel command line, and will look something like this: ```output BOOT_IMAGE=/boot/vmlinuz-6.5.0-1020-aws root=PARTUUID=2ca5cb77-b92b-4112-a3e0-eb8bd3cee2a2 ro cma=128 default_hugepagesz=1G hugepagesz=1G hugepages=300 isolcpus=nohz,domain,0-9 irqaffinity=10-95 nohz_full=0-9 rcu_nocbs=0-9 iommu.passthrough=1 transparent_hugepage=madvise console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1 @@ -73,7 +74,9 @@ The output shows the isolated processors: ## Set up the Snort 3 rule set -Download the rule set from https://www.snort.org/ and extract it into your working directory. Start in the `build` directory you used to build Snort. +Download the rule set from https://www.snort.org/ and extract it into your working directory. + +Start in the `build` directory you used to build Snort: ```bash cd $HOME/build @@ -103,7 +106,7 @@ You can obtain PCAP files at: https://www.netresec.com/?page=MACCDC. Visit https://share.netresec.com/s/wC4mqF2HNso4Ten and download a PCAP file. -Copy the file to your working directory, and extract it. If you downloaded a different PCAP file, you might want to change the file name. +Copy the file to your working directory, and extract it. If you downloaded a different PCAP file, you can change the file name. ```bash gunzip maccdc2010_00000_20100310205651.pcap.gz @@ -205,7 +208,7 @@ Run using the command: snort --daq-dir ./snort3/dependencies/libdaq/install/lib/daq --daq-list ``` -The output should look like: +The output should look like this: ```output Available DAQ modules: @@ -269,7 +272,7 @@ The following example shows you how to use multiple Snort threads to analyze PCA MPSE=hyperscan POLICY=./snortrules/lua/snort.lua TCMALLOC_MEMFS_MALLOC_PATH=/dev/hugepages/test snort -c ./snortrules/lua/snort.lua --lua detection.allow_missing_so_rules=true --pcap-filter maccdc2010_00000_20100310205651.pcap --pcap-loop 10 --snaplen 0 --max-packet-threads 10 --daq dump --daq-dir /usr/local/lib/daq --daq-var output=none -H --pcap-dir Pcap -Q --warn-conf-strict --tweaks security ``` -Use `--pcap-loop` to loop PCAP files a number of time, 10 in this example. +Use `--pcap-loop` to loop PCAP files a number of times, 10 in this example. Use `--max-packet-threads` to specify the number of threads, 10 in this example. @@ -297,7 +300,7 @@ The output is similar to: ## Test Snort 3 multithreading to process a single PCAP file -The example usage demonstrates how multithreading increases the number of packets processed per second. +The example demonstrates how multithreading increases the number of packets processed per second. PCAP File Description From acfdafe9a44fdeed4aa67c78a5e78088cf2936d8 Mon Sep 17 00:00:00 2001 From: Maddy Underwood Date: Thu, 19 Dec 2024 05:37:06 +0000 Subject: [PATCH 13/16] Further improvements. --- .../snort3-multithreading/_index.md | 2 +- .../build-and-install.md | 6 ++--- .../snort3-multithreading/usecase.md | 22 +++++++++---------- 3 files changed, 14 insertions(+), 16 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md index d0d007d26..e484d97bf 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md @@ -6,7 +6,7 @@ minutes_to_complete: 45 who_is_this_for: This Learning Path is for software developers familiar with Snort who want to optimize performance by leveraging the benefits of multithreading. learning_objectives: - - Install Snort and all of its dependencies. + - Install Snort and dependencies. - Configure Snort Lua files to enable multithreading. - Use multithreading to process capture files and measure performance. diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md index 8224ab15a..6e611b64a 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md @@ -6,7 +6,7 @@ weight: 2 layout: learningpathall --- -#### Snort 3 +## Snort 3 Snort is an Open Source Intrusion Prevention System (IPS). Snort uses a series of rules to define malicious network activity. If malicious activity is detected, Snort generates alerts. @@ -23,7 +23,7 @@ In order to enable multithreading in Snort 3, specify the quantity of threads de These instructions have been tested on an AWS EC2 Graviton4 instance, based on Arm Neoverse V2. The examples work best if you have at least 16 cores in your system. {{%/notice%}} -## Compile and build Snort 3 +### How do I compile and build Snort 3? To install Snort 3, use a text editor to copy-and-paste the text below and save the script on your Arm server in a file named `install-snort.sh`. @@ -47,7 +47,7 @@ declare -a PACKAGE_URLS=( "https://github.com/gperftools/gperftools/releases/download/gperftools-2.13/gperftools-2.13.tar.gz" ) -downlaodPackages() +downloadPackages() { for url in "${PACKAGE_URLS[@]}"; do # Extract the file name from the URL diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md index 0f3791dfb..605f71f5d 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md @@ -14,7 +14,7 @@ Before testing the Snort 3 multithreading, configure your system by following th * Download the PCAP files. * Adjust Lua configurations. -### Configure Grub settings +#### Configure Grub settings To enable Transparent HugePages (THP) and configure CPU isolation and affinity, append the following line to the `/etc/default/grub file`, modifying the CPU numbers as required: @@ -72,7 +72,7 @@ The output shows the isolated processors: 0-9 ``` -## Set up the Snort 3 rule set +#### Set up the Snort 3 rule set Download the rule set from https://www.snort.org/ and extract it into your working directory. @@ -98,7 +98,7 @@ Copy the `lua` folder from the `snort3` source directory into the rules director cp -r snort3/lua/ Test/snortrules/ ``` -## Download the Packet Capture (PCAP) files +#### Download the Packet Capture (PCAP) files You can use any PCAP files that are relevant to your test scenario. @@ -114,14 +114,14 @@ mkdir Test/Pcap cp maccdc2010_00000_20100310205651.pcap Test/Pcap/ ``` -## Adjust Lua configurations +#### Adjust Lua configurations Now make two modifications to the Lau configurations: * Pin each Snort thread to a unique core, ensuring that the cores match those isolated in the GRUB configuration. * Enable the desired ruleset and enabling profiling. -### Pin Snort Threads to Unique CPU Core +#### Pin Snort Threads to Unique CPU Core Navigate to the `Test/snortrules/lua` directory. @@ -160,7 +160,7 @@ Edit `snort.lua` to include the contents above, and then add in the line below t include('common.lua') ``` -### Modify the snort.lua file to enable rules and profiling +#### Modify the snort.lua file to enable rules and profiling Use an editor to modify the `snort.lua` file. @@ -175,9 +175,7 @@ rules = [[ Continue to edit `snort.lua` and comment out the `profiler` and `latency` lines to enable profiling and packet statistics. -## Review the Snort parameters - -### Modify the IPS policy +#### Review the Snort parameters: modify the IPS policy Snort 3 allows you to fine-tune setups with the `--tweaks` parameter. This feature allows you to use one of Snort's policy files to enhance the detection engine for improved performance or increased security. @@ -190,7 +188,7 @@ Snort 3 includes four preset policy files: The max_detect policy focuses on maximum security, and the connectivity policy focuses on performance and uptime, which might come at the expense of security. -### Specify the data acquisition module +#### Specify the data acquisition module Snort supports DAQ modules which serves as an abstraction layer for interfacing with data source such as network interface. @@ -258,7 +256,7 @@ trace(v1): inline unpriv wrapper For testing, you can use `--daq dump` to analyze Pthe CAP files. -## Spawn Snort 3 process with multithreading +#### Spawn Snort 3 process with multithreading To run Snort 3 with multithreading, start from the `Test` directory. @@ -298,7 +296,7 @@ The output is similar to: 22:52:28 9 97.50 0.00 2.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ``` -## Test Snort 3 multithreading to process a single PCAP file +#### Test Snort 3 multithreading to process a single PCAP file The example demonstrates how multithreading increases the number of packets processed per second. From fda161427af24549482563ee4a815294113eaf8d Mon Sep 17 00:00:00 2001 From: Maddy Underwood Date: Thu, 19 Dec 2024 05:50:30 +0000 Subject: [PATCH 14/16] Fixing Next Steps --- .../snort3-multithreading/_next-steps.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md index 311f62435..2e7b32886 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md @@ -1,11 +1,11 @@ --- -next_step_guidance: You can now try the Learning Path about enabling hyperscan on arm. See the link below. +next_step_guidance: To continue learning about enabling hyperscan on arm,please refer to the learning path provided below. recommended_path: /learning-paths/servers-and-cloud-computing/vectorscan/ further_reading: - resource: - title: Snort 3 Documentation + title: Snort3 Documentation link: https://docs.snort.org/start/ type: documentation - resource: @@ -20,3 +20,5 @@ weight: 21 # set to always be larger than the content in this p title: "Next Steps" # Always the same layout: "learningpathall" # All files under learning paths have this same wrapper --- + + From 58de3b399297c6b0aea5338da418ed1662f72549 Mon Sep 17 00:00:00 2001 From: Maddy Underwood Date: Thu, 19 Dec 2024 05:58:03 +0000 Subject: [PATCH 15/16] Final fix of Next Steps. --- .../snort3-multithreading/_next-steps.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md index 2e7b32886..52ddbd993 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md @@ -1,5 +1,5 @@ --- -next_step_guidance: To continue learning about enabling hyperscan on arm,please refer to the learning path provided below. +next_step_guidance: To continue learning, try this next Learning Path about enabling hyperscan on Arm. recommended_path: /learning-paths/servers-and-cloud-computing/vectorscan/ From c7eb602b24b73488b69d7df03f8b3b7e4584f402 Mon Sep 17 00:00:00 2001 From: Maddy Underwood Date: Thu, 19 Dec 2024 14:25:30 +0000 Subject: [PATCH 16/16] Correct download typo in code. --- .../snort3-multithreading/build-and-install.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md index 6e611b64a..c63057445 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md @@ -96,7 +96,7 @@ installPackages() sudo apt-get install -y $LIST_OF_APPS # required to get optimized result from Snort3 - downlaodPackages + downloadPackages mkdir -p ${ROOT_DIR}/snort3 tar -xzf 3.3.5.0.tar.gz --directory ${ROOT_DIR}/snort3 --strip-components=1 echo "@@@@@@@@@@@@@@@@@@ Installing Snort3 Dependencies ... @@@@@@@@@@@@@@@@@@@@"