diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md index d4b553a23..e484d97bf 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md @@ -1,21 +1,17 @@ --- -title: Scaling Snort 3 - use multithreading for improved performance - -draft: true -cascade: - draft: true +title: Optimize the performance of Snort 3 using multithreading minutes_to_complete: 45 -who_is_this_for: This blog is for engineers familiar with Snort who want to enhance its performance by leveraging the benefits of multithreading. +who_is_this_for: This Learning Path is for software developers familiar with Snort who want to optimize performance by leveraging the benefits of multithreading. learning_objectives: - - Install Snort with all of its dependencies. + - Install Snort and dependencies. - Configure Snort Lua files to enable multithreading. - Use multithreading to process capture files and measure performance. prerequisites: - - An Arm-based instance from a cloud provider or an Arm server running Ubuntu 20.04 or 22.04. + - An Arm-based instance from a cloud provider, or an Arm server running Ubuntu 20.04 or 22.04. - A basic understanding of Snort's operation and configuration. diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md index 5d7e1d691..52ddbd993 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_next-steps.md @@ -1,5 +1,5 @@ --- -next_step_guidance: To continue learning about enabling hyperscan on arm,please refer to the learning path provided below. +next_step_guidance: To continue learning, try this next Learning Path about enabling hyperscan on Arm. recommended_path: /learning-paths/servers-and-cloud-computing/vectorscan/ @@ -20,3 +20,5 @@ weight: 21 # set to always be larger than the content in this p title: "Next Steps" # Always the same layout: "learningpathall" # All files under learning paths have this same wrapper --- + + diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md index 52fd4d778..836a4cf01 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_review.md @@ -7,7 +7,7 @@ review: - It allows Snort to detect encrypted traffic. - It improves packet processing performance. - It enables Snort to be run on legacy hardware. - - It support multiple rule sets at the same time. + - It supports multiple rule sets at the same time. correct_answer: 2 explanation: > It improves packet processing performance by parallelizing tasks. @@ -16,26 +16,25 @@ review: question: > Which parameter is used to enable multithreading in Snort 3? answers: - - --max-packet-threads - - --enable-threads - - --enable-multithreading - - --packet-loop + - --max-packet-threads. + - --enable-threads. + - --enable-multithreading. + - --packet-loop. correct_answer: 1 explanation: > --max-packet-threads parameter is used to enable and configure multithreading. - questions: question: > - In Snort 3, which DAQ (Data Acquisition) module is used to read capture files for packet processing? + In Snort 3, which Data Acquisition (DAQ) module is used to read capture files for packet processing? answers: - - afpacket - - vpp - - dump - - pcap + - afpacket. + - vpp. + - dump. + - pcap. correct_answer: 3 explanation: > - The dump module in Snort 3 is used to read capture files (such as .pcap or .pcapng files) for offline packet analysis. - + The dump module in Snort 3 is used to read capture files, such as .pcap or .pcapng files, for offline packet analysis. # ================================================================================ diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md index 0fd335d0e..35196ca88 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md @@ -1,24 +1,32 @@ --- -title: Installing Snort 3 and the required dependencies + +title: Install Snort 3 and Dependencies weight: 2 ### FIXED, DO NOT MODIFY layout: learningpathall --- -Snort is an Open Source Intrusion Prevention System (IPS). Snort uses a series of rules to define malicious network activity. If malicious activity is found, Snort generates alerts. +## Snort 3 + +Snort is an Open Source Intrusion Prevention System (IPS). Snort uses a series of rules to define malicious network activity. If malicious activity is detected, Snort generates alerts. + +Snort 3 benefits from multithreading, which means that it enables the concurrent processing of multiple packet processing threads with a single Snort instance. This optimization frees up additional memory for further packet processing. + +#### Enable multithreading -Multithreading in Snort 3 refers to the ability to associate multiple threads with a single Snort instance enabling the concurrent processing of multiple packet files. This optimization frees up additional memory for further packet processing. +In order to enable multithreading in Snort 3, specify the quantity of threads designated for processing network traffic using either of these two options: -In order to enable multithreading in Snort 3, specify the number of threads designated for processing network traffic using either the `--max-packet-threads` or `-z` option. +* `--max-packet-threads` +* `-z` {{%notice Note%}} - The instructions provided have been tested on AWS EC2 Graviton4 instance, based on Neoverse V2. The examples are easiest to use if you have at least 16 cores in the system. + These instructions have been tested on an AWS EC2 Graviton4 instance, based on Arm Neoverse V2. The examples work best if you have at least 16 cores in your system. {{%/notice%}} -## Compile and build Snort3 +### How do I compile and build Snort 3? -To install Snort 3, use a text editor to save the script below on your Arm server in a file named `install-snort.sh`. +To install Snort 3, use a text editor to copy-and-paste the text below and save the script on your Arm server in a file named `install-snort.sh`. ``` bash @@ -40,7 +48,7 @@ declare -a PACKAGE_URLS=( "https://github.com/gperftools/gperftools/releases/download/gperftools-2.13/gperftools-2.13.tar.gz" ) -downlaodPackages() +downloadPackages() { for url in "${PACKAGE_URLS[@]}"; do # Extract the file name from the URL @@ -89,7 +97,7 @@ installPackages() sudo apt-get install -y $LIST_OF_APPS # required to get optimized result from Snort3 - downlaodPackages + downloadPackages mkdir -p ${ROOT_DIR}/snort3 tar -xzf 3.3.5.0.tar.gz --directory ${ROOT_DIR}/snort3 --strip-components=1 echo "@@@@@@@@@@@@@@@@@@ Installing Snort3 Dependencies ... @@@@@@@@@@@@@@@@@@@@" @@ -193,21 +201,21 @@ echo 'make sure to source ~/.bashrc or set LD_LIBRARY_PATH using:"' echo ' export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"' ``` -The script takes 2 arguments: -- the directory used to build Snort3 and its dependencies -- the number of processors to use for the build. +The script takes two arguments: +* The directory used to build Snort 3 and its dependencies. +* The number of processors to use for the build. -To build in a new directory named `build` with the number of processors in your system, run the script: +To create a new directory named `build` which lists the number of processors in your system, run the script: ```bash bash ./install-snort.sh build `nproc` ``` -You don't need to run the script as `root` but it assumes you are on Ubuntu 20.04 or 22.04 and have sudo permission. +You do not need to run the script as `root`, but you do need to be running Ubuntu 20.04 or 22.04, and have sudo permission. -When the build completes you have the `snort3` directory with all compiled software, and the `snort` executable is located in `/usr/local/bin`. +When the build completes, you will have the Snort 3 directory with all compiled software, and the `snort` executable will be located in `/usr/local/bin`. -To verify the installation is complete, run the command below and see the version printed: +To verify completed installation, run the command below and look at the version that it prints to screen: ```bash { output_lines = "2-20" } snort -V @@ -228,6 +236,8 @@ To verify the installation is complete, run the command below and see the versio ``` -Don't delete the `build` directory as it will be used in the next step. +{{% notice Note %}} +Do not delete the `build` directory as you will use it in the next step. +{{% /notice %}} -Proceed to learn how to test Snort3 multithreading. \ No newline at end of file +Now you can move on to learn about how to test Snort 3 multithreading. diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md index 918c217ba..aa7200a02 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md @@ -1,26 +1,28 @@ --- -title: Test Snort3 multithreading +title: Test Snort 3 multithreading weight: 3 ### FIXED, DO NOT MODIFY layout: learningpathall --- +## System Configuration -Before testing multithreading performance, perform the following steps to configure your system: +Before testing the Snort 3 multithreading, configure your system by following these steps: -1. Configure the Grub settings -2. Set up the Snort 3 rule set -3. Download the packet capture files -4. Adjust the Lua configurations -5. Review the Snort parameters +* Configure Grub settings. +* Set up the Snort 3 rule set. +* Download the PCAP files. +* Adjust Lua configurations. -## Configure the Grub settings +#### Configure Grub settings -To enable Transparent HugePages (THP) and configure CPU isolation and affinity, append the following line to the /etc/default/grub file: +To enable Transparent HugePages (THP) and configure CPU isolation and affinity, append the following line to the `/etc/default/grub file`, modifying the CPU numbers as required: +{{% notice Note %}} For the total available online CPUs ranging from 0 to 95, with CPUs 0 to 9 pinned to Snort, the grubfile configuration is shown below. +{{% /notice %}} + -Feel free to modify the CPU numbers as needed. ```bash CMDLINE="cma=128" HUGEPAGES="default_hugepagesz=1G hugepagesz=1G hugepages=300" @@ -34,13 +36,13 @@ THP="transparent_hugepage=madvise" GRUB_CMDLINE_LINUX="${CMDLINE} ${HUGEPAGES} ${ISOLCPUS} ${IRQAFFINITY} ${NOHZ} ${RCU} ${MAXCPUS} ${IOMMU} ${THP}" ``` -After making this change, execute update-grub to apply the configuration: +After making this change, execute `update-grub` to apply the configuration: ```bash sudo update-grub ``` -Reboot the system to activate the settings. +Reboot the system to activate the settings: ```bash sudo reboot @@ -52,9 +54,7 @@ Confirm the new command line was used for the last boot: cat /proc/cmdline ``` -The output shows the additions to the kernel command line. - -It is similar to: +The output shows the additions to the kernel command line, and will look something like this: ```output BOOT_IMAGE=/boot/vmlinuz-6.5.0-1020-aws root=PARTUUID=2ca5cb77-b92b-4112-a3e0-eb8bd3cee2a2 ro cma=128 default_hugepagesz=1G hugepagesz=1G hugepages=300 isolcpus=nohz,domain,0-9 irqaffinity=10-95 nohz_full=0-9 rcu_nocbs=0-9 iommu.passthrough=1 transparent_hugepage=madvise console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1 @@ -72,9 +72,11 @@ The output shows the isolated processors: 0-9 ``` -## Set up the Snort 3 rule set +#### Set up the Snort 3 rule set -Download the rule set from https://www.snort.org/ and extract it into your working directory. You should start in the `build` directory you used to build snort. +Download the rule set from https://www.snort.org/ and extract it into your working directory. + +Start in the `build` directory you used to build Snort: ```bash cd $HOME/build @@ -96,16 +98,15 @@ Copy the `lua` folder from the `snort3` source directory into the rules director cp -r snort3/lua/ Test/snortrules/ ``` -## Download the packet capture files +#### Download the Packet Capture (PCAP) files You can use any packet capture (PCAP) files that are relevant to your test scenario. -One place to get PCAP files is: -https://www.netresec.com/?page=MACCDC +You can obtain PCAP files at: https://www.netresec.com/?page=MACCDC. Visit https://share.netresec.com/s/wC4mqF2HNso4Ten and download a PCAP file. -Copy the file to your working directory and extract it, adjust the file name as needed if you downloaded a different PCAP file. +Copy the file to your working directory, and extract it. If you downloaded a different PCAP file, you can change the file name. ```bash gunzip maccdc2010_00000_20100310205651.pcap.gz @@ -113,13 +114,14 @@ mkdir Test/Pcap cp maccdc2010_00000_20100310205651.pcap Test/Pcap/ ``` -## Adjust the Lua configurations +#### Adjust Lua configurations + +Now make two modifications to the Lau configurations: -There are two modifications to the Lau configurations: -- pin each Snort thread to a unique core, ensuring that the cores match those isolated in the GRUB configuration -- enable the desired ruleset and enabling profiling. +* Pin each Snort thread to a unique core, ensuring that the cores match those isolated in the GRUB configuration. +* Enable the desired ruleset and enabling profiling. -### Pin snort threads to unique cpu core +#### Pin Snort Threads to Unique CPU Core Navigate to the `Test/snortrules/lua` directory: @@ -127,7 +129,7 @@ Navigate to the `Test/snortrules/lua` directory: cd Test/snortrules/lua ```` -Use an editor to create a file named `common.lua` with the contents below: +Use an editor to create a file named `common.lua`, and copy-and-paste in the contents below: ```bash ------------------------------------------------------------------------------- @@ -151,14 +153,13 @@ process = { threads = threads } search_engine = { } snort_whitelist_append("threads") ``` - -Include the above file in `snort.lua` by editing the file and adding the line below to the end of the file: +Edit `snort.lua` to include the contents above, and then add in the line below to the end of the file: ``` bash include('common.lua') ``` -### Modify the snort.lua file to enable rules and profiling +#### Modify the snort.lua file to enable rules and profiling Use an editor to modify the `snort.lua` file. @@ -173,21 +174,24 @@ rules = [[ Continue to edit `snort.lua` and comment out the `profiler` and `latency` lines to enable profiling and packet statistics. -## Review the Snort parameters - -### Modify the IPS policy +#### Review the Snort parameters: modify the IPS policy Snort 3 allows you to fine-tune setups with the `--tweaks` parameter. This feature allows you to use one of Snort's policy files to enhance the detection engine for improved performance or increased security. -Snort 3 includes four preset policy files: `max_detect`, `security`, `balanced`, and `connectivity`. +Snort 3 includes four preset policy files: + +* Max_detect. +* Security. +* Balanced. +* Connectivity. -The `max_detect` policy favors maximum security, whereas the `connectivity` policy focuses on performance and uptime, which may come at the expense of security. +The max_detect policy focuses on maximum security, and the connectivity policy focuses on performance and uptime, which might come at the expense of security. -### Specify the data acquisition module +#### Specify the data acquisition module Snort supports data acquisition (DAQ) modules which serve as an abstraction layer for interfacing with a data source such as a network interface. -To see list of DAQ modules supported by Snort use `--daq-list` command. +To see list of DAQ modules supported by Snort use the `--daq-list` command. Return to the `build` directory: @@ -201,7 +205,7 @@ Run Snort with the command: snort --daq-dir ./snort3/dependencies/libdaq/install/lib/daq --daq-list ``` -The output is: +The output should look like this: ```output Available DAQ modules: @@ -249,23 +253,22 @@ trace(v1): inline unpriv wrapper file - Filename to write text traces to (default: inline-out.txt) ``` -For testing, you can use `--daq dump` to analyze PCAP files. +For testing, you can use `--daq dump` to analyze Pthe CAP files. -## How do I spawn a Snort 3 process with multithreading? +#### Spawn Snort 3 process with multithreading -To run Snort 3 with multithreading start from the `Test` directory. +To run Snort 3 with multithreading, start from the `Test` directory. ```bash cd $HOME/build/Test ``` - -The following example shows how to use multiple Snort threads to analyze PCAP files: +The following example shows you how to use multiple Snort threads to analyze PCAP files. ``` bash MPSE=hyperscan POLICY=./snortrules/lua/snort.lua TCMALLOC_MEMFS_MALLOC_PATH=/dev/hugepages/test snort -c ./snortrules/lua/snort.lua --lua detection.allow_missing_so_rules=true --pcap-filter maccdc2010_00000_20100310205651.pcap --pcap-loop 10 --snaplen 0 --max-packet-threads 10 --daq dump --daq-dir /usr/local/lib/daq --daq-var output=none -H --pcap-dir Pcap -Q --warn-conf-strict --tweaks security ``` -Use `--pcap-loop` to loop PCAP files a number of time, 10 in this example. +Use `--pcap-loop` to loop PCAP files a number of times, 10 in this example. Use `--max-packet-threads` to specify the number of threads, 10 in this example. @@ -290,10 +293,9 @@ The output is similar to: 22:52:28 8 98.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 22:52:28 9 97.50 0.00 2.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ``` +#### Test Snort 3 multithreading to process a single PCAP file -## How do I test Snort 3 multithreading to process a single pcap file? - -The example usage demonstrates how multithreading increases the number of packets processed per second. +The example demonstrates how multithreading increases the number of packets processed per second. PCAP File Description @@ -308,4 +310,4 @@ Performance results | 1 | 940960 | 91.777964 | | 10 | 9406134 | 9.181182 | -The results demonstrate how increasing the thread count by ten times results in a ten times increase in packets processed per second, while reducing the execution time by ten times. \ No newline at end of file +The results demonstrate how increasing the thread count by ten times results in a ten times increase in packets processed per second, while reducing the execution time by ten times.