Skip to content
This repository has been archived by the owner on Apr 6, 2021. It is now read-only.

Using nix(DevOps) to deploy Network Security Monitoring System on Debian

Notifications You must be signed in to change notification settings

hardenedlinux/debian-nix-manager

Repository files navigation

Hardenedlinux User’s Debian nix home-manager configuration

1 Install nix

wget https://github.com/hardenedlinux/nix-fpm-multiuser/releases/download/Jan/nix_3.0.pre19700104.master_amd64.deb
sudo dpkg -i nix_3.0.pre19700104.master_amd64.deb
nix-channel --add https://github.com/NixOS/nixpkgs/archive/75d69a0a27ee9262805e6a7a605f00e654b75a28.tar.gz nixpkgs
nix-channel --update
  • then relaod bash to load nix executable
  • check nix-daemon status (if the task status is inactive to restart it first)
    systemctl start nix-daemon.service
    systemctl enable nix-daemon.service
    systemctl status nix-daemon.service
        
    ● nix-daemon.service - Nix Daemon
         Loaded: loaded (/lib/systemd/system/nix-daemon.service; disabled; vendor preset: enabled)
         Active: active (running) since Wed 2020-09-30 21:50:11 EDT; 1min 16s ago
       Main PID: 22027 (nix-daemon)
          Tasks: 7 (limit: 4915)
         Memory: 3.5M
         CGroup: /system.slice/nix-daemon.service
                 └─22027 /nix/store/66n31il66pmf9hnd9yccmv42xfi0fm4p-nix-3.0pre19700101_dirty/bin/nix-daemon --daemon
        
nix-shell -p 'git'
git clone https://github.com/hardenedlinux/debian-nix-manager.git ~/.config/nixpkgs/
cd ~/.config/nixpkgs/
nix-shell
  • Authorized User
  • use make by nix own. make sure you have nix/bin/path/ on sudoers
  • /etc/sudoers
# echo $HOME/.nix-profile/bin
# /home/test/.nix-profile/bin

Defaults        secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/test/.nix-profile/bin"
  • set User’s NOPASSWORD to sudoers
test   ALL=(ALL:ALL) NOPASSWD: ALL

2 using flake to shell environment

Edit either /.config/nix/nix.conf or /etc/nix/nix.conf and add:

experimental-features = nix-command flakes

then run:

nix develop

3 Install home-manager

nix-env -i home-manager

add home-manager channel

nix-channel --add https://github.com/rycee/home-manager/archive/master.tar.gz home-manager
nix-channel --update
  • For China’s region users

set substituters to /.config/nix/nix.conf

substituters = https://mirrors.tuna.tsinghua.edu.cn/nix-channels/store https://cache.nixos.org

4 Build

  • register your own password.json
    mv ./secrets/password.json ./secrets/password-old.json
    cp ./lib/password/password-example.json ./secrets/password.json
        
home-manager switch
  • set priority for home-manager
nix-env --set-flag priority 10 home-manager
##setting flag on 'home-manager-2020-03-17'
  • set zsh as default shell
chsh --shell /home/$USER/.nix-profile/bin/zsh $USER

4.1 Or using hardenedlinux Hydra CI binary cache speed your building process

home-manager switch --option substituters "https://cache.nixos.org http://221.4.35.244:8301" --option trusted-public-keys "221.4.35.244:3ehdeUIC5gWzY+I7iF3lrpmxOMyEZQbZlcjOmlOVpeo="

5 Install cachix

nix-env -iA cachix -f https://cachix.org/api/v1/install
sudo mkdir -p /etc/nix
echo "trusted-users = root $USER" | sudo tee -a /etc/nix/nix.conf
sudo pkill nix-daemon
cachix use nsm-data-analysis
sudo systemctl restart nix-daemon.service

5.1 Build nixpkgs-hardenedlinux

clone https://github.com/hardenedlinux/nixpkgs-hardenedlinux

git clone https://github.com/hardenedlinux/nixpkgs-hardenedlinux
cd nixpkgs-hardenedlinux/
nix-build --option substituters "https://cache.nixos.org https://nsm-data-analysis.cachix.org"

6 Start pkgs service

systemctl --user start vast.service
systemctl --user status vast.service
● vast.service
   Loaded: loaded (/nix/store/59sx0prx1fi93653kkgcsdr4schqa7bv-vast.service/vast.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-03-13 03:17:56 EDT; 1h 9min ago
 Main PID: 28612 (vast)
   CGroup: /user.slice/user-1000.slice/user@1000.service/vast.service
           └─28612 /nix/store/n6vm1zifpl65445k6w866sf109j2imwm-vast/bin/vast -c /nix/store/kg43s4bdarkg3g79kxii3h9cmbfym2sj-vast.conf start

7 Deploy zeek and Model

  • Deploy it with Sytemd

    Zeek Service Demo

services.zeek = {
  enable = true;
  standalone = true;
  interface = "enp0s3";
  listenAddress = "localhost";
  privateScript = ''
  @load /home/gtrun/project/hardenedlinux-zeek-script/scripts/zeek-query.zeek
  '';
};
systemctl --user start zeek.service
  • Check status
sudo zeekctl status

DONE FOR TEST

7.1 Zeek PLugin postgresql

Default enable Kafka and PsotgreSqll alread in nix of Zeek.

Johanna::PostgreSQL - PostgreSQL log writer and input reader (dynamic, version 0.2.0)

8 Database

8.1 postgresql

systemctl --user start  postgresql.service
systemctl --user status  postgresql.service
● postgresql.service
   Loaded: loaded (/nix/store/32xm7dcwlnjais6b42iaa8jh4zkfc3ji-postgresql.service/postgresql.service; linked; vendor preset: enabled)
   Active: active (running) since Sun 2020-03-29 23:11:28 EDT; 15min ago
 Main PID: 3542 (mp0sg0q78h9bwa0)
   CGroup: /user.slice/user-1000.slice/user@1000.service/postgresql.service
           ├─3542 /nix/store/828g2nqfgivscv79xykkmgjk0znll08l-bash-4.4-p23/bin/bash -e /nix/store/mp0sg0q78h9bwa0z45x4n4alc0ffg24f-run-postgresql
           ├─3551 /nix/store/gl7xj33j9fsklbwlgwlgdw6ggj57l7fh-postgresql-11.7/bin/postgres -k /var/db/postgresql/11
           ├─3563 postgres: checkpointer
           ├─3564 postgres: background writer
           ├─3565 postgres: walwriter
           ├─3566 postgres: autovacuum launcher
           ├─3567 postgres: stats collector
           └─3568 postgres: logical replication launcher

9 Deploy osquery

9.1 enable osquery Service

sudo mkdir -p /var/osquery/log
sudo chown $USER /var/osquery
systemctl --user status osquery.service
● osquery.service
   Loaded: loaded (/nix/store/mxpjazyy6b4hymxk9hkivfs1kqk7jvly-osquery.service/osquery.service; linked; vendor preset: enabled)
   Active: active (running) since Fri 2020-03-27 02:49:17 EDT; 37s ago
 Main PID: 26822 (osqueryd)
   CGroup: /user.slice/user-1000.slice/user@1000.service/osquery.service
           ├─26822 /nix/store/acx6mvslzxbzw7fyl4nr87m9pybb9wmn-osquery-4.2.0/bin/osqueryd --database_path /var/osquery/osquery.db --logger_path /var/osquery/log --pidfile /var/osquery/osqueryd.pidfile --database_path /var/osquery/osquery.db --extensions_socket /var/osquery/osquery.em --config_path /home/test/.osquery/osquery.conf
           └─26841 /nix/store/acx6mvslzxbzw7fyl4nr87m9pybb9wmn-osquery-4.2.0/bin/osqueryd

10 Deploy ELK

10.1 Start Elastic service

create /var/lib/elasticsearch/ and make yourself the owner of that directory

sudo mkdir -p /var/lib/elasticsearch/
sudo chown $USER /var/lib/elasticsearch/

start service

systemctl --user start elasticsearch.service
systemctl --user status elasticsearch.service
● elasticsearch.service
   Loaded: loaded (/nix/store/8dncyqmv46xa6j3cr52czs3ky86nsiyh-elasticsearch.service/elasticsearch.service; linked; vendor preset: enabled)
   Active: active (running) since Mon 2020-03-23 19:37:34 EDT; 8min ago
 Main PID: 24715 (java)
   CGroup: /user.slice/user-1000.slice/user@1000.service/elasticsearch.service
           ├─24715 /nix/store/8wmf6apz3yss4vz67z6xdwhhd08yz4cb-openjdk-headless-8u222-ga-jre/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-5006850798322202895 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/var/lib/elasticsearch -Des.path.conf=/var/lib/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=tar -cp /nix/store/6czj00nnxdzr18by4n3rqlfcp0csak0b-elasticsearch-6.8.3/lib/* org.elasticsearch.bootstrap.Elasticsearch
           └─24810 /var/lib/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

11 Performance

11.1 PF_RING

sudo apt-get update
sudo apt install linux-headers-$(uname -r)
nix-shell ~/.config/nixpkgs/pkgs/network/pf_ring_model.nix --command "sudo make install && sudo modprobe pf_ring"
  • Output test
modinfo pf_ring && cat /proc/net/pf_ring/info
filename:       /lib/modules/4.19.0-8-amd64/kernel/net/pf_ring/pf_ring.ko
alias:          net-pf-27
version:        7.6.0
description:    Packet capture acceleration and analysis
author:         ntop.org
license:        GPL
srcversion:     A80A92A0F9D4CB8168B549A
depends:
retpoline:      Y
name:           pf_ring
vermagic:       4.19.0-8-amd64 SMP mod_unload modversions
parm:           min_num_slots:Min number of ring slots (uint)
parm:           perfect_rules_hash_size:Perfect rules hash size (uint)
parm:           enable_tx_capture:Set to 1 to capture outgoing packets (uint)
parm:           enable_frag_coherence:Set to 1 to handle fragments (flow coherence) in clusters (uint)
parm:           enable_ip_defrag:Set to 1 to enable IP defragmentation(only rx traffic is defragmentead) (uint)
parm:           quick_mode:Set to 1 to run at full speed but with upto one socket per interface (uint)
parm:           force_ring_lock:Set to 1 to force ring locking (automatically enable with rss) (uint)
parm:           enable_debug:Set to 1 to enable PF_RING debug tracing into the syslog, 2 for more verbosity (uint)
parm:           transparent_mode:(deprecated) (uint)
PF_RING Version          : 7.6.0 (unknown)
Total rings              : 0

Standard (non ZC) Options
Ring slots               : 4096
Slot version             : 17
Capture TX               : Yes [RX+TX]
IP Defragment            : No
Socket Mode              : Standard
Cluster Fragment Queue   : 0
Cluster Fragment Discard : 0

Releases

No releases published

Packages

No packages published