Summary

Hint
Notes
License

Hint

Please check the Github repository if a code block isn't displayed properly.

Notes

Linux

2017-12-06: Screenfetch

Screenfetch is a bash tool to display system information.

#: Install screenfetch
neikei@workstation:~$ sudo apt install screenfetch

#: Add the following line to the end of your ~/.bashrc or ~/.zshrc
if [ -f /usr/bin/screenfetch ]; then echo ""; screenfetch; echo ""; fi

#: Example output during shell start

                          ./+o+-       neikei@workstation
                  yyyyy. 'yyyyyy+      OS: Ubuntu 16.04 xenial
              .;//+/////h yyyyyyo      Kernel: x86_64 Linux 4.4.0-67-generic
           .++ .:/++++++/-.`sss/`      Uptime: 5h 41m
         .:++o: `\++++++++/:---:/-     Packages: 2273
        o:+o+:++. `````'-/ooo+++++\    Shell: zsh 5.1.1
       .:+o:+o/.          `+sssooo+\   Resolution: 5760x1200
  .++/+ +oo+o:`             \sssooo;   DE: GNOME
 /+++//+: oo+o               ```````   WM: GNOME Shell
 \+/+o+++ o++o               ydddhh+   WM Theme:
  .++.o+ +oo+:`             /dddhhh;   GTK Theme: Adwaita [GTK2/3]
       .+.o+oo:.           oddhhhh+    Icon Theme: Adwaita
        \+.++o+o` -,,,,.:ohdhhhhh+     Font: Cantarell 11
         `:o+++  ohhhhhhhhyo++os:      Disk: 154G / 242G (68%)
           .o: .syhhhhhhh'.oo++o.      CPU: Intel Core i7-6600U @ 4x 3.4GHz
               /osyyyyyyy.oooo+++\     RAM: 6102MiB / 15464MiB
                   ````` +oo+++o:/
                          `oo++'`

2017-11-27: Locate, partition and create filesystem on new disk

History of the attachment of a new disk to a CentOS 7.4 VM.

#: Compare the list of available devices and mounted devices to locate the new device
[root@localhost ~]# echo "Available devices:" && ls /dev/sd? | sort && echo "Available partitions:" && ls /dev/sd?? | sort
Available devices:
/dev/sda
/dev/sdb
/dev/sdc
Available partitions:
/dev/sda1
/dev/sda2
/dev/sdb1
#: /dev/sdc is the new device, because it has no available partition yet

#: Check available partitions
[root@localhost ~]# fdisk /dev/sdc
Command (m for help): p

Disk /dev/sdc: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7b407523

   Device Boot      Start         End      Blocks   Id  System

#: Create new partition
Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-16777215, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-16777215, default 16777215):
Using default value 16777215
Partition 1 of type Linux and of size 8 GiB is set

#: Write the partition to the disk
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

#: Re-check available partitions
Command (m for help): p

Disk /dev/sdc: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7b407523

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048    16777215     8387584   83  Linux

#: Create a filesystem on the new created partition
[root@localhost ~]# mkfs -t ext4 /dev/sdc1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
524288 inodes, 2096896 blocks
104844 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2147483648
64 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

#: Mount the new filesystem
[root@localhost ~]# mount /dev/sdc1 /mnt/

#: Add the new created partition to /etc/fstab file if it should be mounted during the boot process

2017-11-22: System CLI monitoring

CLI tools for monitoring the system.

#: Overview of CPU, RAM, load and processes
top

#: Like top but with interactive fitlers and coloured
htop

#: MySQL processes
mytop

#: Disk usage in human-readable format
df -h

#: Read and write operations
iotop

#: Network traffic
iftop

#: Network packets
iptraf

#: Dump network traffic
tcpdump

2017-11-21: System versions

Snippets to check system versions on debian based systems.

#: Show kernel version
neikei@workstation:~$ uname -r
4.4.0-67-generic

#: Show debian version
neikei@workstation:~$ cat /etc/debian_version
stretch/sid

#: Show os release
neikei@workstation:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

#: Use lsb_release to check os version
neikei@workstation:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:    16.04
Codename:   xenial

2017-11-11: Search and replace strings in files

Just some basic snippets to search and replace in text files e.g. config files with sed and vim.

sed

#: Search and replace in the whole file
sed -i -e 's/search/replace/g' file.txt

vim

#: Open your file with vim
vim file.txt

#: Search and replace only next match after the current cursor position
:s/search/replace/g

#: Search and replace in the whole file
:%s/search/replace/g

2017-10-28: Pin ansible apt package

The simplest solution is apt-mark, but the solution with apt preferences is more flexible.

apt-mark: Prevent the package from being automatically installed, upgraded or removed.

#: Set ansible package to hold
sudo apt-mark hold ansible

#: Show packages on hold
sudo apt-mark showhold
   ansible

#: Set ansible package to unhold
sudo apt-mark unhold ansible

apt preferences: Pin the package to a specific version, but allow apt to update the package with patches.

#: Pin ansible package
echo "Package: ansible
Pin: version 2.1.*
Pin-Priority: 1000" | sudo tee /etc/apt/preferences.d/ansible

#: Unpin ansible package
sudo rm /etc/apt/preferences.d/ansible

2017-10-28: Install specific apt package version

Example of the ansible package installation on Ubuntu.

#: Check available ansible packages
sudo apt-cache madison ansible
   ansible | 2.4.1.0-1ppa~xenial | http://ppa.launchpad.net/ansible/ansible/ubuntu xenial/main amd64 Packages
   ansible | 2.4.1.0-1ppa~xenial | http://ppa.launchpad.net/ansible/ansible/ubuntu xenial/main i386 Packages
   ansible | 2.1.1.0-1~ubuntu16.04.1 | http://de.archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages
   ansible | 2.1.1.0-1~ubuntu16.04.1 | http://de.archive.ubuntu.com/ubuntu xenial-backports/universe i386 Packages

#: Install the required ansible package
sudo apt install ansible=2.1.1.0-1~ubuntu16.04.1

2017-10-20: SystemD multi spawn processes

This great documentation on StackExchange explains how to spawn multi processes with SystemD.

2017-10-15: Ubuntu optimize jpeg images

The jpegoptim tool optimizes jpg images and compresses them without loss of quality.

#: Install jpegoptim
sudo apt install jpegoptim

#: Optimize a single image. Caution: this will overwrite the original image.
jpegoptim ./pictures/testimage.jpg
./Pictures/testimage.jpg 3120x4160 24bit N Exif  [OK] 841351 --> 729471 bytes (13.30%), optimized.

#: Optimize a single file and store it in a different directory to keep the original image.
jpegoptim ./pictures/testimage.jpg --dest="/tmp/"
./Pictures/testimage.jpg 3120x4160 24bit N Exif  [OK] 841351 --> 729471 bytes (13.30%), optimized.

#: Optimize images of a find result
find . -type f -regextype posix-extended -regex "^.*\.(jpg|jpeg|JPG|JPEG)$" -exec jpegoptim {} \;
./Pictures/testimage1.jpg 3120x4160 24bit N Exif  [OK] 841351 --> 729471 bytes (13.30%), optimized.
./Pictures/testimage2.jpg 3120x4160 24bit N Exif  [OK] 841351 --> 729471 bytes (13.30%), optimized.

2017-10-04: Debian ntpdate time synchronization

Configure ntpdate to synchronize the time with the internet hourly. Especially recommended for VMs in a network without a self-hosted ntp server.

#: Remove ntp if it is installed
sudo apt remove ntp

#: Install ntpdate
sudo apt install ntpdate

#: Configure cron to resync the time hourly
echo $'#!/bin/bash\n/usr/sbin/ntpdate -s pool.ntp.org' | sudo tee /etc/cron.hourly/ntpdate
sudo chmod +x /etc/cron.hourly/ntpdate

2017-08-21: Database replication status

Bash snippets to check the database replication.

#: MySQL slave status
mysql -e "SHOW SLAVE STATUS \G"

#: MySQL slave status as watch to refresh every 2 seconds
watch -n 2 'mysql -e "SHOW SLAVE STATUS \G"'

#: MySQL replication lag only
mysql -e 'SHOW SLAVE STATUS \G' | grep Seconds_Behind_Master

#: PostgreSQL replication lag only
cd /tmp && sudo -u postgres psql -d postgres -c "SELECT CASE WHEN pg_last_xlog_receive_location() = pg_last_xlog_replay_location()
    THEN 0
    ELSE EXTRACT (EPOCH FROM now() - pg_last_xact_replay_timestamp())
END AS replication_lag_in_seconds;"

2017-08-14: Validate archives

Bash snippets to validate the most common archives.

#: Validate .tar.gz archives
tar -tzf archive.tar.gz >/dev/null

#: Validate .tar archives
tar -tf archive.tar >/dev/null

#: Validate .gz archives
gzip -t archive.gz

#: Validate .bz2 archives
bzip2 -t archive.bz2

#: Validate .zip archives
zip -T archive.zip

2017-08-12: Install Oracle Java 8 on Debian Stretch

#: Install the network service dirmngr to manage certificate servers
apt install dirmngr

#: Add the repository and the repository key
echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" > /etc/apt/sources.list.d/webupd8team-java.list
echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" >> /etc/apt/sources.list.d/webupd8team-java.list
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886

#: Update package informations
apt update

#: Install Java
apt install oracle-java8-installer

2017-07-16: Command line convenience

Some useful code snippets to increase the convenience of command line tools.

~/.bashrc

Prevent inadvertently crontab deletions
Add execution time to bash history

function crontab {
    if [[ $* == *"-r"* ]];
    then
        echo "INFO: crontab -r is blocked to prevent inadvertently crontab deletions."
    else
        /usr/bin/crontab $*
    fi
}

HISTTIMEFORMAT="%d.%m.%y %T "

~/.inputrc

Bash history search with page up and down

"\e[5~": history-search-backward
"\e[6~": history-search-forward

~/.vimrc

Color theme
Line numbers
Syntax highlighting

" Color theme for dark backgrounds
:color desert

" Show line numbers
set number

" Enable syntax highlighting
syntax on

2017-07-15: Fix locale warnings on Linux servers

Problem: Warnings about wrong or missing locale configurations

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "de_DE.UTF-8",
LC_ADDRESS = "de_DE.UTF-8",
LC_MONETARY = "de_DE.UTF-8",
LC_NUMERIC = "de_DE.UTF-8",
LC_TELEPHONE = "de_DE.UTF-8",
LC_IDENTIFICATION = "de_DE.UTF-8",
LC_MEASUREMENT = "de_DE.UTF-8",
LC_CTYPE = "en_US.UTF-8",
LC_TIME = "de_DE.UTF-8",
LC_NAME = "de_DE.UTF-8",
LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").

Solution 1: dpkg-reconfigure: Use dpkg-reconfigure to configure LC_ALL

sudo dpkg-reconfigure locales

Solution 2: /etc/default/locale: Set LC_ALL in the default local configuration

sudo su -
echo LC_ALL="en_US.UTF-8" >> /etc/default/locale

2017-07-15: summarizeMD

summarizeMD is a ruby script to create a table of contents for Markdown files. The short bash script will merge all Markdown files before the run of summarizeMD.rb.

Windows

2018-04-14: Kubernetes Minikube

Minikube is a local kubernetes environment. The following snippet shows the installation via Chocolatey and requires Virtualbox or Hyper-V installed on your computer.

#: Start Powershell as administrator

#: Install Minikube and kubernetes-cli
choco install minikube --version 0.25.2

#: Start Minikube (starts a VM in Virtualbox with pre-installed Kubernetes)
minikube start

#: Start and open Kubernetes Dashboard in webbrowser
minikube dashboard # wait a few minutes if it doesn't work at the first try

#: Stop Minikube (stops the VM in Virtualbox)
minikube stop

2018-03-13: Chocolatey basics

Chocolatey is a package manager for Windows. The following snippets are just basics for the installation of my required tools and the complete documentation is really useful.

#: Start Powershell as administrator

#: Install chocolatey
PS C:\WINDOWS\system32> Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

#: Search a package like vagrant and show all available versions
PS C:\WINDOWS\system32> choco search vagrant --all
Chocolatey v0.10.8
...
vagrant 2.0.2 [Approved] Downloads cached for licensed users
vagrant 2.0.1 [Approved] Downloads cached for licensed users
vagrant 2.0.0 [Approved] Downloads cached for licensed users
vagrant 1.9.8 [Approved] Downloads cached for licensed users
vagrant 1.9.7 [Approved]
...
87 packages found.

#: Install a package
PS C:\WINDOWS\system32> choco install vagrant --version 2.0.1 -y

#: Install all required tools at once
PS C:\WINDOWS\system32> choco install keepassx git visualstudiocode sourcetree virtualbox vagrant -y

#: Upgrade all installed packages expect vagrant
PS C:\WINDOWS\system32> choco upgrade all --except="'vagrant'"

2018-01-12: Shortcuts for virtual desktops

Shortcut	Description
Windows + Ctrl + D	Open a new virtual Desktop
Windows + Ctrl + Arrow	Switch between existing virtual Desktops
Windows + Ctrl + F4	Close the current virtual Desktop

Source: Microsoft Windows Blog

2018-01-07: Use Ubuntu Bash as Visual Studio Code Terminal

Add the following line to the user settings. Install Ubuntu Bash if you haven't it done before.

"terminal.integrated.shell.windows": "C:\\WINDOWS\\System32\\bash.exe"

2018-01-03: ASUS Zenbook UX310UA FanControl

Download and install the newest version of NoteBook FanControl
Start NoteBook FanControl
Select the required config
Set the Fan control service status to enabled
Enable autostart in the settings

2017-12-26: Linux system administrator tools

PuTTY / mRemoteNG
Visual Studio Code
- Extensions:
- vscode-icons
- markdownlint
- Ansible
- Git Blame
SourceTree
Ubuntu Bash
KeePassX
Virtualbox, Git and Vagrant

Visual Studio Code line endings on Windows are CRLF. Change it to LF if you want to use your code on Linux/Unix systems.

2017-12-15: Ubuntu Bash installation on Windows 10

Enable "Windows Subsystem for Linux" in the "Windows Features" settings
Restart Windows
Install "Ubuntu" from the "Microsoft Store"
Start Ubuntu

Installing, this may take a few minutes...
Installation successful!
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username: neikei
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Default UNIX user set to: neikei
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

neikei@workstations:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Continuous Integration & Continuous Delivery

2018-06-07: Configuration management

Configuration management is an exciting topic in computer science. The following graphic gives a small insight into how configuration management can be implemented with iTop as CMDB, Ansible for the provisioning and Check_MK for the monitoring. There are further open source projects like itop-utilities to export the data of the CMDB and transfer it to Ansible readable data.

Infrastructure overview

                              -----------
                      ------->| Ansible |--------
                     |        -----------        |
-------------        |                           |        ------------------
| iTop CMDB |--------                             ------->| Managed server |
-------------        |                           |        ------------------
                     |        ------------       |
                      ------->| Check_MK |--------
                              ------------

2018-04-16: Selfstudy Docker

Portainer is a simple GUI to manage Docker containers. A Vagrantbox to test the webinterface is available here.

Minikube is a local kubernetes environment. Kubernetes is more complex, but has tons features and is used by the global players. Read here how to start the testing environment on Windows.

2018-01-30: DevOps learnings 2017

A few notes after one year as System Engineer DevOps.

Define a standard server and make the configuration reproducible
- Ansible
- Puppet
- Chef
Create local environments for the developer
- Vagrant
- Docker
Manage the Code with Git
- Bitbucket
- Gitlab
Run the build process after every commit on the build server
Make build processes transparent and manage them in the software repository
Use frameworks for the deployment
- Deployer
- Ansistrano
Most common buzzwords in our company in the last year
- Continuous integration
- Continuous delivery
- Continuous testing
- Cloud environments
- XaaS

2018-01-18: Makefile for WebApps

Makefile example for a PHP web application developed on Debian 9.

#: Required packages: sudo apt install git unzip php-cli python ruby composer

#: Configuration
SHELL := /bin/bash
BRANCH := $(shell git rev-parse --abbrev-ref HEAD)

#: Target groups
.PHONY: build
build: precheck composer syntaxcheck test

.PHONY: artifact
artifact: cleanup archive

#: Targets
.PHONY: precheck
precheck:
	@echo "==> Precheck started."
	which git
	which unzip
	which php
	which python
	which ruby
	which composer
	@echo -e "==> Precheck finished.\n"

.PHONY: composer
composer:
	@echo "==> Composer started."
	composer install --no-interaction
	@echo -e "==> Composer finished.\n"

.PHONY: syntaxcheck
syntaxcheck:
	@echo "==> Syntaxcheck started."
	wget https://github.com/neikei/syntaxchecks/archive/master.zip
	unzip master.zip && rm master.zip
	syntaxchecks-master/syntaxchecks.sh -p "`pwd`" -a -s
	rm -rf syntaxchecks-master
	@echo -e "==> Syntaxcheck finished.\n"

.PHONY: test
test:
	@echo "==> Syntaxcheck started."
	# Add tests here
	@echo -e "==> Syntaxcheck finished.\n"

.PHONY: cleanup
cleanup:
	@echo "==> Cleanup started."
	# Add cleanup tasks here
	@echo -e "==> Cleanup finished.\n"

.PHONY: archive
archive:
	@echo "==> Archive started."
	tar cfz `date +%Y%m%d%H%M%S`_${BRANCH}.tar.gz * --exclude=test-reports
	@echo -e "==> Archive finished.\n"

2017-12-21: Bamboo JUnit Parser fake results for hotfixes

Bamboo task to create fake testresults for the JUnit Parser if a hotfix build is running without the execution of tests.

#!/bin/bash

#: Pre-Check
is_hotfix=`git rev-parse --abbrev-ref HEAD | grep "hotfix" | wc -l`

#: Run Coveragechecks
if [ $is_hotfix -eq 1 ]; then

echo '<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="Hotfix Fake Test Suite" tests="1" assertions="0" failures="0" errors="0" time="0.01">
      <testcase name="Hotfix Fake Test"/>
  </testsuite>
</testsuites>' > phpunit_hotfix_fake_results.xml

fi

2017-11-23: Common HTTP status codes

Code	Description
200	OK
201	Created
204	No content
301	Moved permanently
304	Not modified
307	Temporary redirect
400	Bad request
401	Unauthorized
403	Forbidden
404	Not found
405	Method not allowed
500	Internal server error
502	Bad Gateway
504	Gateway timeout

Great overview with further explanations.

2017-10-26: Bamboo branch based configs

Bamboo task for branch based configuration replacements. The following example will only replace the config if the current branch is develop.

#!/bin/bash

current_branch_name=`git rev-parse --abbrev-ref HEAD`
echo "==> Branch: $current_branch_name"

if [[ $current_branch_name =~ ^develop$ ]]; then

    sed -i "s/placeholder/value/g" app/config/parameters.yml.dist

fi

2017-09-26: Jenkins centralized syntaxchecks

Installation of syntaxchecks for web applications on a Linux Jenkins server.

#: Install syntaxchecks in the home path of the jenkins application user
su - jenkins
cd ~
git clone https://github.com/neikei/syntaxchecks.git

Jenkins task to execute the sytaxchecks during the build process.

#: Execute syntaxchecks to validate all changed files in the last commit and stop in case of an error
~/syntaxchecks/syntaxchecks.sh -p "`pwd`" -c 1 -s

2017-09-19: Bamboo centralized syntaxchecks

Installation of syntaxchecks for web applications on a Linux Bamboo server.

#: Install syntaxchecks in the home path of the bamboo application user
su - bamboo
cd ~
git clone https://github.com/neikei/syntaxchecks.git

Bamboo task to execute the sytaxchecks during the build process.

#: Execute syntaxchecks to validate all changed files in the last commit and stop in case of an error
~/syntaxchecks/syntaxchecks.sh -p "${bamboo.build.working.directory}" -c 1 -s

2017-09-18: Deploy archives with Bamboo and PHP Deployer

Bamboo task to create the archive during the deployment process.

#: Create tarball and remove archived files
tar cfz artifact.tar.gz * --remove-files

PHP Deployer task to extract the archive on a remote server.

// Extract archive on a remote server
desc('Extract archive');
task('archive:extract', function () {
    run('cd {{ release_path }} && tar xf artifact.tar.gz && rm artifact.tar.gz');
});

// Deploy process
desc('Deploy');
task('deploy', [
    'deploy:prepare',
    'deploy:lock',
    'deploy:release',
    'rsync',
    'archive:extract',
    'deploy:symlink',
    'deploy:unlock',
    'cleanup',
    'success'
]);

2017-09-15: Use PHP Deployer with Bamboo

Use PHP Deployer to simplify Bamboo deployments to a bunch of servers.

PHP Deployer placement: Place the scripts in the home directory of the bamboo application user in a subfolder named "deployer" and create an own subfolder for every deployment project.

~/deployer/$project/deploy.php
~/deployer/$project/composer.json

Hint: "composer update" is needed in every project directory.

Bamboo task to execute PHP Deployer scripts during the deployment process.

#: Config
project=<projectname>
stage=<stage>

#: Deployment
logfile="/tmp/$project-$stage.log"

#: Change into the directroy of your deployer scripts and start the deployment with the build directory as parameter
cd ~/deployer/$project
php vendor/bin/dep deploy $stage --artifact_directory="${bamboo.build.working.directory}" -vv 2>&1 | tee $logfile

#: Check Result
errors=`grep "\[FatalException\]\|RuntimeException\|\[Error\]\|Fatal error\|Exception trace\|General error" $logfile | wc -l`
if [ $errors -ne 0 ]; then
    echo "==> Deployer failed... Check the output above.";
    exit 1;
else
    echo "==> Deployer finished.";
fi

PHP Deployer modifications to use a Bamboo build directory for the deployment.

// Include Symfony component for input options
use Symfony\Component\Console\Input\InputOption;

// Get artifact directory from start parameters
option('artifact_directory', null, InputOption::VALUE_REQUIRED, 'Artifact directory');

// Set rsync source to artifact directory
set('rsync_src', function () {
    return input()->getOption('artifact_directory');
});

2017-09-14: Bamboo htaccess authentication errors

Bamboo .htaccess restricted authentication will be forwarded to the Tomcat which can cause some authentication errors. It is possible to prevent the forwarding with the following proxy header setting in the Nginx configuration.

proxy_set_header   Authorization "";

2017-08-07: HipChat notifications

Bash snippets for HipChat notifications.

#: HipChat API v2
curl -X POST \
     -H "Content-Type: application/json" \
     --data "{ \"color\":\"green\", \"message\":\"HipCHat API v2\", \"message_format\":\"text\" }" \
     https://api.hipchat.com/v2/room/<room_api_id>/notification?auth_token=<auth_token>

#: HipChat API v1
curl --data "from=Sender&room_id=<room_api_id>&message=%28successful%29+HipChat+API+v1+&message_format=text&color=green" "https://api.hipchat.com/v1/rooms/message?format=json&auth_token=<auth_token>"

Further links:

Official HipChat Documentation

2017-08-01: Bamboo workaround for empty directories in artifacts

Problem: Bamboo doesn't allow empty directories in artifacts. See Bamboo issue tracking.

Workaround 1 - Logging: Write empty directories into a file during the build process and re-create them during the deployment process.

Build step:

#: Find empty directories and write them into a logging file
find . -empty -type d > empty_directories.txt

Deployment step:

#: Create empty directories based on the logging file
cat empty_directories.txt | while read directory; do
    mkdir -p $directory
done

Workaround 2 - Archive: Create an archive during the build process and extract it during the deployment process.

Build step:

#: Create tar.gz archive of all files and remove base files
tar czf artifact.tar.gz * --remove-files

Deployment step:

#: Extract archive and remove it
tar xzfv artifact.tar.gz && rm artifact.tar.gz

2017-07-20: Syntaxchecks

Bash snippets to check the syntax of other files.

PHP

Syntaxchecks for PHP files.

#: Syntaxcheck for one file
php -l <file_name>

#: Syntaxcheck for all files in the current directory
find . -name "*.php" -exec php -l {} \;

#: Syntaxcheck for all files in the last Git commit
git diff --name-only --diff-filter=ACMR HEAD~1..HEAD | grep -E "^.*.php$" | xargs -i php -l {}

YAML

Syntaxchecks for YAML files.

#: Syntaxcheck for one file
ruby -e "require 'yaml';puts YAML.load_file('<file_name>')"

#: Syntaxcheck for all files in the current directory
for file in `find . -name "*.yaml"`
do
  ruby -e "require 'yaml';puts YAML.load_file(\"$file\")" > /dev/null 2>&1
  if [ $? -eq 0 ]; then
    echo "No syntax errors detected in $file"
  else
    echo "Some syntax errors detected in $file"
  fi
done

#: Syntaxcheck for all files in the last Git commit
for file in `git diff --name-only --diff-filter=ACMR HEAD~1..HEAD | grep -E "^.*.yaml$"`
do
  ruby -e "require 'yaml';puts YAML.load_file(\"$file\")" > /dev/null 2>&1
  if [ $? -eq 0 ]; then
    echo "No syntax errors detected in $file"
  else
    echo "Some syntax errors detected in $file"
  fi
done

Bash

2018-05-30: Array and for loop

THe following snippet shows a simple array initialization and a for loop over each element of the array.

#!/bin/bash
array=(a b c d e)
for char in ${array[@]}
do
    echo $char
done

2018-05-05: SSH-Agent

As admin or developer you should use SSH keys for the login on your server.

Create a SSH Key on your jumpserver and set a passphrase

ssh-keygen -t rsa -b 4096 -C "yourname@mail.tld"

Add the public key to your destination servers to enable login via your new created SSH key

#: Check your public key on the jumpserver
cat ~/.ssh/id_rsa.pub

#: Add the public key to your destination servers authorized_keys
echo "<your_public_key" >> ~/.ssh/authorized_keys

#: Sometimes you have to check the file permissions on your destination server
chmod 700 /root/.ssh
chmod 600 ~/.ssh/authorized_keys

SSH-Agent autostart during login on your jumpserver

Add the following snippet to your ~/.bash_profile on your jumpserver...

echo "==> SSH agent startup"
if [ -z "$SSH_AUTH_SOCK" ] ; then
  eval `ssh-agent -s`
  ssh-add
fi
echo "==> SSH agent startup"

... and add the following snippet to your ~/.bash_logout to close the ssh-agent properly

if [ -n "$SSH_AUTH_SOCK" ] ; then
  eval `/usr/bin/ssh-agent -k`
fi

2018-01-15: Lock function

Bash functions to lock and unlock a script to make sure it is only running once. The lockfile is named like the script itself but with the added .lock ending.

#!/bin/bash

lock() {
    lockFile="${0##*/}.lock"
    if [ ! -f $lockFile ]; then
        `touch $lockFile`;
    else
        echo "==> The script is already running!";
        exit 1;
    fi
}

unlock() {
    lockFile="${0##*/}.lock"
    rm $lockFile
}

2018-01-06: Simple service check

Bash snippet to check if a service is up and running. Save the script as check_service.sh and call it with the servicename as parameter.

#!/bin/bash

if [ -n "$1" ];then
    if pgrep $1 >/dev/null ;then
        echo "$1: up"
    else
        echo "$1: down"
    fi
else
    echo "Error: missing parameter"
    echo "Usage: bash check_service.sh <servicename>"
fi

2017-12-13: Simple disk usage monitoring

Bash snippet to monitor disk usage on non-production systems.

#!/bin/bash
USED=`df /dev/sda1 | awk '{print $5}' | sed -ne 2p | cut -d"%" -f1`
if [ "$USED" -gt 90 ]; then
    echo "NOT OK: disk usage is above $USED percent."
    # Add some alerting via mail or chat here
else
    echo "OK: $USED disk space used."
fi

2017-09-21: Bash condition to check if file is directory

The tilde as indicator for the home directory isn't interpreted right during a bash condition validation.

#: problematic condition
if [ -d "~/directory" ]; then echo "... is a directory."; fi

#: working condition
if [ -d "$HOME/directory" ]; then echo "... is a directory."; fi

Ansible

2018-03-16: Ansible Galaxy Infrastructure

Ansible Galaxy is a hub for ansible roles. I prefer generic Ansible Roles, which can be used on all parts of the infrastructure. This also avoids any misunderstandings between the development environments.

Infrastructure overview

------------        --------------        --------------        ---------------
| role php |        | role nginx |        | role mysql |        | role custom |
------------        --------------        --------------        ---------------
      |                   |                     |                     |
      -----------------------------------------------------------------
                                     |
                            ------------------
                            | Ansible Galaxy |
                            ------------------
                                     |
        -------------------------------------------------------------------
        |                       |                    |                    |
-----------------        ---------------        -----------        --------------
| local dev-box |        | integration |        | staging |        | production |
-----------------        ---------------        -----------        --------------

Ansible roles are plugins for your different ansible projects. They are managed in the requirements.yml file and easy to install with ansible-galaxy.

#: Installation of roles via ansible-galaxy
ansible-galaxy install -r requirements.yml -p roles/

#: Example of a requirements.yml
- name: php
  src: https://github.com/geerlingguy/ansible-role-php.git
  scm: git
  version: master

Thanks to Jeff Geerling for the great ansible roles on Github.

2018-01-02: Load variables based on distribution information

The Ansible Snippet will search in the vars/ folder of a role and loads the variables of the first found file.

- name: Load variables based on distributon information
  include_vars: "{{ item }}"
  with_first_found:
    - "{{ ansible_distribution }}-{{ ansible_distribution_version }}.yml"
    - "{{ ansible_distribution }}.yml"
    - "{{ ansible_os_family }}.yml"
    - "default.yml"

2017-09-06: Conditionals for release versions

Ansible tasks examples:

- name: "Run only on Debian 8"
  debug:
    msg: "OS: {{ ansible_distribution }} // Version: {{ ansible_distribution_major_version }}"
  when: (ansible_distribution == "Debian" and ansible_distribution_major_version == "8")

- name: "Run on Debian 8 and Debian 9"
  debug:
    msg: "OS: {{ ansible_distribution }} // Version: {{ ansible_distribution_major_version }}"
  when: (ansible_distribution == "Debian" and ansible_distribution_major_version == "8") or
        (ansible_distribution == "Debian" and ansible_distribution_major_version == "9")

Jinja2 template examples for the template module:

{% if ansible_distribution_major_version == '8' %}
Debian 8
{% endif %}

{% if ansible_distribution_major_version == '8' or ansible_distribution_major_version == '9' %}
Debian 8 or Debian 9
{% endif %}

2017-08-11: Identify Vagrantboxes

Ansible snippet to identify Vagrantboxes.

- name: Check if Server is a Vagrantbox
  shell: 'grep vagrant /etc/passwd | wc -l'
  check_mode: no
  changed_when: false
  register: vagrantbox

- name: Server is a vagrantbox
  debug: msg="Server is a vagrantbox"
  when: vagrantbox.stdout != "0"

- name: Server is not a vagrantbox
  debug: msg="Server is not a vagrantbox"
  when: vagrantbox.stdout == "0"

Vagrant

2018-02-05: Re-sync default shared directory

Snippet to re-sync the default shared directory between the host and a Vagrantbox managed by Virtualbox.

#: Re-sync shared directory
neikei@workstation:~/vagrant/testing$ vagrant rsync
==> default: Rsyncing folder: /home/neikei/vagrant/testing/ => /vagrant

2017-12-01: NFS share on Ubuntu

NFS is the fastest Vagrant Share and the following snippet will explain how to use it on Ubuntu.

#: Install nfs-kernel-server
neikei@workstation:~$ sudo apt install nfs-kernel-server

#: Add the following line to the Vagrantfile
config.vm.synced_folder ".", "/vagrant", id: "v-root", mount_options: ["rw", "tcp", "nolock", "noacl", "async"], type: "nfs", nfs_udp: false

#: Vagrant up
neikei@workstation:~$ vagrant up
...
==> default: Exporting NFS shared folders...
==> default: Preparing to edit /etc/exports. Administrator privileges will be required...
[sudo] password for neikei:
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: active (exited) since Fr 2017-12-01 08:04:48 CET; 1h 30min ago
  Process: 1269 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
  Process: 1266 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 1269 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

Dez 01 08:04:48 workstation systemd[1]: Starting NFS server and services...
Dez 01 08:04:48 workstation systemd[1]: Started NFS server and services.
==> default: Mounting NFS shared folders...
...

2017-08-08: PHP-FPM not running after vagrant up (Vagrantbox by bento)

The Vagrantboxes from the bento project are really good, but the PHP-FPM service didn't start properly during a vagrant up. So I did some research and found a cleanup script (/etc/init.d/mountall-bootclean.sh) which is executed during every startup and removes temporary directories like /var/run where the PHP-FPM socket was placed. Move the socket to a static directory or use the following provisioning command in your Vagrantfile to ensure the PHP-FPM is running after a vagrant up.

#: Ensure PHP-FPM and Nginx restart after vagrant up
config.vm.provision "shell", inline: "service php7.1-fpm restart && service nginx restart", run: "always"

2017-07-31: Basic command list

Basic commands for Vagrant.

#: Show Vagrant Version
vagrant --version

#: Show installed plugins and versions
vagrant plugin list

#: Create Vagrantfile based on a box from https://app.vagrantup.com/
vagrant init debian/stretch64

#: Start the virtual machine based on the Vagrantfile
vagrant up

#: Show the status of the virtual machine
vagrant status

#: Connect to the virtual machine
vagrant ssh

#: Stop the virtual machine
vagrant halt

#: Destroy the virtual machine
vagrant destroy

#: Validate the Vagrantfile
vagrant validate

Further links:

Git

2017-08-10: Revert git commits

Git snippets to revert commits.

#: Revert the last commit
git revert HEAD

#: Revert a specific commit by commit id
git revert <commit_id>

#: Revert the last merge
git revert -m1 HEAD

#: Revert a specific merge by commit id
git revert -m1 <commit_id>

2017-07-17: Git user switcher

Bash script to switch between git accounts.

#!/bin/bash
echo "";
echo "1) <name_account1>";
echo "2) <name_account2>";
echo "";

while true; do
    read -p "Which account do you need? " choice
    case $choice in
        [1]* ) git config --global --replace-all user.name "<username_account1>"; git config --global user.email "<usermail_account1>"; break;;
        [2]* ) git config --global --replace-all user.name "<username_account2>"; git config --global user.email "<usermail_account2>"; break;;
        * ) echo "Please answer 1 or 2.";;
    esac
done

mail=`git config --global user.email`;
user=`git config --global user.name`;

echo "";
echo "############ Activated ############";
echo "User: $user";
echo "Mail: $mail";
echo "";

License

This are just a few notes without any warranty. Please check the license here.

Name		Name	Last commit message	Last commit date
Latest commit History 82 Commits
notes		notes
tools		tools
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
_config.yml		_config.yml

License

neikei/notes

Folders and files

Latest commit

History

Repository files navigation

Summary

Hint

Notes

Linux

2017-12-06: Screenfetch

2017-11-27: Locate, partition and create filesystem on new disk

2017-11-22: System CLI monitoring

2017-11-21: System versions

2017-11-11: Search and replace strings in files

2017-10-28: Pin ansible apt package

2017-10-28: Install specific apt package version

2017-10-20: SystemD multi spawn processes

2017-10-15: Ubuntu optimize jpeg images

2017-10-04: Debian ntpdate time synchronization

2017-08-21: Database replication status

2017-08-14: Validate archives

2017-08-12: Install Oracle Java 8 on Debian Stretch

2017-07-16: Command line convenience

~/.bashrc

~/.inputrc

~/.vimrc

2017-07-15: Fix locale warnings on Linux servers

2017-07-15: summarizeMD

Windows

2018-04-14: Kubernetes Minikube

2018-03-13: Chocolatey basics

2018-01-12: Shortcuts for virtual desktops

2018-01-07: Use Ubuntu Bash as Visual Studio Code Terminal

2018-01-03: ASUS Zenbook UX310UA FanControl

2017-12-26: Linux system administrator tools

2017-12-15: Ubuntu Bash installation on Windows 10

Continuous Integration & Continuous Delivery

2018-06-07: Configuration management

2018-04-16: Selfstudy Docker

2018-01-30: DevOps learnings 2017

2018-01-18: Makefile for WebApps

2017-12-21: Bamboo JUnit Parser fake results for hotfixes

2017-11-23: Common HTTP status codes

2017-10-26: Bamboo branch based configs

2017-09-26: Jenkins centralized syntaxchecks

2017-09-19: Bamboo centralized syntaxchecks

2017-09-18: Deploy archives with Bamboo and PHP Deployer

2017-09-15: Use PHP Deployer with Bamboo

2017-09-14: Bamboo htaccess authentication errors

2017-08-07: HipChat notifications

2017-08-01: Bamboo workaround for empty directories in artifacts

2017-07-20: Syntaxchecks

PHP

YAML

Bash

2018-05-30: Array and for loop

2018-05-05: SSH-Agent

2018-01-15: Lock function

2018-01-06: Simple service check

2017-12-13: Simple disk usage monitoring

2017-09-21: Bash condition to check if file is directory

Ansible

2018-03-16: Ansible Galaxy Infrastructure

2018-01-02: Load variables based on distribution information

2017-09-06: Conditionals for release versions

2017-08-11: Identify Vagrantboxes

Vagrant

2018-02-05: Re-sync default shared directory

2017-12-01: NFS share on Ubuntu

2017-08-08: PHP-FPM not running after vagrant up (Vagrantbox by bento)

2017-07-31: Basic command list

Git

2017-08-10: Revert git commits

2017-07-17: Git user switcher

License

About

Topics

Resources

License

Packages