-
Notifications
You must be signed in to change notification settings - Fork 51
Debugging Salt
Set the log level of some or all salt components to debug or trace. trace logs a lot of information, it can be hard to find the needle in the haystack. It's recommended to start with debug and only use trace if debug is not enough.
Increasing the log level is always a good idea when debugging salt. Just remember to turn it off afterwards as logging slows programs down and fills up the disk.
Replace debug with trace for even more logging. All locations below are on the system that's being debugged unless noted otherwise.
echo 'log_level: debug' >/etc/salt/master.d/zzz-debug.conf
systemctl restart salt-master salt-api
The logs are located at /var/log/salt/master and /var/log/salt/api.
echo 'ExecStart=/usr/bin/salt-api --log-file-level=debug' >/etc/systemd/system/salt-api.service.d/logging.conf
systemctl restart salt-api
The log is located at /var/log/salt/api.
echo 'log_level: debug' >/etc/salt/minion.d/zzz-debug.conf
systemctl restart salt-minion
The log is located at /var/log/salt/minion.
echo 'log_level: debug' >/etc/venv-salt-minion/minion.d/zzz-debug.conf
systemctl restart venv-salt-minion
The log is located at /var/log/venv-salt-minion.log.
On the Salt Master:
cat >/etc/salt/master.d/zzz-salt-ssh-debug.conf <<EOF
ssh_minion_opts:
log_file: ../../../../../var/log/salt-ssh.log
log_level: debug
EOF
systemctl restart salt-master
The log is located on the Salt SSH-controlled system at /var/log/salt-ssh.log
The Salt Master has an EventPublisher that publishes events on a bus. It's possible to connect to this bus and read the events as they are happening.
This is very useful when working with beacons and/or reactors. It's also a general tool to take a look at the communication as it shows not only which Salt jobs are published, but also the responses.
salt-run is only available on the Salt Master.
salt-run state.event | tee salt-events.txt
salt-run state.event | while read -r tag data; do
echo $tag
echo $data | jq .
done
salt-run state.event pretty=True | tee salt-events.txt
salt myminionid system.reboot
salt-run state.event 'salt/minion/myminionid/start' count=1 && echo 'Reboot finished'
SLS files are first rendered and then applied to the system. It's possible to see the result of the rendering steps without doing applying the SLS file.
States can't be applied due to rendering errors or during the development of new states.
salt myminionid slsutil.renderer salt://path/to/state.sls
salt-call --local slsutil.renderer /absolute/path/to/state.sls
By specifying default_renderer=jinja, only Jinja templating is evaluated. The result is (by default) YAML.
salt myminionid slsutil.renderer salt://path/to/state.sls default_renderer=jinja
salt-call --local slsutil.renderer /absolute/path/to/state.sls default_renderer=jinja