-
Notifications
You must be signed in to change notification settings - Fork 22
diagnostic system logs
The pigrow has scripts designed to log system performance and health, these are run by cron at set intervals. Generally ever 15 min is suggested.
The tables below show what is captures and the command used to gather it, asking chatGPT or similar 'what does do on the raspberry pi' is a great way of getting a detailed explanation.
~/Pigrow/scripts/cron/selflog.py
This is the more basic system logging tool it captures;
st = os.statvfs("./")
disk_total - st.f_blocks * st.f_frsize
disk_used - (st.f_blocks - st.f_bfree) * st.f_frsize
disk_free - st.f_bavail * st.f_frsize
disk_percent - (used / total) * 100
timenow - datetime.datetime.now()
uptime_sec - /proc/uptime
uptime_str - human readable uptime
load_ave1 - os.getloadavg()
load_ave5 - os.getloadavg()
load_ave15 - os.getloadavg()
memtotal - /proc/meminfo
memfree - /proc/meminfo
memavail - /proc/meminfo
cpu_temp - /opt/vc/bin/vcgencmd measure_temp
it also checks if 'reddit_setings_ear.py' and 'checkDHT.py' are running using pidof and checking /proc//status to return a count of the times it's running and it's status, though now checkDHT is obsolete this function will soon be changed to monitor the newer scripts.
log is saved to '/home//Pigrow/logs/selflog.txt' currently there are no command line arguments for this script beside -h and --help.
~/Pigrow/scripts/cron/selflog_adv.py
A more advanced system logging tool which captures more information, much of it is superfluous but can be useful when testing new code and tools - the average user does not need this amount of data.
timenow - datetime.datetime.now()
mem_avail - /proc/meminfo
mem_free - /proc/meminfo
mem_total - /proc/meminfo
ps_all_info - ps -A
ps_all_threads - ps -AL
ps_all_users - ps aux -w
ps_for_root - ps -U root
ps_for_user - ps -U
ps_sshd - ps -C sshd
uptime_seconds - /proc/uptime
cpu_ctx_switches - psutil.cpu_stats()[0]
cpu_frequency - psutil.cpu_freq(percpu=False)[1]
cpu_interrupts - psutil.cpu_stats()[1]
cpu_max_frequency - psutil.cpu_freq(percpu=False)[2]
cpu_min_frequency - psutil.cpu_freq(percpu=False)[1]
cpu_soft_interrupts - psutil.cpu_stats()[2]
cpu_temp_<name> - psutil.sensors_temperatures(fahrenheit=False)[x][0][1]
disk_busy_time - psutil.disk_io_counters(perdisk=False, nowrap=True)[8]
disk_free - psutil.disk_usage("/home/")[2]
disk_read_bytes - psutil.disk_io_counters(perdisk=False, nowrap=True)[2]
disk_read_count - psutil.disk_io_counters(perdisk=False, nowrap=True)[0]
disk_read_merged_count - psutil.disk_io_counters(perdisk=False, nowrap=True)[6]
disk_read_time - psutil.disk_io_counters(perdisk=False, nowrap=True)[5]
disk_usage_pcent - psutil.disk_usage("/home/")[3]
disk_used - psutil.disk_usage("/home/")[1]
disk_write_bytes - psutil.disk_io_counters(perdisk=False, nowrap=True)[3]
disk_write_count - psutil.disk_io_counters(perdisk=False, nowrap=True)[1]
disk_write_merged_count - psutil.disk_io_counters(perdisk=False, nowrap=True)[7]
disk_write_time - psutil.disk_io_counters(perdisk=False, nowrap=True)[5]
load_ave1 - psutil.getloadavg()[0]
load_ave5 - psutil.getloadavg()[1]
load_ave15 - psutil.getloadavg()[2]
net_bytes_recv - psutil.net_io_counters(pernic=False, nowrap=True)[1]
net_bytes_sent - psutil.net_io_counters(pernic=False, nowrap=True)[0]
net_dropin - psutil.net_io_counters(pernic=False, nowrap=True)[6]
net_dropout - psutil.net_io_counters(pernic=False, nowrap=True)[7]
net_errin - psutil.net_io_counters(pernic=False, nowrap=True)[4]
net_errout - psutil.net_io_counters(pernic=False, nowrap=True)[5]
net_packets_recv - psutil.net_io_counters(pernic=False, nowrap=True)[3]
net_packets_sent - psutil.net_io_counters(pernic=False, nowrap=True)[2]
swap_in - psutil.swap_memory()[4]
swap_out - psutil.swap_memory()[5]
swap_pcent - psutil.swap_memory()[3]
vmem_active - psutil.virtual_memory()[5]
vmem_available - psutil.virtual_memory()[1]
vmem_free - psutil.virtual_memory()[4]
vmem_pcent - psutil.virtual_memory()[2]
vmem_total - psutil.virtual_memory()[0]
vmem_used - psutil.virtual_memory()[3]
clock_H264 - vcgencmd measure_clock H264
clock_arm - vcgencmd measure_clock arm
clock_audio - vcgencmd measure_clock pwm
clock_core - vcgencmd measure_clock core
clock_dpi - vcgencmd measure_clock dpi
clock_emmc - vcgencmd measure_clock emmc
clock_hdmi - vcgencmd measure_clock hdmi
clock_isp - vcgencmd measure_clock isp
clock_pixel - vcgencmd measure_clock pixel
clock_uart - vcgencmd measure_clock uart
clock_v3d - vcgencmd measure_clock v3d
clock_vec - vcgencmd measure_clock vec
display_power - vcgencmd display_power
get_throttled = 0x0 - vcgencmd get_throttled
lb_fails - vcgencmd mem_reloc_stats
lcd_info - vcgencmd get_lcd_info
libcam_interfaces - vcgencmd get_camera
oom_events - vcgencmd mem_oom
oom_lifesize - vcgencmd mem_oom
oom_maxtime - vcgencmd mem_oom
oom_totaltime - vcgencmd mem_oom
picam_detected - vcgencmd get_camera
picam_supported - vcgencmd get_camera
vc4_alloc_fail - vcgencmd mem_reloc_stats
vc4_compactions - vcgencmd mem_reloc_stats
volts_core - vcgencmd measure_volts core
volts_sdram_c - vcgencmd measure_volts sdram_c
volts_sdram_i - vcgencmd measure_volts sdram_i
volts_sdram_p - vcgencmd measure_volts sdram_p
~/Pigrow/scripts/cron/selflog_netspeed.py
A tool to create a log of internet speed using speedtest-cli
sudo pip3 install speedtest-cli
It collects several useful bits of information and saves them to /Pigrow/logs/netspeed_log.txt
Server ID
Sponsor
Server_Name
Distance
Ping
Download
Upload
As this uses internet bandwidth it's not recommended to run this if unrequired, to use it set it repeating as required. Depending on the level of information required every fifteen min, 3 hours, 12 hours or daily are good choices.
a tool for one pi to check another pi and log health data, soon going to be replaced with a better version.
~/Pigrow/scripts/build_test/selflog_benchmark.py
This runs a series of benchmarking tests which can be compared to other boards to check performance, in the future there will be a tool to automatically compare these with collected averages.
it performs the following tests;
basic_counting - count to 100000000
basic_doubling - count down from 100000000
basic_halving - i * 2 repeated 50000 times
basic_subtraction - i / 2 halving from 100000000 repeated 50000 times
image_create_chess_25 - procedurally generate a 25000x25000 pixel chess board
image_create_frame_set - procedurally generate 500 colour fading images
video_render_gif - use MPV to create a 500 frame timelapse .gif using above images
video_render_mp4 - as above with .mp4
the following are the metrics it measures between start and finish, note these will be affected by other processes running on the pi.
change_disk_busy_time - psutil.disk_io_counters(perdisk=False, nowrap=True)[8]
change_disk_read_bytes - psutil.disk_io_counters(perdisk=False, nowrap=True)[2]
change_disk_read_time - psutil.disk_io_counters(perdisk=False, nowrap=True)[4]
change_disk_write_bytes - psutil.disk_io_counters(perdisk=False, nowrap=True)[3]
change_disk_write_time - psutil.disk_io_counters(perdisk=False, nowrap=True)[5]
change_swap_in - psutil.swap_memory()[4]
change_swap_out - psutil.swap_memory()[5]
total_time - the total time of all tests
A log is created ~/Pigrow/log/selflog_benchmark.py