Release News #109

Ricks-Lab · 2021-03-05T02:01:30Z

Ricks-Lab
Mar 5, 2021
Maintainer

I have just released v3.5.7 to PyPI. It includes the following changes:

More robust determination of install type and display this with --about and in logger.
Implementation of scroll within PAC window.
Fixed plot crash for invalid ticker increment.
Code robustness improvements with more typing for class variables.

Ricks-Lab · 2021-03-17T04:44:04Z

Ricks-Lab
Mar 17, 2021
Maintainer Author

I have just released v3.5.8 to PyPI. It includes the following changes:

Fixed bug in determining AMD GPU card type. Now it properly identifies APU and Legacy types.

Plan to make an official new minor release once this release is verified by enough users. Please provide your feedback here.

0 replies

csecht · 2021-03-17T18:16:20Z

csecht
Mar 17, 2021

I couldn’t uninstall or install PyPi 3.6.8. Both pip3 commands’ error messages indicate that 3.5.6 is already installed. 3.5.6 is the latest debian apt installation I did. I don’t recall ever doing a PyPi install, but there is a /usr/lib/python3/dist-packages/rickslab_gpu_utils-3.5.6.egg-info folder. Anyway, I downloaded the latest commit for 3.5.8 from GitHub. Here are my observations for my Ubuntu 20.04.2 system with the 5.4.0 kernel, running a RX 5600 XT: gpu-ls —clinfo does not provide any OpenCL info; it’s the same output as gpu-ls. This is true in 3.5.6 and 3.5.8. The terminal command $ clinfo does provide OpenCL data. Cosmetic: The gpu-mon —gui window is one character too narrow to show the full GPU model name of “RADEON RX 5600 XT”. This is true in 3.5.6 and 3.5.8. Everything else seems to check out. Nice! Just a minor feature suggestion for gpu-plot: Have the Plot Item buttons indicate whether an item is active or not, e.g. color vs. greyed, sunken vs. raised, etc. As it is, they each look the same before and after you click on them. Cheers, Craig

…

On Mar 16, 2021, at 11:44 PM, Rick ***@***.***> wrote: I have just released v3.5.8 to PyPI. It includes the following changes: Fixed bug in determining AMD GPU card type. Now it properly identifies APU and Legacy types. Plan to make an official new minor release once this release is verified by enough users. Please provide your feedback here. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#109 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMVCQU6YVDWSY6CU6K2TWDTEAXSHANCNFSM4YULGNEA>.

5 replies

Ricks-Lab Mar 18, 2021
Maintainer Author

For the pip3 install issue, I remember a past discussion on the issue. I suggest trying the following:

pip3 install rickslab-gpu-utils -U
sudo pip3 uninstall rickslab-gpu-utils
pip3 uninstall ricks-amdgpu-utils
sudo pip3 uninstall ricks-amdgpu-utils
pip3 install --no-cache-dir rickslab-gpu-utils

Make sure that you are not in a virtual env when trying the above. I have just installed from PyPI on 3 systems with no issues.

Ricks-Lab Mar 18, 2021
Maintainer Author

For the clinfo issue, my 18.04.5, there is no issue. On my 20.04.2 system, clinfo gives a core dump. I uninstalled and reinstalled latest amdgpu package from AMD and it failed with many errors. Perhaps the amdgpu drivers for 20.04.2 are not ready yet.

Ricks-Lab Mar 18, 2021
Maintainer Author

Cosmetic: The gpu-mon —gui window is one character too narrow to show the full GPU model name of “RADEON RX 5600 XT”. This is true in 3.5.6 and 3.5.8. Everything else seems to check out. Nice! Just a minor feature suggestion for gpu-plot: Have the Plot Item buttons indicate whether an item is active or not, e.g. color vs. greyed, sunken vs. raised, etc. As it is, they each look the same before and after you click on them. Cheers, Craig

I will try to fit in the suggested UI improvements for 3.6.0 release.

Ricks-Lab Mar 18, 2021
Maintainer Author

Latest in the repository has cosmetic changes and updates to install instructions in README.

Ricks-Lab Mar 23, 2021
Maintainer Author

I just noticed that AMD has release amdgpu for Ubuntu 20.04.2 yesterday. I have not tried it yet.

csecht · 2021-03-19T02:51:46Z

csecht
Mar 19, 2021

Hmm, it’s not working for me. I can’t seem to uninstall or install over the 3.5.6 package ~$ sudo pip3 uninstall rickslab-gpu-utils Found existing installation: rickslab-gpu-utils 3.5.6 Not uninstalling rickslab-gpu-utils at /usr/lib/python3/dist-packages, outside environment /usr Can't uninstall 'rickslab-gpu-utils'. No files were found to uninstall. ~$ pip3 uninstall rick-amdgpu-utils WARNING: Skipping rick-amdgpu-utils as it is not installed. ~$ pip3 install --no-cache-dir rickslab-gpu-utils Requirement already satisfied: rickslab-gpu-utils in /usr/lib/python3/dist-packages (3.5.6)

…

On Mar 17, 2021, at 7:13 PM, Rick ***@***.***> wrote: For the pip3 install issue, I remember a past discussion on the issue. I suggest trying the following: sudo pip3 uninstall rickslab-gpu-utils pip3 uninstall ricks-amdgpu-utils sudo pip3 uninstall ricks-amdgpu-utils pip3 install --no-cache-dir rickslab-gpu-utils Make sure that you are not in a virtual env when trying the above. I have just installed from PyPI on 3 systems with no issues. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMVCQWJAWCKUCTDCIRDFSTTEFARFANCNFSM4YULGNEA>.

1 reply

Ricks-Lab Mar 19, 2021
Maintainer Author

I was able to duplicate the issue on my system. Seems like you can not pip3 install the package if it is already apt installed. I was able to apt install a package that was already pip installed. I suggest apt purging the package and then pip install. This worked on my system.

csecht · 2021-03-19T15:29:30Z

csecht
Mar 19, 2021

Sweet. Yes, that works. I got 3.5.9 installed. Thanks.

…

On Mar 19, 2021, at 3:17 AM, Rick ***@***.***> wrote: I was able to duplicate the issue on my system. Seems like you can not pip3 install the package if it is already apt installed. I was able to apt install a package that was already pip installed. I suggest apt purging the package and then pip install. This worked on my system. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMVCQTKQBE2TOQ3IHMXVMTTEMCBXANCNFSM4YULGNEA>.

1 reply

Ricks-Lab Mar 20, 2021
Maintainer Author

In preparation for v3.6.0, I have updated the installation instructions and divided it between the readme and the users guide. Let me know of any feedback.

Ricks-Lab · 2021-03-22T02:58:15Z

Ricks-Lab
Mar 22, 2021
Maintainer Author

v3.5.11 (v3.6.0 release candidate) has been released to PyPI. Please report any observation here. Thanks!

2 replies

csecht Mar 24, 2021

I finally used pip3 install rickslab-gpu-utils -U to do an update and was impressed with how fast that loaded v3.5.12. Everything in that release is working fine for me.

Regarding my previous comments:
The gpu-plot button active/deactive display is nicely done.
gpu-mon --gui now nicely displays my "RX 5600 XT" model name.

I'm going to soon try loading the latest AMDGPU drivers for Ubuntu 20.04.2 to see whether that gets gpu-ls --clinfo to work.
Until then, a suggestion for an enhancement:
Could use a gpu-utils --help (-h) or gpu-help command option (or both!) to bring up all gpu-utils run options.

Ricks-Lab Mar 24, 2021
Maintainer Author

Until then, a suggestion for an enhancement:
Could use a gpu-utils --help (-h) or gpu-help command option (or both!) to bring up all gpu-utils run options.

Have you checked out the man pages? I was hoping to provide that kind of coverage in the man pages, but I don't include a top level page for gpu-utils. Maybe that would be a good addition.

csecht · 2021-03-24T21:35:42Z

csecht
Mar 24, 2021

I upgraded drivers to AMD: amdgpu version: 20.50-1234664. Had to upgrade BOINC to 7.16.6 to get BOINC Manager to run, but everything is working fine.
On my machine, however, gpu-ls --clinfo still does not provide clinfo data (but $ clinfo does), giving the output below. I'm not sure whether the 0 number of devices has anything to do with anything.

$ clinfo
dlerror: libnvidia-opencl.so.1: cannot open shared object file: No such file or directory
Number of platforms                               1
  Platform Name                                   AMD Accelerated Parallel Processing
  Platform Vendor                                 Advanced Micro Devices, Inc.
  Platform Version                                OpenCL 2.0 AMD-APP (3224.0)
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd cl_amd_event_callback 
  Platform Extensions function suffix             AMD

  Platform Name                                   AMD Accelerated Parallel Processing
Number of devices                                 0

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  No platform
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   No platform
  clCreateContext(NULL, ...) [default]            No platform
  clCreateContext(NULL, ...) [other]              No platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  No devices found in platform

2 replies

Ricks-Lab Mar 24, 2021
Maintainer Author

This definitely doesn't look right. gpu-utils will parse this output and since there is no information, it will not have anything to install. Seems like a system issue. Do you have NV GPUs installed? Weird that it is looking for libnvidia-opencl.so.1. Perhaps an uninstall of amdgpu and reinstall. Maybe same for clinfo. Also, maybe a clean install of 20.04.2. This would have the effect of running the latest kernel also.

Ricks-Lab Mar 24, 2021
Maintainer Author

I upgraded drivers to AMD: amdgpu version: 20.50-1234664. Had to upgrade BOINC to 7.16.6 to get BOINC Manager to run, but everything is working fine.

I have noticed BOINC is a bit kludgy on my systems. Where are you getting the latest BOINC release? I should probably upgrade.

csecht · 2021-03-25T00:26:39Z

csecht
Mar 25, 2021

There’s the https://boinc.berkeley.edu/download.php site, but I do `apt upgrade boinc`

…

On Mar 24, 2021, at 6:50 PM, Rick ***@***.***> wrote: I upgraded drivers to AMD: amdgpu version: 20.50-1234664. Had to upgrade BOINC to 7.16.6 to get BOINC Manager to run, but everything is working fine. I have noticed BOINC is a bit kludgy on my systems. Where are you getting the latest BOINC release? I should probably upgrade. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMVCQSOLI3WG5DXVSHB2I3TFJ3GDANCNFSM4YULGNEA>.

0 replies

csecht · 2021-03-25T00:33:40Z

csecht
Mar 25, 2021

I uninstall/install amdgpu each time there’s a new package. At one time I had a nvidia card on that machine, but not now. amdgpu installs it’s own version of clinfo I noticed from the recent upgrade/install. I think that a clean install of 20.04.2 is what is needed. I hadn’t realized that my kernel wasn’t keeping up on the upgrades. Ugh.

…

On Mar 24, 2021, at 6:46 PM, Rick ***@***.***> wrote: This definitely doesn't look right. gpu-utils will parse this output and since there is no information, it will not have anything to install. Seems like a system issue. Do you have NV GPUs installed? Weird that it is looking for libnvidia-opencl.so.1. Perhaps an uninstall of amdgpu and reinstall. Maybe same for clinfo. Also, maybe a clean install of 20.04.2. This would have the effect of running the latest kernel also. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMVCQXSU4QQWRTAXV5UAZTTFJ2VRANCNFSM4YULGNEA>.

0 replies

csecht · 2021-03-25T00:45:50Z

csecht
Mar 25, 2021

A top level gpu-utils page would be nice. Yes, the man pages are good, but still need to know the name of the specific gpu program. My old brain can’t remember always them all without going to the User Guide.

…

On Mar 24, 2021, at 6:48 PM, Rick ***@***.***> wrote: Until then, a suggestion for an enhancement: Could use a gpu-utils --help (-h) or gpu-help command option (or both!) to bring up all gpu-utils run options. Have you checked out the man pages? I was hoping to provide that kind of coverage in the man pages, but I don't include a top level page for gpu-utils. Maybe that would be a good addition. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMVCQV4MI2COKATP4J2JVDTFJ26BANCNFSM4YULGNEA>.

1 reply

Ricks-Lab Mar 26, 2021
Maintainer Author

I added the list of tools in the package for the --about option of each tool. Also, the man page of each tool, includes references to the man pages of all other tools in the last section. Hopefully this, with the simplified README, will make it easier for new users.

Ricks-Lab · 2021-03-27T01:50:46Z

Ricks-Lab
Mar 27, 2021
Maintainer Author

I have just released v3.6.0! The release includes PyPI, rickslab.com Debian, and GitHub release. Need to work with official Debian package maintainer to update it in official Debian packages. This release includes the following changes:

Rewrite of the installation guide.
Roll-up all v3.5.x patches into a new minor revision release.

4 replies

csecht Mar 27, 2021

Looks good! Thanks!
For my Ubuntu Desktop app for gpu-mon, what is the Path to the Icon? For what I assume is the pip3 installation, I'm using the path: `/home/craig/.local/bin/gpu-mon --gui', but can't figure out where the .png icon files are.

Ricks-Lab Mar 27, 2021
Maintainer Author

If you run in debug mode, there will be an entry in the resultant log file indicating the location.

csecht Mar 28, 2021

Yes, cool, got it. Thanks!

Ricks-Lab Mar 28, 2021
Maintainer Author

Do you have a RX 5600 XT GPU installed? A user posted an issue, #115, indicating that his card is coming up as a Legacy GPU in the new version of the utils.

KeithMyers · 2021-03-27T22:32:13Z

KeithMyers
Mar 27, 2021

I don't think any of the applications ship with any icon files. They just use the default python icon.

1 reply

Ricks-Lab Mar 27, 2021
Maintainer Author

The package definitely includes icon files. The install location is dependent on package type.

KeithMyers · 2021-03-27T22:57:10Z

KeithMyers
Mar 27, 2021

OK, thanks for the tip, Rick. I didn't see any in the default location with the binaries.
My icons are in ~/.local/share/rickslab-gpu-utils/icons

1 reply

csecht Mar 28, 2021

Yep!

csecht · 2021-03-28T02:20:26Z

csecht
Mar 28, 2021

Yes. With gpu-utils 3.6.0 the Radeon RX 5600 XT comes up as expected.

…

On Mar 27, 2021, at 8:36 PM, Rick ***@***.***> wrote: Do you have a RX 5600 XT GPU installed? A user posted an issue, #115 <#115>, indicating that his card is coming up as a Legacy GPU in the new version of the utils. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMVCQWDKNGMY7WSZ6JIZP3TF2B3JANCNFSM4YULGNEA>.

0 replies

csecht · 2021-03-28T02:37:49Z

csecht
Mar 28, 2021

His reading of `amdgpu/rocm version: UNKNOWN` is suspicious. Because he did a general apt upgrade, if the kernel was upgraded, then he’d need to uninstall/re-install amdgpu drivers. At least that’s what I need to do with every kernel update; doesn’t matter if amdgpu has changed or not. What works for me is: `$ /usr/bin/amdgpu-uninstall` then from within the latest AMD driver directory, `$ ./amdgpu-install -y --opencl=rocr,legacy --headless --no-dkms`

…

On Mar 27, 2021, at 8:36 PM, Rick ***@***.***> wrote: Do you have a RX 5600 XT GPU installed? A user posted an issue, #115 <#115>, indicating that his card is coming up as a Legacy GPU in the new version of the utils. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMVCQWDKNGMY7WSZ6JIZP3TF2B3JANCNFSM4YULGNEA>.

0 replies

csecht · 2021-04-09T23:49:27Z

csecht
Apr 9, 2021

Regarding the issue I mentioned earlier of gpu-ls --clinfo not showing OpenCL info; a clean Ubuntu install or a kernel upgrade was not need, but this did work:
sudo usermod -a -G video $LOGNAME
sudo usermod -a -G render $LOGNAME
The solution was from https://community.amd.com/t5/drivers-software/no-opencl-devices-ubuntu-20-04-2/td-p/444567

2 replies

Ricks-Lab Apr 11, 2021
Maintainer Author

Perhaps it would be best to find a way to fit this into the User Guide.

csecht Apr 12, 2021

Good idea. I'll work on that tomorrow.

Ricks-Lab · 2022-05-14T06:51:16Z

Ricks-Lab
May 14, 2022
Maintainer Author

Release candidate 3.7.0-RC1 is ready for testing. Here is a list of changes:

Fixed error in calculating power when invalid sensor data is returned.
Check for OSError when reading from all sensor files. Disable sensor reading on error.
Check for system type. Only systemD is fully supported. Issues in reading sockets in systemV are handled.
Added read of Power DPM State for AMD GPUs.
On read error, make read for the parameter False instead of indicating card is not readable.
Add gpu-ls option --force_all to attempt to read all relevant sensors, regardless of card classification.
Improve error message handling. Minor (expected) errors are suppressed unless --verbose is specified. GPU output will indicate all sensors that were disabled due to read errors.
Implemented gpu-ls option --raw to give a summary view of the content of all available driver files.
Enable gpu-plot and gpu-mon capability to include GPUs with incomplete driver coverage.

Let me know of your observations. Thanks!

0 replies

Ricks-Lab · 2022-05-17T03:34:24Z

Ricks-Lab
May 17, 2022
Maintainer Author

Release candidate 3.7.0-RC2 is ready for testing. Here is a list of changes:

Fixed error in calculating power when invalid sensor data is returned.
Check for OSError when reading from all sensor files. Disable sensor reading on error.
Check for system type. Only systemD is fully supported. Issues in reading sockets in systemV are handled.
Added read of Power DPM State for AMD GPUs.
On read error, make read for the parameter False instead of indicating card is not readable.
Add gpu-ls option --force_all to attempt to read all relevant sensors, regardless of card classification.
Improve error message handling. Minor (expected) errors are suppressed unless --verbose is specified. GPU output will indicate all sensors that were disabled due to read errors.
Implemented gpu-ls option --raw to give a summary view of the content of all available driver files.
Enable gpu-plot and gpu-mon capability to include GPUs with incomplete driver coverage.
Allow plain text instead of formatted/color coded output with the --no_markup option.
Use pp_dpm_*clk files as a source of P-state information.

0 replies

Marjorie-R · 2022-05-17T16:15:20Z

Marjorie-R
May 17, 2022

Hi Rick,

Here's the outputs from gpu-utils function on my AMD5600G Zen3 processor/APU running Devuan 4 Chimaera (=Debian Bullseye without systemd) and Linux kernel version 5.16.
Notable changes from the last re-write of 3.6 you asked me to test are that the APU no longer shows as writeable and that gpu-pac no longer breaks (maybe as APU shows as read only).

--
Marjorie

Outputs below:

(rickslab-gpu-utils-env) marjorie@grendel:~/gpu-utils$ ./gpu-ls

OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: AMD: 1
AMD: Wattman features enabled: 0xfffd7fff
1 total GPUs, 0 rw, 1 r-only, 0 w-only

Card Number: 0
   Vendor:  AMD 
   Readable: True
   Writable: False
   Compute: False
   Device ID: {'device': '0x1638', 'subsystem_device': '0x1636', 'subsystem_vendor': '0x1002', 'vendor': '0x1002'}
   Decoded Device ID: Cezanne
   Card Model: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne (rev c9)
   Display Card Model:  Cezanne
   PCIe ID: 30:00.0
      Link Speed: 8.0 GT/s PCIe
      Link Width: 16
   ##################################################
   Driver: amdgpu
   vBIOS Version: 113-CEZANNE-017
   Compute Platform: None
   GPU Type: APU
   HWmon: /sys/class/drm/card0/device/hwmon/hwmon1
   Card Path: /sys/class/drm/card0/device
   System Card Path: /sys/devices/pci0000:00/0000:00:08.1/0000:30:00.0
   ##################################################
   Current Power (W): 0.004
   ##################################################
   Current Memory Loading (%): None
   Current GTT Memory Usage (%): 10.437
   Current VRAM Usage (%): 10.501
   Current  Temps (C): {'edge': 29.0}
   Critical Temps (C): {}
   Current Voltages (V): {'vddgfx': 1368, 'vddnb': 1099}
      Vddc Range: ['', '']
   Current Clk Frequencies (MHz): {'sclk': 400.0}
   Current SCLK P-State: [1, '400Mhz']
      SCLK Range: ['', '']
   Current MCLK P-State: [0, '1600Mhz']
      MCLK Range: ['', '']
   Power DPM State: performance
   Power DPM Force Performance Level: auto

(rickslab-gpu-utils-env) marjorie@grendel:~/gpu-utils$ ./gpu-ls --debug

Devuan: Validated
OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: AMD: 1
AMD: Wattman features enabled: 0xfffd7fff
1 total GPUs, 0 rw, 1 r-only, 0 w-only

Card Number: 0
   Vendor:  AMD 
   Readable: True
   Writable: False
   Compute: False
   Device ID: {'device': '0x1638', 'subsystem_device': '0x1636', 'subsystem_vendor': '0x1002', 'vendor': '0x1002'}
   Decoded Device ID: Cezanne
   Card Model: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne (rev c9)
   Display Card Model:  Cezanne
   PCIe ID: 30:00.0
      Link Speed: 8.0 GT/s PCIe
      Link Width: 16
   ##################################################
   Driver: amdgpu
   vBIOS Version: 113-CEZANNE-017
   Compute Platform: None
   GPU Type: APU
   HWmon: /sys/class/drm/card0/device/hwmon/hwmon1
   Card Path: /sys/class/drm/card0/device
   System Card Path: /sys/devices/pci0000:00/0000:00:08.1/0000:30:00.0
   ##################################################
   Current Power (W): 0.002
   ##################################################
   Current Memory Loading (%): None
   Current GTT Memory Usage (%): 10.437
   Current VRAM Usage (%): 10.476
   Current  Temps (C): {'edge': 28.0}
   Critical Temps (C): {}
   Current Voltages (V): {'vddgfx': 881, 'vddnb': 1099}
      Vddc Range: ['', '']
   Current Clk Frequencies (MHz): {'sclk': 400.0}
   Current SCLK P-State: [1, '400Mhz']
      SCLK Range: ['', '']
   Current MCLK P-State: [0, '1600Mhz']
      MCLK Range: ['', '']
   Power DPM State: performance
   Power DPM Force Performance Level: auto
   
  (rickslab-gpu-utils-env) marjorie@grendel:~/gpu-utils$ ./gpu-ls --pstate
OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: AMD: 1
AMD: Wattman features enabled: 0xfffd7fff
1 total GPUs, 0 rw, 1 r-only, 0 w-only

Card Number: 0
   Vendor:  AMD 
   Readable: True
   Writable: False
   Compute: False
   Device ID: {'device': '0x1638', 'subsystem_device': '0x1636', 'subsystem_vendor': '0x1002', 'vendor': '0x1002'}
   Decoded Device ID: Cezanne
   PCIe ID: 30:00.0
   GPU Type: APU
   HWmon: /sys/class/drm/card0/device/hwmon/hwmon1
   Card Path: /sys/class/drm/card0/device
   System Card Path: /sys/devices/pci0000:00/0000:00:08.1/0000:30:00.0

   ### P-State Data #################################
   ##################################################
   PP OD States:
   SCLK:                   MCLK:
   ##################################################
   All Pstates:
   mclk:
      0: 1600Mhz
   dcefclk:
      0: 400Mhz, 1: 464Mhz, 2: 514Mhz, 3: 576Mhz, 4: 626Mhz, 5: 685Mhz, 6: 757Mhz, 7: 847Mhz
   socclk:
      0: 400Mhz, 1: 445Mhz, 2: 520Mhz, 3: 600Mhz, 4: 678Mhz, 5: 780Mhz, 6: 866Mhz, 7: 975Mhz
   fclk:
      0: 1600Mhz, 1: 445Mhz, 2: 520Mhz, 3: 600Mhz, 4: 678Mhz, 5: 780Mhz, 6: 866Mhz, 7: 975Mhz
   sclk:
      0: 200Mhz, 1: 400Mhz, 2: 1900Mhz, 3: 600Mhz, 4: 678Mhz, 5: 780Mhz, 6: 866Mhz, 7: 975Mhz

(rickslab-gpu-utils-env) marjorie@grendel:~/gpu-utils$ ./gpu-ls --clinfo

OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: AMD: 1
AMD: Wattman features enabled: 0xfffd7fff
1 total GPUs, 0 rw, 1 r-only, 0 w-only

Card Number: 0
   Vendor:  AMD 
   Readable: True
   Writable: False
   Compute: False
   Device ID: {'device': '0x1638', 'subsystem_device': '0x1636', 'subsystem_vendor': '0x1002', 'vendor': '0x1002'}
   Decoded Device ID: Cezanne
   Card Model: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne (rev c9)
   Display Card Model:  Cezanne
   PCIe ID: 30:00.0
      Link Speed: 8.0 GT/s PCIe
      Link Width: 16
   ##################################################
   Driver: amdgpu
   vBIOS Version: 113-CEZANNE-017
   Compute Platform: None
   GPU Type: APU
   HWmon: /sys/class/drm/card0/device/hwmon/hwmon1
   Card Path: /sys/class/drm/card0/device
   System Card Path: /sys/devices/pci0000:00/0000:00:08.1/0000:30:00.0
   ##################################################
   Current Power (W): 0.002
   ##################################################
   Current Memory Loading (%): None
   Current GTT Memory Usage (%): 10.372
   Current VRAM Usage (%): 11.290
   Current  Temps (C): {'edge': 28.0}
   Critical Temps (C): {}
   Current Voltages (V): {'vddgfx': 887, 'vddnb': 1099}
      Vddc Range: ['', '']
   Current Clk Frequencies (MHz): {'sclk': 400.0}
   Current SCLK P-State: [1, '400Mhz']
      SCLK Range: ['', '']
   Current MCLK P-State: [0, '1600Mhz']
      MCLK Range: ['', '']
   Power DPM State: performance
   Power DPM Force Performance Level: auto

(rickslab-gpu-utils-env) marjorie@grendel:~/gpu-utils$ ./gpu-ls --raw

OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: AMD: 1
AMD: Wattman features enabled: 0xfffd7fff
1 total GPUs, 0 rw, 1 r-only, 0 w-only

Card Number: 0
   Vendor:  AMD 
   Readable: True
   Writable: False
   Compute: False
   Device ID: {'device': '0x1638', 'subsystem_device': '0x1636', 'subsystem_vendor': '0x1002', 'vendor': '0x1002'}
   Decoded Device ID: Cezanne
   PCIe ID: 30:00.0
   GPU Type: APU
   HWmon: /sys/class/drm/card0/device/hwmon/hwmon1
   Card Path: /sys/class/drm/card0/device
   System Card Path: /sys/devices/pci0000:00/0000:00:08.1/0000:30:00.0

### Raw Diver File Data ##########################
### DEVICE #######################################
### File: hdcp_srm, SensorKey: None, Label: Not defined in gpu-utils
    BINARY
### File: uevent, SensorKey: None, Label: Not defined in gpu-utils
    DRIVER=amdgpu
    PCI_CLASS=30000
    PCI_ID=1002:1638
    PCI_SUBSYS_ID=1002:1636
    PCI_SLOT_NAME=0000:30:00.0
    MODALIAS=pci:v00001002d00001638sv00001002sd00001636bc03sc00i00
### File: resource5, SensorKey: None, Label: Not defined in gpu-utils
    PermissionError
### File: resource0_wc, SensorKey: None, Label: Not defined in gpu-utils
    PermissionError
### File: gpu_metrics, SensorKey: None, Label: Not defined in gpu-utils
    BINARY
### File: power_dpm_force_performance_level, SensorKey: power_dpm_force, Label: Power DPM Force Performance Level
    auto
### File: serial_number, SensorKey: None, Label: Not defined in gpu-utils
    
### File: product_name, SensorKey: None, Label: Not defined in gpu-utils
    
### File: power_state, SensorKey: None, Label: Not defined in gpu-utils
    D0
### File: smartshift_apu_power, SensorKey: None, Label: Not defined in gpu-utils
    6%
### File: broken_parity_status, SensorKey: None, Label: Not defined in gpu-utils
    0
### File: power_dpm_state, SensorKey: power_dpm_state, Label: Power DPM State
    performance
### File: mem_info_vram_vendor, SensorKey: None, Label: Not defined in gpu-utils
    unknown
### File: gpu_busy_percent, SensorKey: loading, Label: Current GPU Loading (%)
    1
### File: subsystem_device, SensorKey: id, Label: Device ID
    0x1636
### File: rom, SensorKey: None, Label: Not defined in gpu-utils
    PermissionError
### File: dma_mask_bits, SensorKey: None, Label: Not defined in gpu-utils
    44
### File: mem_info_preempt_used, SensorKey: None, Label: Not defined in gpu-utils
    0
### File: vendor, SensorKey: id, Label: Device ID
    0x1002
### File: pp_table, SensorKey: None, Label: Not defined in gpu-utils
    OSError
### File: local_cpus, SensorKey: None, Label: Not defined in gpu-utils
    00000fff
### File: pp_dpm_mclk, SensorKey: mclk_ps, Label: Current MCLK P-State
    0: 1600Mhz *
### File: class, SensorKey: None, Label: Not defined in gpu-utils
    0x030000
### File: reset, SensorKey: None, Label: Not defined in gpu-utils
    PermissionError
### File: smartshift_dgpu_power, SensorKey: None, Label: Not defined in gpu-utils
    0%
### File: numa_node, SensorKey: None, Label: Not defined in gpu-utils
    -1
### File: resource, SensorKey: None, Label: Not defined in gpu-utils
    0x00000000d0000000 0x00000000dfffffff 0x000000000014220c
    0x0000000000000000 0x0000000000000000 0x0000000000000000
    0x00000000e0000000 0x00000000e01fffff 0x000000000014220c
    0x0000000000000000 0x0000000000000000 0x0000000000000000
    0x000000000000e000 0x000000000000e0ff 0x0000000000040101
    0x00000000fcc00000 0x00000000fcc7ffff 0x0000000000040200
    0x00000000000c0000 0x00000000000dffff 0x0000000000000212
    0x0000000000000000 0x0000000000000000 0x0000000000000000
    0x0000000000000000 0x0000000000000000 0x0000000000000000
    0x0000000000000000 0x0000000000000000 0x0000000000000000
    0x0000000000000000 0x0000000000000000 0x0000000000000000
    0x0000000000000000 0x0000000000000000 0x0000000000000000
    0x0000000000000000 0x0000000000000000 0x0000000000000000
### File: rescan, SensorKey: None, Label: Not defined in gpu-utils
    PermissionError
### File: max_link_width, SensorKey: None, Label: Not defined in gpu-utils
    16
### File: resource2_wc, SensorKey: None, Label: Not defined in gpu-utils
    PermissionError
### File: pcie_replay_count, SensorKey: None, Label: Not defined in gpu-utils
    0
### File: pp_dpm_pcie, SensorKey: None, Label: Not defined in gpu-utils
    
### File: msi_bus, SensorKey: None, Label: Not defined in gpu-utils
    1
### File: device, SensorKey: id, Label: Device ID
    0x1638
### File: mem_info_vram_used, SensorKey: mem_vram_used, Label: Current VRAM Used (GB)
    221102080
### File: boot_vga, SensorKey: None, Label: Not defined in gpu-utils
    1
### File: pp_num_states, SensorKey: None, Label: Not defined in gpu-utils
    states: 1
    0 default
### File: current_link_width, SensorKey: link_wth, Label: Link Width
    16
### File: pp_cur_state, SensorKey: None, Label: Not defined in gpu-utils
    -22
### File: max_link_speed, SensorKey: None, Label: Not defined in gpu-utils
    16.0 GT/s PCIe
### File: pp_sclk_od, SensorKey: None, Label: Not defined in gpu-utils
    0
### File: resource4, SensorKey: None, Label: Not defined in gpu-utils
    PermissionError
### File: pp_dpm_dcefclk, SensorKey: None, Label: Not defined in gpu-utils
    0: 400Mhz 
    1: 464Mhz 
    2: 514Mhz 
    3: 576Mhz 
    4: 626Mhz 
    5: 685Mhz 
    6: 757Mhz 
    7: 847Mhz *
### File: mem_info_vis_vram_used, SensorKey: None, Label: Not defined in gpu-utils
    221102080
### File: local_cpulist, SensorKey: None, Label: Not defined in gpu-utils
    0-11
### File: pp_dpm_socclk, SensorKey: None, Label: Not defined in gpu-utils
    0: 400Mhz 
    1: 445Mhz 
    2: 520Mhz 
    3: 600Mhz 
    4: 678Mhz 
    5: 780Mhz 
    6: 866Mhz 
    7: 975Mhz *
### File: mem_info_vis_vram_total, SensorKey: None, Label: Not defined in gpu-utils
    2147483648
### File: driver_override, SensorKey: None, Label: Not defined in gpu-utils
    (null)
### File: d3cold_allowed, SensorKey: None, Label: Not defined in gpu-utils
    1
### File: irq, SensorKey: None, Label: Not defined in gpu-utils
    36
### File: revision, SensorKey: None, Label: Not defined in gpu-utils
    0xc9
### File: resource2, SensorKey: None, Label: Not defined in gpu-utils
    PermissionError
### File: pp_dpm_fclk, SensorKey: None, Label: Not defined in gpu-utils
    0: 1600Mhz *
### File: current_link_speed, SensorKey: link_spd, Label: Link Speed
    8.0 GT/s PCIe
### File: reset_method, SensorKey: None, Label: Not defined in gpu-utils
    bus
### File: pp_power_profile_mode, SensorKey: ppm, Label: Power Profile Mode
    1 3D_FULL_SCREEN 
     3          VIDEO*
     4             VR 
     5        COMPUTE 
     6         CUSTOM
### File: mem_info_gtt_used, SensorKey: mem_gtt_used, Label: Current GTT Memory Used (GB)
    336211968
### File: consistent_dma_mask_bits, SensorKey: None, Label: Not defined in gpu-utils
    44
### File: mem_info_vram_total, SensorKey: mem_vram_total, Label: Total VRAM (GB)
    2147483648
### File: vbios_version, SensorKey: vbios, Label: vBIOS Version
    113-CEZANNE-017
### File: resource0, SensorKey: None, Label: Not defined in gpu-utils
    PermissionError
### File: pp_od_clk_voltage, SensorKey: pp_od_clk_voltage, Label: read/write driver file
    
### File: config, SensorKey: None, Label: Not defined in gpu-utils
    BINARY
### File: ari_enabled, SensorKey: None, Label: Not defined in gpu-utils
    0
### File: pp_force_state, SensorKey: None, Label: Not defined in gpu-utils
    
### File: remove, SensorKey: None, Label: Not defined in gpu-utils
    PermissionError
### File: thermal_throttling_logging, SensorKey: None, Label: Not defined in gpu-utils
    0000:30:00.0: thermal throttling logging enabled, with interval 60 seconds
### File: enable, SensorKey: fan_enable, Label: Fan Enable
    1
### File: product_number, SensorKey: None, Label: Not defined in gpu-utils
    
### File: pp_mclk_od, SensorKey: None, Label: Not defined in gpu-utils
    0
### File: mem_info_gtt_total, SensorKey: mem_gtt_total, Label: Total GTT Memory (GB)
    3221225472
### File: modalias, SensorKey: None, Label: Not defined in gpu-utils
    pci:v00001002d00001638sv00001002sd00001636bc03sc00i00
### File: subsystem_vendor, SensorKey: id, Label: Device ID
    0x1002
### File: pp_dpm_sclk, SensorKey: sclk_ps, Label: Current SCLK P-State
    0: 200Mhz 
    1: 700Mhz 
    2: 1900Mhz *
### HWMON ########################################
### File: uevent, SensorKey: None, Label: Not defined in gpu-utils
    
### File: in1_label, SensorKey: None, Label: Not defined in gpu-utils
    vddnb
### File: freq1_label, SensorKey: None, Label: Not defined in gpu-utils
    sclk
### File: in0_input, SensorKey: None, Label: Not defined in gpu-utils
    1362
### File: power1_label, SensorKey: None, Label: Not defined in gpu-utils
    slowPPT
### File: temp1_label, SensorKey: None, Label: Not defined in gpu-utils
    edge
### File: in1_input, SensorKey: None, Label: Not defined in gpu-utils
    1099
### File: power1_average, SensorKey: power, Label: Current Power (W)
    4000
### File: freq1_input, SensorKey: None, Label: Not defined in gpu-utils
    1900000000
### File: temp1_input, SensorKey: None, Label: Not defined in gpu-utils
    28000
### File: in0_label, SensorKey: None, Label: Not defined in gpu-utils
    vddgfx
### File: name, SensorKey: None, Label: Not defined in gpu-utils
    amdgpu
##################################################

(rickslab-gpu-utils-env) marjorie@grendel:~/gpu-utils$ ./gpu-mon --gui

(rickslab-gpu-utils-env) marjorie@grendel:~/gpu-utils$ ./gpu-pac

OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: AMD: 1
AMD: Wattman features enabled: 0xfffd7fff
All GPUs:
1 total GPUs, 0 rw, 1 r-only, 0 w-only

Compatible GPUs:
None are writable, exiting...

(rickslab-gpu-utils-env) marjorie@grendel:~/gpu-utils$ ./gpu-plot

OS command [nvidia-smi] executable found: [/usr/bin/nvidia-smi]
Detected GPUs: AMD: 1
AMD: Wattman features enabled: 0xfffd7fff
Compatible GPUs:
1 total GPUs, 0 rw, 1 r-only, 0 w-only

gpu-plot waiting for initial data...

3 replies

Ricks-Lab May 18, 2022
Maintainer Author

Thanks @Marjorie-R ! Yes, latest version will indicate a GPU is not writable when read pp_od_clk_voltage file is possible but returns nothing. I noticed that pp_od_clk_voltage doesn't show up in the disabled list, so I made some modifications in how the read_disabled and read_skip lists are managed. Once I complete testing on my side, I will issue a new release candidate.

Are you seeing color coded text output for gpu-ls? If so, does it have any readability issues?

csecht May 18, 2022

I had replied through my e-mail client, but it's not showing up here, so here is an update of that reply:
Yes, the colors look good. I like how the gpu-ls --pstates provides contrast for the current active states.
For considerations of colorblind users, I found that the Firefox browser has an add-on to simulate different types of color-blindness. I dropped in a screenshot png of the gpu-ls terminal output, and the simulated colors for deuteranopia are readable. (In my email I said that the red background for the AMD vendor would appear as grey, but that doesn't seem to be the case using the simulator).

Ricks-Lab May 20, 2022
Maintainer Author

Thanks @csecht ! It has been a while. Life has been busy, but I am now back to GitHub.
Since the color scheme does combine red and green or blue and yellow, everything should be readable for those with the 2 major types of color blindness. Maybe the will not see the difference between the highlighting of AMD and nVidia, but since it is spelled out, in contrast compatible with both types it should be readable. Maybe green and white may not be easily distinguishable in formatted --raw, but still readable. I have added --no_markup to eliminate color formatting.

Marjorie-R · 2022-05-18T16:42:50Z

Marjorie-R
May 18, 2022

Hi Rick,

in a supplementary you asked:

Are you seeing color coded text output for gpu-ls? If so, does it have any readability issues?

As a normally sighted person (no colour blindness) the text seems to display OK to me:

--
Marjorie

1 reply

Ricks-Lab May 19, 2022
Maintainer Author

Thanks for sharing the image. First time to see it in light theme, since all my systems use dark theme. Perhaps I should make a light themed terminal window on one of my test systems.

Ricks-Lab · 2022-05-20T05:46:24Z

Ricks-Lab
May 20, 2022
Maintainer Author

Release candidate 3.7.0-RC3 is ready for testing. Only minor changes from RC2. Plan to release to PyPI in a week if no issues reported here.

1 reply

Ricks-Lab May 21, 2022
Maintainer Author

I noticed repository had been cloned by a significant number of unique users, so I decided to release v3.7.1 to PyPI. Latest patch version was due to an issue with the formatting of ppm output.

Ricks-Lab · 2022-06-04T03:18:07Z

Ricks-Lab
Jun 4, 2022
Maintainer Author

v3.7.5 is released to both PyPI and rickslab.com. The 3.7.x improvements are complete in this version. I plan to provide a locked version for the official Debian package by October. Please help to provide feedback so we can assure robustness of the Debian package. Thanks!

4 replies

csecht Jun 5, 2022

Thanks for the update! I notice that gpu-ls --table, gpu-mon, and gpu-mon --gui now list Sclk and Mclk values with their units, whereas previous versions did not because units are provided in the row headers. I think the previous format is preferred for it's cleaner look.

...:~/CountBOINCtasks-master$ gpu-ls --table
Detected GPUs: AMD: 3
AMD: amdgpu version: 21.10-1247438
AMD: Wattman features enabled: 0xfffd7fff
3 total GPUs, 2 rw, 0 r-only, 0 w-only

Status of Compatible GPUs:
┌─────────────┬────────────────────┬────────────────────┐
│Card #       │       card1        │       card2        │
├─────────────┼────────────────────┼────────────────────┤
│Model        │       RX 570       │       RX 570       │
│GPU Load %   │        100         │        100         │
│Mem Load %   │         93         │         94         │
│VRAM Usage % │       69.845       │       81.867       │
│GTT Usage %  │       5.166        │       6.298        │
│Power (W)    │       72.133       │       83.238       │
│Power Cap (W)│       120.0        │       120.0        │
│Energy (kWh) │     1.000e-06      │     1.000e-06      │
│T (C)        │        74.0        │        75.0        │
│VddGFX (mV)  │        906         │        906         │
│Fan Spd (%)  │         41         │         41         │
│Sclk (MHz)   │      1071Mhz       │      1071Mhz       │
│Sclk Pstate  │         6          │         6          │
│Mclk (MHz)   │      1850Mhz       │      1850Mhz       │
│Mclk Pstate  │         2          │         2          │
│Perf Mode    │     5-COMPUTE      │     5-COMPUTE      │
└─────────────┴────────────────────┴────────────────────┘

System:

Using rickslab-gpu-utils  3.7.5 
Using python 3.8.10
Using Linux Kernel: 5.4.0-84-generic
Using Linux distribution: Ubuntu 20.04.3 LTS
AMD: amdgpu version: 21.10-1247438

Ricks-Lab Jun 8, 2022
Maintainer Author

Thanks for raising this issue. It was definitely not intentional. I consolidated formatting plot and table items into a single helper function for consistency and simplicity. I overlooked this detail. I have pushed a version, but still have not tested for plot of incomplete GPU data.

Ricks-Lab Jun 8, 2022
Maintainer Author

Completed change has been pushed.

Ricks-Lab Jun 19, 2022
Maintainer Author

I have released the fix to both PyPI and debian.rickslab.com.

csecht · 2022-10-11T07:24:01Z

csecht
Oct 11, 2022

The gpu-ls colors look good; Intel showing with a blue background, as @Marjorie-R showed, AMD with red and the data in a blue hue. I’m guessing that different GPU types will have different color backgrounds? Red will just be seen as grey for most color blind folks. Blue, orange and yellow do show nice color contrasts though. From my tinkering with tkinter apps, I’ve found these colors generally work across the board: blue ‘#0000ff’ orange ‘#b4b400' khaki '#f0e68c' yellow ‘#ffff00’ orchid #9898f9’

…

On May 17, 2022, at 11:34 PM, Rick ***@***.***> wrote: Thanks @Marjorie-R <https://github.com/Marjorie-R> ! Yes, latest version will indicate a GPU is not writable when read pp_od_clk_voltage file is possible but returns nothing. I noticed that pp_od_clk_voltage doesn't show up in the disabled list, so I made some modifications in how the read_disabled and read_skip lists are managed. Once I complete testing on my side, I will issue a new release candidate. Are you seeing color coded text output for gpu-ls? If so, does it have any readability issues? — Reply to this email directly, view it on GitHub <#109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMVCQR66XI36QL4X3L3MZ3VKRXMNANCNFSM4YULGNEA>. You are receiving this because you commented.

0 replies

Ricks-Lab · 2022-10-15T11:47:05Z

Ricks-Lab
Oct 15, 2022
Maintainer Author

I am using red for AMD, green for NVidia, and blue for all others. I have added the --no_markup option for plain text instead of using color blind friendly colors. I am using basic escape codes for color coding, so there is not a lot of flexibility.

0 replies

Ricks-Lab · 2024-05-29T22:42:54Z

Ricks-Lab
May 29, 2024
Maintainer Author

Testers Needed!

I have released latest release candidate on PyPI:
3.9.0rc2

Please remove all other versions of rickslab-gpu-utils and pip install RC2:
pip install rickslab-gpu-utils==3.9.0rc2

Let me know of any issues here. Thanks!

0 replies

Release News #109

Ricks-Lab Mar 5, 2021 Maintainer

Replies: 24 comments · 29 replies

Ricks-Lab Mar 17, 2021 Maintainer Author

Ricks-Lab Mar 18, 2021 Maintainer Author

Ricks-Lab Mar 18, 2021 Maintainer Author

Ricks-Lab Mar 18, 2021 Maintainer Author

Ricks-Lab Mar 18, 2021 Maintainer Author

Ricks-Lab Mar 23, 2021 Maintainer Author

Ricks-Lab Mar 19, 2021 Maintainer Author

Ricks-Lab Mar 20, 2021 Maintainer Author

Ricks-Lab Mar 22, 2021 Maintainer Author

Ricks-Lab Mar 24, 2021 Maintainer Author

Ricks-Lab Mar 24, 2021 Maintainer Author

Ricks-Lab Mar 24, 2021 Maintainer Author

Ricks-Lab Mar 26, 2021 Maintainer Author

Ricks-Lab Mar 27, 2021 Maintainer Author

Ricks-Lab Mar 27, 2021 Maintainer Author

Ricks-Lab Mar 28, 2021 Maintainer Author

Ricks-Lab Mar 27, 2021 Maintainer Author

Ricks-Lab Apr 11, 2021 Maintainer Author

Ricks-Lab May 14, 2022 Maintainer Author

Ricks-Lab May 17, 2022 Maintainer Author

Ricks-Lab May 18, 2022 Maintainer Author

Ricks-Lab May 20, 2022 Maintainer Author

Ricks-Lab May 19, 2022 Maintainer Author

Ricks-Lab May 20, 2022 Maintainer Author

Ricks-Lab May 21, 2022 Maintainer Author

Ricks-Lab Jun 4, 2022 Maintainer Author

Ricks-Lab Jun 8, 2022 Maintainer Author

Ricks-Lab Jun 8, 2022 Maintainer Author

Ricks-Lab Jun 19, 2022 Maintainer Author

Ricks-Lab Oct 15, 2022 Maintainer Author

Ricks-Lab May 29, 2024 Maintainer Author

Ricks-Lab
Mar 5, 2021
Maintainer

Replies: 24 comments 29 replies

Ricks-Lab
Mar 17, 2021
Maintainer Author

Ricks-Lab Mar 18, 2021
Maintainer Author

Ricks-Lab Mar 18, 2021
Maintainer Author

Ricks-Lab Mar 18, 2021
Maintainer Author

Ricks-Lab Mar 18, 2021
Maintainer Author

Ricks-Lab Mar 23, 2021
Maintainer Author

Ricks-Lab Mar 19, 2021
Maintainer Author

Ricks-Lab Mar 20, 2021
Maintainer Author

Ricks-Lab
Mar 22, 2021
Maintainer Author

Ricks-Lab Mar 24, 2021
Maintainer Author

Ricks-Lab Mar 24, 2021
Maintainer Author

Ricks-Lab Mar 24, 2021
Maintainer Author

Ricks-Lab Mar 26, 2021
Maintainer Author

Ricks-Lab
Mar 27, 2021
Maintainer Author

Ricks-Lab Mar 27, 2021
Maintainer Author

Ricks-Lab Mar 28, 2021
Maintainer Author

Ricks-Lab Mar 27, 2021
Maintainer Author

Ricks-Lab Apr 11, 2021
Maintainer Author

Ricks-Lab
May 14, 2022
Maintainer Author

Ricks-Lab
May 17, 2022
Maintainer Author

Ricks-Lab May 18, 2022
Maintainer Author

Ricks-Lab May 20, 2022
Maintainer Author

Ricks-Lab May 19, 2022
Maintainer Author

Ricks-Lab
May 20, 2022
Maintainer Author

Ricks-Lab May 21, 2022
Maintainer Author

Ricks-Lab
Jun 4, 2022
Maintainer Author

Ricks-Lab Jun 8, 2022
Maintainer Author

Ricks-Lab Jun 8, 2022
Maintainer Author

Ricks-Lab Jun 19, 2022
Maintainer Author

Ricks-Lab
Oct 15, 2022
Maintainer Author

Ricks-Lab
May 29, 2024
Maintainer Author