Harvest update from 23.0.2

[KyndrylLaszlo]

Hi, I would like to update the harvest application, but I did not find straight forward compatibility matrix regarding the harvest/ONTAP and other component versions.

Few information regarding the production environment below

Host OS - > Red Hat Enterprise Linux release 8.10 (Ootpa)
Current harvest version - >harvest-23.02.0-1.x86_64
Visualization Dashboard - >Grafana v10.0.5 (1413841850)
Exporter - > InfluxDB 2.7.1

My questions are the following:

1. Which Harvest version is recommended for ONTAP 9.11.1P15.
2. In case that i can go to the latest release shall i do the update directly?

Thank you in advance.

[cgrinds]

hi @KyndrylLaszlo,

The latest release is recommended, and upgrading depends on how you installed. If you installed via RPM, use the upgrade instructions at https://netapp.github.io/harvest/nightly/install/package-managers/#upgrade

[KyndrylLaszlo]

Thank you for the quick answer, I appreciate that.

[KyndrylLaszlo]

Hi, please allow me to raise my further questions here, as it is related to the upgrade.

I've upgraded the harvest application to the latest code and enabled the REST collector as well to collect the "cifs_session" count, only this kind of data collection was enabled regarding the REST collector. (the other REST collection related metrics was commented out on the default.yaml file). So the Zapi and the REST collector was enabled on the same time and the data collection was working, but after a couple of hours I've noticed that the harvest application is eating the memory of the HOST OS and had to do a restart to prevent the OS crash. Usually the harvest was operating with 1,5-2,5 GB memory in case that only we are using the Zapi for data collection. If both of the collection method (Zapi, REST )was enabled the HOST OS just running out of memory ( 8GB memory was not enough). At the moment only the Zapi collection strategy is enabled and the memory consumption is normal.

There is a possibility that the Zapi and the REST collector can not do the job on the same time and cause memory issues on the HOST OS ? Shall we try to use only one kind of collector method ? I mean maybe we need to go only with the REST and completely disable the Zapi collection?

Side note:
Since the upgrade I can see "cache-max-memory-size exceeded" error messages regarding the influx export on the log files. This value should be increased on the influxdb size ?

Your suggestions will be highly appreciated. Thank you in advance.

[rahulguptajss]

Hi @KyndrylLaszlo,

As noted here, the CIFSSession REST template may generate a high number of metrics, which is the likely reason that you see Harvest taking more memory. It shouldn't matter as such to run ZAPI and REST together. Memory consumed is related to the number of records collected for that relevant object.

If you can share your logs to ng-harvest-files@netapp.com, we can take a look at the number of records collected for CIFSSession. You may also check the same by calling the below CLI for ONTAP:

curl -s -k -u USER:PASS 'https://CLUSTER_IP/api/protocols/cifs/sessions?fields=*'

Please replace USER, PASS, and CLUSTER_IP as applicable.

[KyndrylLaszlo]

Hi, thank you for the information, I will extend the memory of the HOST OS and collecting the cifs session logs, once im done will send the log files as requested.

Other note>
I can see on every poller log files the message below.
"cluster1","collector":"ZapiPerf:VolumeSvm","error":"unable to emit object: svm_vol, uuid: ZapiPerf, err=API request rejected: {"code":"internal error","message":"unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1086557225/1073741824)"}","exporter":"influxdb1","caller":"collector/collector.go:508","time":"2024-08-27T08:14:18+02:00","message":"export data"}

shall we extend this cache-max-memory-size ?

Thanks.

[rahulguptajss]

@KyndrylLaszlo The error cache-max-memory-size exceeded indicates that the memory allocated for the cache in InfluxDB has been surpassed. The values (1086557225/1073741824) show that the current cache size is 1086557225 bytes, which exceeds the maximum allowed size of 1073741824 bytes (1 GB).

Yes, You can increase the cache-max-memory-size setting in the InfluxDB configuration file.

[KyndrylLaszlo]

Thanks.

[KyndrylLaszlo]

Hi, since the upgrade we noticed that the influxdb write and the read operation are way bigger then is was before. Before the upgrade usually ~30 MB/s throughput was all the time, but after the upgrade the read/write operation are significantly larger (~60-120MB/s)

I already disabled a few obj. collection and only the Zapi is activated, please see the entry from the def. yaml cfg files regarding zapi/zapiperf

There is a chance to identify which object collection is causing this kind of behavior ?

Thank you in advance.
zapi-zapiperf.txt

[rahulguptajss]

@KyndrylLaszlo If you can share Harvest logs to ng-harvest-files@netapp.com, we can take a look at which object may be driving this traffic.

[KyndrylLaszlo]

Good morning, thank you for the analysis.

I've upgraded the harvest instances on 8th of august, since then the throughput is significantly increased, CIFSession for REST was disabled one that day as well, because we had serious memory problem on the harvest HOST OS ( 8GB memory was not enough). I noticed that the zapi is also capable of collecting metrics regarding the CIFSession and i just enabled it on all harvest instances and it was collecting the CIFSession metrics for 2,5 weeks maybe, but i disabled the zapi CIFSession metrics as well, because the influxdb itself is doubled regarding the size and just keep growing. The usual size was around 700 GB, but we reached almost 1,7 TB! zapi CIFSession was disabled on 27th of august and also i disabled the following unnecessary metrics as well on 2nd of September

EmsDestination
CloudTarget
copy manager
FlexCache
NtpServer
Security:
SecurityAccount:
SecurityAuditDestination
SecurityCert:
SecurityLogin:
Status_7mode
ExtCacheObj
FlexCache
Iwarp
Rwctx
CopyManager
WAFLCompBin
NFSv42
NFSv42Node

The cifs session count was not visible on the influxdb (with zapi ), we just need to count the cifs sessions numbers based on the contract. The REST CIFSession part was working fine, but consumed to much memory and need to disabled. The higher throughput is with us since the upgrade. I disabled several metric to normalize the influx db size\growing. So at the moment I'm not sure what is the exact size for the influxdb, because we have 30 days retention for the data which is resided the influxdb and we have to wait until the data will expired and purged. At the moment the influxdb is not showing significant data grow.

This error message can be ignored (already disabld it)- "level":"error","Poller":"xxxxxx","collector":"ZapiPerf:NFSv42","error":"connection error => Post "https://xxxxxxxxxx/servlets/netapp.servlets.admin.XMLrequest_filer\": dial tcp: lookup xxxxxxxxxxxx: no such host","request":"perf-object-instance-list-info-iter","batchTag":"","caller":"zapiperf/zapiperf.go:1358","time":"2024-07-19T14:06:11+02:00","message":"InvokeBatchRequest failed"}

panel querry.txt

[rahulguptajss]

Thanks @KyndrylLaszlo The list of disabled templates is helpful. Based on the metrics count, Could you try disabling the ZapiPerf:SMB2 template? This template is generating a significant amount of metrics due to the creation of histogram metrics. Based on logs, This template is generating ~250k metrics across all pollers per poll.

[KyndrylLaszlo]

I will disable the mentioned ZapiPerf:SMB2 as it was confirmed that we don't have any contractual requirements regarding this metrics. Good finding, thanks for the help.

[KyndrylLaszlo]

since I've disabled the ZapiPerf:SMB2, the write to the volume is significantly decreased.

[cgrinds]

That's great to hear @KyndrylLaszlo. Thanks for letting us know