Limit which instances are collected

[luddite516]

We have a use case where we are collecting file lock statistics for a specific vserver. I have built the following template to collect lock stats:

name: Locks
query: lock-get-iter
object: lock

collect_only_labels: true

counters:
lock-info:
- ^node
- ^volume
- ^vserver
- ^path
- ^^lockid
- ^client-address => client_ip
- ^smb-open-type => smb_open_type

export_options:
instance_keys:
- lockid
instance_labels:
- vserver
- volume
- node
- path
- client_ip
- smb_open_type

The above template collects all the locks in the cluster, and there may be hundreds of thousands of them.

Is there any way to limit which instances are collected, to reduce load on cluster? For example, limit collection to only a specific vserver or volume?

Thanks.

[cgrinds]

hi @luddite516 that's a great question. ZAPIs have the ability to filter results that way, but Harvest doesn't currently expose that kind of filtering via our template language. We'll take a look at adding that.

The best workaround at the moment, would be to create a new collector that adds the appropriate query to the ZAPI request. If you're interested in trying that, we can point you in the right direction to make the changes while we pickup the template-level changes.

[luddite516]

Thanks @cgrinds . I follow what you are saying- basically copy the zapi collector to a local custom collector, and tweak the code a workaround.

This (the ability to add ZAPI constraints to queries) feels a little bit like feature-creep. It would provide limited value and would likely make the templates more complex. But let's see if anyone else has a need for this.

[cgrinds]

that's right, copy the zapi collector and change it to also send a query clause in the ZAPI request it creates. The query section would have your specific svm, volume, etc.

I agree it may not be generally useful enough to warrant adding (and more importantly supporting). I want to discuss with the broader team and agree with you 100% that we'll leave this open to see if others have a need.

[luddite516]

@cgrinds, I have a further question related to this.

Because collecting lock data is so 'expensive' and produces so much data, and because we only need to monitor locks at a very low resolution (1 hour or less often), I am now trying to configure Harvest and Prometheus to collect the data only once per hour. I could even go less often- every 4 hours.

I have set up my custom collection template as follows, specifying collection only once every hour:

collector: Zapi

schedule:

• instance: 3600s
• data: 3600s

objects:
Locks: lock.yaml

The collection template works fine. Lock metrics collected only once per hour.

Here is the issue-- The collected metrics only appear in the Harvest endpoint, for maybe 3 minutes after they are collected. Then the metrics disappear. So, if Prometheus does not scrape the endpoint right after Harvest collects, the metrics will not get into the database.

Is this a known issue, or is there something I have to set in harvest that tells it to retain/cache the metrics for longer than 3 min?

[cgrinds]

hi @luddite516 try increasing the cache_max_keep in your Prometheus exporter section and let us know if that works for you.

I think this is what you outlined above, but to make sure I understood, you can also more narrowly target this by creating a Prometheus exporter (with the increased cache_max_keep) that is paired with a Prometheus target that has an increased scrape_interval.