How to apply jitter in Harvest (rpm/deb/native)

[rahulguptajss]

What is jitter?
By default, Harvest sends all ZAPI/REST requests at almost the same time. To smear the collector startup times across a wider window, you can use jitter with a given time, let's say one minute, then all requests are distributed across that minute randomly.

Jitter can be configured for any of the Harvest collectors, including: Ems, Rest, RestPerf, StorageGrid, Zapi, ZapiPerf

Note:

Below steps are applicable for rpm/deb/native installation. In this example, let's assume that Harvest is installed in /opt/harvest
Refer here for NABox.

ZapiPerf

Step 1: Create custom.yaml

Navigate to the zapiperf directory:

cd /opt/harvest/conf/zapiperf

Create custom.yaml:

vi custom.yaml

Update the file with the following content. If you already have custom.yaml then just add below line in the file at top of custom.yaml

jitter: 1m

Step 2: Restart Harvest

RestPerf

Step 1: Create custom.yaml

Navigate to the restperf directory (assuming this should be different from zapiperf):

cd /opt/harvest/conf/restperf

Create custom.yaml:

vi custom.yaml

Update the file with the following content. If you already have custom.yaml then just add below line in the file at top of custom.yaml

jitter: 1m

Step 2: Restart Harvest

[faguayot]

Hello @rahulguptajss

I've applied the change and restart the harvest.

How can I know if it is working? This functionality is only for the performance information, right? I've applied the config at the same level that collector. Is that correct?

[cgrinds]

hi @faguayot Yes, perf collectors only.

Currently jitter is only logged at the debug level.

You can start an individual poller like this:

bin/poller --poller $pollerName --promPort $promPort --collectors ZapiPerf --objects Volume --loglevel 1

and confirm you see a log line like this

2024-04-25T10:25:09-04:00 DBG collector/collector.go:185 > Poller=u2 collector=ZapiPerf:Volume jitter=56.550092395s schedule=24h task=counter

[faguayot]

Hi @cgrinds

I have executed the command that you said but I don't see the jitter parameter in the log.

2024-04-25T16:45:45+02:00 INF collector/collector.go:585 > Collected Poller=POLLER apiMs=232 calcMs=0 collector=ZapiPerf:Volume exportMs=176 instances=178 instancesExported=366 metrics=4806 metricsExported=5228 parseMs=53 pluginMs=1 pollMs=289 skips=0 zBegin=1714056344924

Maybe the jitter was implemented in the nightly version?

Thanks.

[cgrinds]

Hi @faguayot Jitter was added in 23.11 so you should have it on 24.02. Look earlier in your log file for a DBG line that includes jitter. Let's grep for the log lines like so:

bin/poller --poller $pollerName --promPort $promPort --collectors ZapiPerf --objects Volume --loglevel 1 2>&1 | grep jitter

You should see something similar to this:
2024-04-25T11:02:40-04:00 DBG collector/collector.go:185 > Poller=u2 collector=ZapiPerf:Volume jitter=36.13938436s schedule=24h task=counter
2024-04-25T11:02:40-04:00 DBG collector/collector.go:185 > Poller=u2 collector=ZapiPerf:Volume jitter=36.13938436s schedule=10m task=instance
2024-04-25T11:02:40-04:00 DBG collector/collector.go:185 > Poller=u2 collector=ZapiPerf:Volume jitter=36.13938436s schedule=1m task=data

[rahulguptajss]

@faguayot Were you able to check the steps as shared by @cgrinds ?

[faguayot]

Hello @cgrinds

I have tested without luck, I don't see the jitter parameters.

I don't know what I am doing wrong.

[cgrinds]

Hi @faguayot I gave you the wrong grep for 24.02. Sorry about that. In 24.02, jitter was logged as delay and we changed it to be logged as jitter in March.

Try this if you are on 24.02

bin/poller --poller $pollerName --promPort $promPort --collectors ZapiPerf --objects Volume --loglevel 1 2>&1 | grep delay

[faguayot]

Hi chris,

Yes, with that grep I am receiving traces of logs: