DAOS-8331 client: Add client side metrics (#14030) #14204
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Bug-tracker data: |
mjmac
changed the title
daosbuild1
previously requested changes
Apr 20, 2024
|
Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14204/1/execution/node/175/log |
|
Test stage Functional on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-14204/1/testReport/ |
mjmac
force-pushed
the
mjmac/DAOS-8331-2.4-backport
branch
2 times, most recently
from
1070f9d
to
c6f802b
Compare
daosbuild1
previously requested changes
Apr 21, 2024
|
Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14204/3/execution/node/205/log |
mjmac
force-pushed
the
mjmac/DAOS-8331-2.4-backport
branch
from
c6f802b
to
2aa0249
Compare
daosbuild1
reviewed
Apr 21, 2024
|
Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14204/4/execution/node/1263/log |
mjmac
force-pushed
the
mjmac/DAOS-8331-2.4-backport
branch
from
2aa0249
to
91ab43a
Compare
daosbuild1
reviewed
Apr 22, 2024
daosbuild1
reviewed
Apr 22, 2024
mjmac
force-pushed
the
mjmac/DAOS-14850-2.4
branch
from
542aa10
to
1ce8e5c
Compare
mjmac
force-pushed
the
mjmac/DAOS-8331-2.4-backport
branch
from
91ab43a
to
d6d083e
Compare
daosbuild1
reviewed
Apr 23, 2024
daosbuild1
reviewed
Apr 24, 2024
Adding tests for WAL commit, reply, and checkpoint metrics. Signed-off-by: Phil Henderson <phillip.henderson@intel.com>
Required-githooks: true Change-Id: I17ac20dd02462edcf09af1267a66e31d39eac691 Signed-off-by: Michael MacDonald <mjmac@google.com>
This commit comprises two separate patches to enable optional collection and export of client-side telemetry. The daos_agent configuration file includes new parameters to control collection and export of per-client telemetry. If the telemetry_port option is set, then per-client telemetry will be published in Prometheus format for real-time sampling of client processes. By default, the client telemetry will be automatically cleaned up on client exit, but may be optionally retained for some amount of time after client exit in order to allow for a final sample to be read. Example daos_agent.yml updates: telemetry_port: 9192 # export on port 9192 telemetry_enable: true # enable client telemetry for all connected clients telemetry_retain: 1m # retain metrics for 1 minute after client exit If telemetry_enable is false (default), client telemetry may be enabled on a per-process basis by setting D_CLIENT_METRICS_ENABLE=1 in the environment for clients that should collect telemetry. Notes from the first patch by Di: Move TLS to common, so both client and server can have TLS, which metrics can be attached metrics on it. Add object metrics on the client side, enabled by export D_CLIENT_METRICS_ENABLE=1. And client metrics are organized as "/jobid/pid/xxxxx". During each daos thread initialization, it will created another shmem (pid/xxx), which all metrics of the thread will be attached to. And this metric will be destroyed once the thread exit, though if D_CLIENT_METRICS_RETAIN is set, these client metrics will be retain, and it can be retrieved by daos_metrics --jobid Add D_CLIENT_METRICS_DUMP_PATH dump metrics from current thread once it exit. Some fixes in telemetrics about conv_ptr during re-open the share memory. Add daos_metrics --jobid XXX options to retrieve all metrics of the job. Features: telemetry Required-githooks: true Change-Id: Ib80ff89f39d259e0dce26e0ae8388318f96a3540 Co-authored-by: Di Wang <di.wang@intel.com> Signed-off-by: Di Wang <di.wang@intel.com> Signed-off-by: Michael MacDonald <mjmac@google.com>
mjmac
force-pushed
the
mjmac/DAOS-8331-2.4-backport
branch
from
d6d083e
to
a647825
Compare
daosbuild1
reviewed
Apr 24, 2024
jolivier23
approved these changes
Apr 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit comprises two separate patches to enable optional
collection and export of client-side telemetry.
The daos_agent configuration file includes new parameters to control
collection and export of per-client telemetry. If the telemetry_port option
is set, then per-client telemetry will be published in Prometheus format
for real-time sampling of client processes. By default, the client telemetry
will be automatically cleaned up on client exit, but may be optionally
retained for some amount of time after client exit in order to allow for
a final sample to be read.
Example daos_agent.yml updates:
telemetry_port: 9192 # export on port 9192
telemetry_enable: true # enable client telemetry for all connected clients
telemetry_retain: 1m # retain metrics for 1 minute after client exit
If telemetry_enable is false (default), client telemetry may be enabled on
a per-process basis by setting D_CLIENT_METRICS_ENABLE=1 in the
environment for clients that should collect telemetry.
Notes from the first patch by Di:
Move TLS to common, so both client and server can have TLS,
which metrics can be attached metrics on it.
Add object metrics on the client side, enabled by
export D_CLIENT_METRICS_ENABLE=1. And client metrics are organized
as "/jobid/pid/xxxxx".
During each daos thread initialization, it will created another
shmem (pid/xxx), which all metrics of the thread will be attached
to. And this metric will be destroyed once the thread exit, though
if D_CLIENT_METRICS_RETAIN is set, these client metrics will be
retain, and it can be retrieved by
daos_metrics --jobid
Add D_CLIENT_METRICS_DUMP_PATH dump metrics from current thread
once it exit.
Some fixes in telemetrics about conv_ptr during re-open the
share memory.
Add daos_metrics --jobid XXX options to retrieve all metrics
of the job.
Required-githooks: true
Change-Id: Ib80ff89f39d259e0dce26e0ae8388318f96a3540
Signed-off-by: Di Wang di.wang@intel.com
Signed-off-by: Michael MacDonald mjmac@google.com
Co-authored-by: Di Wang di.wang@intel.com
Signed-off-by: Michael MacDonald mjmac@google.com