Add more Storagegrid Dashboards to Harvest

[erikgruetter]

Hey there guys

At the moment, there is only one Storagegrid dashboard in the new Netapp Harvest Nightly release 22.11.04.
As discussed on Discord, these additional dashboards would be of most use to us:

• Performance General Statistics (CPU, RAM usage, Disk Latency, Network Usage etc.)
• Bucket Performance
• Power Consumption
• Health General Status
• Any Hardware Failures (Disks, Nodes down etc.)

If it helps you prioritise this request, you can add Bank Julius Bär & Co. AG as a company who wants this.

Regards
Erik

[iStep2Step]

Hey,

we would appreciate all points listed from Erik if it is possible.

You can add Landeshauptsadt München for the interest.

Best Regards
Sandro

[adamallmer]

Hi,

we would appreciate all points listed from Erik if it is possible with addition to point bellow
Incoming request rate, Average request duration (non-error), Error response rate (PUT, GET, DELETE, POST, HEAD) on a bucket and tenant level.

Br,
Adam

[cgrinds]

Added StorageGRID overview dashboard with performance, storage, information lifecycle management, and node panels in 23.02.

These changes are in nightly if you want to give feedback before we release 23.02. You can grab the latest nightly build https://github.com/NetApp/harvest/releases/tag/nightly

[cgrinds]

Investigate connecting StorageGRID and ONTAP where applicable

[cgrinds]

Request from Ed on Discord

2 additions I'd like to see:
1: display graph grid consumption by percentage, not just PB. This would allow a display of multiple graphs on the same dashboard to be more useful and make it more obvious just how full the grid is.

2: graph by storage pool. Although we have only 1 storage pool per grid in use here, I expect that some customers will have multiple pools.

[cgrinds]

Release 23.05 includes a new fabric pool dashboard.
Leaving open to collect more ideas for new StorageGRID dashboards.

[cgrinds]

Investigate adding the Audit Overview dashboard

[cgrinds]

ReBanana on Discord suggested the following:

It would be great to have more dashboards for StorageGrid, including hardware metrics and NIC port utilization. This would help identify which ports are heavily used and how their usage changes over time.

This information will assist in planning load balancer extensions and determining whether we need to switch to 100Gb ports instead of 40Gb.

[cgrinds]

List of StorageGRID Prometheus metrics from version 11.7.0

"storagegrid_auto_compactor_cleanup_executor_running",
"storagegrid_auto_compactor_droppable_tombstones_percentage",
"storagegrid_auto_compactor_duplicate_data",
"storagegrid_auto_compactor_effective_shadow_percentage",
"storagegrid_auto_compactor_error",
"storagegrid_auto_compactor_estimated_reclaimed_bytes",
"storagegrid_auto_compactor_execution_error",
"storagegrid_auto_compactor_executor_running",
"storagegrid_auto_compactor_headroom_compaction",
"storagegrid_auto_compactor_headroom_cpu",
"storagegrid_auto_compactor_headroom_disk",
"storagegrid_auto_compactor_headroom_memory",
"storagegrid_auto_compactor_headroom_uptime",
"storagegrid_auto_compactor_metrics_gather_time",
"storagegrid_auto_compactor_reclaimable_shadow_bytes",
"storagegrid_auto_compactor_reclaimed_bytes",
"storagegrid_auto_compactor_safe_to_run",
"storagegrid_auto_compactor_tombstones_as_percentage_of_rows",
"storagegrid_auto_compactor_wants_to_run",
"storagegrid_cassandra_metrics_gather_time",
"storagegrid_cassandra_repair_effective_repair_percentage",
"storagegrid_cassandra_repair_last_run_start_timestamp",
"storagegrid_cassandra_repair_last_run_state",
"storagegrid_cassandra_repair_last_successful_run_duration_seconds",
"storagegrid_cassandra_repair_last_successful_run_start_timestamp",
"storagegrid_cassandra_repair_mean_segment_duration_seconds",
"storagegrid_cassandra_repair_metrics_gathered_timestamp_seconds",
"storagegrid_cassandra_repair_p90_segment_duration_seconds",
"storagegrid_cassandra_repair_reaper_auto_job_abort_initiated",
"storagegrid_cassandra_repair_reaper_auto_job_force_run_initiated",
"storagegrid_cassandra_repair_reaper_auto_restart_initiated",
"storagegrid_cassandra_repair_reaper_reachable",
"storagegrid_cassandra_repair_repaired_percent",
"storagegrid_cassandra_repair_segment_state",
"storagegrid_cassandra_repair_successful_recent_segments",
"storagegrid_chunk_private_chunk_get_vcs_chunk_request_latency_time_ms_bucket",
"storagegrid_chunk_private_chunk_get_vcs_chunk_request_latency_time_ms_count",
"storagegrid_chunk_private_chunk_get_vcs_chunk_request_latency_time_ms_sum",
"storagegrid_chunk_private_chunk_get_vcs_chunk_requests",
"storagegrid_chunk_private_chunk_put_vcs_chunk_request_latency_time_ms_bucket",
"storagegrid_chunk_private_chunk_put_vcs_chunk_request_latency_time_ms_count",
"storagegrid_chunk_private_chunk_put_vcs_chunk_request_latency_time_ms_sum",
"storagegrid_chunk_private_chunk_put_vcs_chunk_requests",
"storagegrid_chunk_private_post_multi_delete_chunk_request_latency_time_ms_bucket",
"storagegrid_chunk_private_post_multi_delete_chunk_request_latency_time_ms_count",
"storagegrid_chunk_private_post_multi_delete_chunk_request_latency_time_ms_sum",
"storagegrid_chunk_private_post_multi_delete_chunk_requests",
"storagegrid_chunk_private_request_queue_size",
"storagegrid_content_buckets_and_containers",
"storagegrid_content_objects",
"storagegrid_content_objects_lost",
"storagegrid_data_mover_private_acct_location_update_time_bucket",
"storagegrid_data_mover_private_acct_location_update_time_count",
"storagegrid_data_mover_private_acct_location_update_time_sum",
"storagegrid_data_mover_private_cloud_storage_pool_head_request_latency_time_ms_bucket",
"storagegrid_data_mover_private_cloud_storage_pool_head_request_latency_time_ms_count",
"storagegrid_data_mover_private_cloud_storage_pool_head_request_latency_time_ms_sum",
"storagegrid_data_mover_private_cloud_storage_pool_head_requests",
"storagegrid_data_mover_private_cloud_storage_pool_request_time_in_queue_in_ms_bucket",
"storagegrid_data_mover_private_cloud_storage_pool_request_time_in_queue_in_ms_count",
"storagegrid_data_mover_private_cloud_storage_pool_request_time_in_queue_in_ms_sum",
"storagegrid_data_mover_private_endpoint_list_request_latency_bucket",
"storagegrid_data_mover_private_endpoint_list_request_latency_count",
"storagegrid_data_mover_private_endpoint_list_request_latency_sum",
"storagegrid_data_mover_private_failed_acct_location_updates",
"storagegrid_data_mover_private_failed_cloud_storage_pool_head_requests",
"storagegrid_data_mover_private_failed_endpoint_list_requests",
"storagegrid_data_mover_private_obtain_endpoint_client_time_ms_bucket",
"storagegrid_data_mover_private_obtain_endpoint_client_time_ms_count",
"storagegrid_data_mover_private_obtain_endpoint_client_time_ms_sum",
"storagegrid_data_mover_private_request_queue_size",
"storagegrid_data_mover_private_total_acct_location_updates",
"storagegrid_data_mover_private_total_cloud_storage_pool_connection_errors",
"storagegrid_data_mover_private_total_endpoint_list_requests",
"storagegrid_disk_io_read_rate",
"storagegrid_disk_io_time",
"storagegrid_disk_io_write_rate",
"storagegrid_dynip_update_dispatch_count",
"storagegrid_dynip_update_dispatch_queue_length",
"storagegrid_dynip_update_dispatch_queue_seconds_count",
"storagegrid_dynip_update_dispatch_queue_seconds_created",
"storagegrid_dynip_update_dispatch_queue_seconds_sum",
"storagegrid_dynip_update_dispatch_seconds_count",
"storagegrid_dynip_update_dispatch_seconds_created",
"storagegrid_dynip_update_dispatch_seconds_sum",
"storagegrid_dynip_update_request_processing_seconds_count",
"storagegrid_dynip_update_request_processing_seconds_created",
"storagegrid_dynip_update_request_processing_seconds_sum",
"storagegrid_ec_jobmanager_job_states",
"storagegrid_fgv_jobmanager_job_states",
"storagegrid_fips_mode",
"storagegrid_fips_mode_updated_timestamp",
"storagegrid_firewall_configuration_timestamp",
"storagegrid_go_failed_remote_config_requests",
"storagegrid_go_remote_config_time_ms_bucket",
"storagegrid_go_remote_config_time_ms_count",
"storagegrid_go_remote_config_time_ms_sum",
"storagegrid_go_total_remote_config_requests",
"storagegrid_http_sessions_incoming_attempted",
"storagegrid_http_sessions_incoming_currently_established",
"storagegrid_http_sessions_incoming_successful",
"storagegrid_identity_service_authorize_by_uuid_time_ms_bucket",
"storagegrid_identity_service_authorize_by_uuid_time_ms_count",
"storagegrid_identity_service_authorize_by_uuid_time_ms_sum",
"storagegrid_identity_service_authorize_time_ms_bucket",
"storagegrid_identity_service_authorize_time_ms_count",
"storagegrid_identity_service_authorize_time_ms_sum",
"storagegrid_identity_service_change_password_time_ms_bucket",
"storagegrid_identity_service_change_password_time_ms_count",
"storagegrid_identity_service_change_password_time_ms_sum",
"storagegrid_identity_service_close_connection_time_ms_bucket",
"storagegrid_identity_service_close_connection_time_ms_count",
"storagegrid_identity_service_close_connection_time_ms_sum",
"storagegrid_identity_service_failed_authorize_by_uuid_requests",
"storagegrid_identity_service_failed_authorize_requests",
"storagegrid_identity_service_failed_change_password_requests",
"storagegrid_identity_service_failed_get_group_requests",
"storagegrid_identity_service_failed_list_groups_requests",
"storagegrid_identity_service_failed_new_connection_dials",
"storagegrid_identity_service_failed_new_connections",
"storagegrid_identity_service_failed_schedule_synchronization_requests",
"storagegrid_identity_service_failed_search_group",
"storagegrid_identity_service_failed_search_groups",
"storagegrid_identity_service_failed_search_user",
"storagegrid_identity_service_failed_search_users",
"storagegrid_identity_service_failed_synchronization_scans",
"storagegrid_identity_service_failed_validate_requests",
"storagegrid_identity_service_get_group_time_ms_bucket",
"storagegrid_identity_service_get_group_time_ms_count",
"storagegrid_identity_service_get_group_time_ms_sum",
"storagegrid_identity_service_list_groups_time_ms_bucket",
"storagegrid_identity_service_list_groups_time_ms_count",
"storagegrid_identity_service_list_groups_time_ms_sum",
"storagegrid_identity_service_new_connection_time_ms_bucket",
"storagegrid_identity_service_new_connection_time_ms_count",
"storagegrid_identity_service_new_connection_time_ms_sum",
"storagegrid_identity_service_schedule_synchronization_time_ms_bucket",
"storagegrid_identity_service_schedule_synchronization_time_ms_count",
"storagegrid_identity_service_schedule_synchronization_time_ms_sum",
"storagegrid_identity_service_search_group_time_ms_bucket",
"storagegrid_identity_service_search_group_time_ms_count",
"storagegrid_identity_service_search_group_time_ms_sum",
"storagegrid_identity_service_search_groups_time_ms_bucket",
"storagegrid_identity_service_search_groups_time_ms_count",
"storagegrid_identity_service_search_groups_time_ms_sum",
"storagegrid_identity_service_search_user_time_ms_bucket",
"storagegrid_identity_service_search_user_time_ms_count",
"storagegrid_identity_service_search_user_time_ms_sum",
"storagegrid_identity_service_search_users_time_ms_bucket",
"storagegrid_identity_service_search_users_time_ms_count",
"storagegrid_identity_service_search_users_time_ms_sum",
"storagegrid_identity_service_synchronization_scan_time_ms_bucket",
"storagegrid_identity_service_synchronization_scan_time_ms_count",
"storagegrid_identity_service_synchronization_scan_time_ms_sum",
"storagegrid_identity_service_tenant_synchronization_time_ms_bucket",
"storagegrid_identity_service_tenant_synchronization_time_ms_count",
"storagegrid_identity_service_tenant_synchronization_time_ms_sum",
"storagegrid_identity_service_total_authorize_by_uuid_requests",
"storagegrid_identity_service_total_authorize_requests",
"storagegrid_identity_service_total_change_password_requests",
"storagegrid_identity_service_total_closed_connections",
"storagegrid_identity_service_total_get_group_requests",
"storagegrid_identity_service_total_list_groups_requests",
"storagegrid_identity_service_total_new_connections",
"storagegrid_identity_service_total_schedule_synchronization_requests",
"storagegrid_identity_service_total_search_group",
"storagegrid_identity_service_total_search_groups",
"storagegrid_identity_service_total_search_user",
"storagegrid_identity_service_total_search_users",
"storagegrid_identity_service_total_synchronization_scans",
"storagegrid_identity_service_total_tenant_synchronizations",
"storagegrid_identity_service_total_validate_requests",
"storagegrid_identity_service_validate_time_ms_bucket",
"storagegrid_identity_service_validate_time_ms_count",
"storagegrid_identity_service_validate_time_ms_sum",
"storagegrid_ilm_awaiting_background_objects",
"storagegrid_ilm_awaiting_client_evaluation_objects_per_second",
"storagegrid_ilm_awaiting_client_objects",
"storagegrid_ilm_awaiting_total_objects",
"storagegrid_ilm_objects_processed",
"storagegrid_ilm_scan_objects_per_second",
"storagegrid_ilm_scan_period_estimated_minutes",
"storagegrid_ilm_total_unachievable_placements",
"storagegrid_ilm_total_unachievable_purges",
"storagegrid_java_offheap_memory",
"storagegrid_load_balancer_endpoint_bound",
"storagegrid_load_balancer_endpoint_cert_chain_expiry_time",
"storagegrid_load_balancer_endpoint_cert_expiry_time",
"storagegrid_metadata_per_object_bytes",
"storagegrid_metadata_queries_average_latency_milliseconds",
"storagegrid_metadata_total_object_capacity_count",
"storagegrid_network_received_bytes",
"storagegrid_network_transmitted_bytes",
"storagegrid_node_cpu_utilization_percentage",
"storagegrid_ntp_chosen_time_source_offset_milliseconds",
"storagegrid_ntp_locked",
"storagegrid_ntp_time_sync_local_clock_offset_milliseconds",
"storagegrid_ntp_time_sync_local_clock_rate_slew",
"storagegrid_platform_services_account_status_latency_ms_bucket",
"storagegrid_platform_services_account_status_latency_ms_count",
"storagegrid_platform_services_account_status_latency_ms_sum",
"storagegrid_platform_services_active_accounts",
"storagegrid_platform_services_active_tenant_cleanup_time_ms_bucket",
"storagegrid_platform_services_active_tenant_cleanup_time_ms_count",
"storagegrid_platform_services_active_tenant_cleanup_time_ms_sum",
"storagegrid_platform_services_committed_processing_time_ms_bucket",
"storagegrid_platform_services_committed_processing_time_ms_count",
"storagegrid_platform_services_committed_processing_time_ms_sum",
"storagegrid_platform_services_committer_time_ms_bucket",
"storagegrid_platform_services_committer_time_ms_count",
"storagegrid_platform_services_committer_time_ms_sum",
"storagegrid_platform_services_dropped_metadata_notifications",
"storagegrid_platform_services_dropped_replications",
"storagegrid_platform_services_dropped_requests",
"storagegrid_platform_services_failed_account_status_requests",
"storagegrid_platform_services_failed_get_config_requests",
"storagegrid_platform_services_failed_metadata_notifications",
"storagegrid_platform_services_failed_persist_requests",
"storagegrid_platform_services_failed_populate_requests",
"storagegrid_platform_services_failed_post_config_requests",
"storagegrid_platform_services_failed_raft_requests",
"storagegrid_platform_services_failed_replications",
"storagegrid_platform_services_failed_s3_notifications",
"storagegrid_platform_services_failed_topology_persist_requests",
"storagegrid_platform_services_failed_validate_requests",
"storagegrid_platform_services_filled_accounts",
"storagegrid_platform_services_get_config_time_ms_bucket",
"storagegrid_platform_services_get_config_time_ms_count",
"storagegrid_platform_services_get_config_time_ms_sum",
"storagegrid_platform_services_leader_changes",
"storagegrid_platform_services_metadata_notification_service_time_ms_bucket",
"storagegrid_platform_services_metadata_notification_service_time_ms_count",
"storagegrid_platform_services_metadata_notification_service_time_ms_sum",
"storagegrid_platform_services_node_ready_bucket",
"storagegrid_platform_services_node_ready_count",
"storagegrid_platform_services_node_ready_sum",
"storagegrid_platform_services_obtain_endpoint_client_time_ms_bucket",
"storagegrid_platform_services_obtain_endpoint_client_time_ms_count",
"storagegrid_platform_services_obtain_endpoint_client_time_ms_sum",
"storagegrid_platform_services_pending_insert_time_ms_bucket",
"storagegrid_platform_services_pending_insert_time_ms_count",
"storagegrid_platform_services_pending_insert_time_ms_sum",
"storagegrid_platform_services_pending_persisted_requests",
"storagegrid_platform_services_pending_remove_time_ms_bucket",
"storagegrid_platform_services_pending_remove_time_ms_count",
"storagegrid_platform_services_pending_remove_time_ms_sum",
"storagegrid_platform_services_permanently_failed_requests",
"storagegrid_platform_services_persist_request_timeouts",
"storagegrid_platform_services_populate_time_ms_bucket",
"storagegrid_platform_services_populate_time_ms_count",
"storagegrid_platform_services_populate_time_ms_sum",
"storagegrid_platform_services_post_config_time_ms_bucket",
"storagegrid_platform_services_post_config_time_ms_count",
"storagegrid_platform_services_post_config_time_ms_sum",
"storagegrid_platform_services_raft_advance_time_ms_bucket",
"storagegrid_platform_services_raft_advance_time_ms_count",
"storagegrid_platform_services_raft_advance_time_ms_sum",
"storagegrid_platform_services_raft_batch_size_bucket",
"storagegrid_platform_services_raft_batch_size_count",
"storagegrid_platform_services_raft_batch_size_sum",
"storagegrid_platform_services_raft_latency_ms_bucket",
"storagegrid_platform_services_raft_latency_ms_count",
"storagegrid_platform_services_raft_latency_ms_sum",
"storagegrid_platform_services_raft_post_time_ms_bucket",
"storagegrid_platform_services_raft_post_time_ms_count",
"storagegrid_platform_services_raft_post_time_ms_sum",
"storagegrid_platform_services_raft_send_time_ms_bucket",
"storagegrid_platform_services_raft_send_time_ms_count",
"storagegrid_platform_services_raft_send_time_ms_sum",
"storagegrid_platform_services_raft_state_bucket",
"storagegrid_platform_services_raft_state_count",
"storagegrid_platform_services_raft_state_sum",
"storagegrid_platform_services_replication_persist_latency_ms_bucket",
"storagegrid_platform_services_replication_persist_latency_ms_count",
"storagegrid_platform_services_replication_persist_latency_ms_sum",
"storagegrid_platform_services_replication_service_time_ms_bucket",
"storagegrid_platform_services_replication_service_time_ms_count",
"storagegrid_platform_services_replication_service_time_ms_sum",
"storagegrid_platform_services_retry_scan_time_ms_bucket",
"storagegrid_platform_services_retry_scan_time_ms_count",
"storagegrid_platform_services_retry_scan_time_ms_sum",
"storagegrid_platform_services_rsm_pending_encode_time_ms_bucket",
"storagegrid_platform_services_rsm_pending_encode_time_ms_count",
"storagegrid_platform_services_rsm_pending_encode_time_ms_sum",
"storagegrid_platform_services_s3_notification_service_time_ms_bucket",
"storagegrid_platform_services_s3_notification_service_time_ms_count",
"storagegrid_platform_services_s3_notification_service_time_ms_sum",
"storagegrid_platform_services_snapshot_clone_time_ms_bucket",
"storagegrid_platform_services_snapshot_clone_time_ms_count",
"storagegrid_platform_services_snapshot_clone_time_ms_sum",
"storagegrid_platform_services_snapshot_restore_time_ms_bucket",
"storagegrid_platform_services_snapshot_restore_time_ms_count",
"storagegrid_platform_services_snapshot_restore_time_ms_sum",
"storagegrid_platform_services_topology_persist_latency_ms_bucket",
"storagegrid_platform_services_topology_persist_latency_ms_count",
"storagegrid_platform_services_topology_persist_latency_ms_sum",
"storagegrid_platform_services_total_account_status_requests",
"storagegrid_platform_services_total_get_config_requests",
"storagegrid_platform_services_total_metadata_notifications",
"storagegrid_platform_services_total_persist_requests",
"storagegrid_platform_services_total_populate_requests",
"storagegrid_platform_services_total_post_config_requests",
"storagegrid_platform_services_total_raft_requests",
"storagegrid_platform_services_total_replications",
"storagegrid_platform_services_total_s3_notifications",
"storagegrid_platform_services_total_topology_persist_requests",
"storagegrid_platform_services_total_validate_requests",
"storagegrid_platform_services_unfilled_accounts",
"storagegrid_platform_services_update_throttling_time_ms_bucket",
"storagegrid_platform_services_update_throttling_time_ms_count",
"storagegrid_platform_services_update_throttling_time_ms_sum",
"storagegrid_platform_services_validate_time_ms_bucket",
"storagegrid_platform_services_validate_time_ms_count",
"storagegrid_platform_services_validate_time_ms_sum",
"storagegrid_private_account_service_failed_requests",
"storagegrid_private_account_service_request_time_ms_bucket",
"storagegrid_private_account_service_request_time_ms_count",
"storagegrid_private_account_service_request_time_ms_sum",
"storagegrid_private_account_service_total_requests",
"storagegrid_private_ade_http_client_processing_time_ms_bucket",
"storagegrid_private_ade_http_client_processing_time_ms_count",
"storagegrid_private_ade_http_client_processing_time_ms_sum",
"storagegrid_private_ade_pending_messages",
"storagegrid_private_ade_processes",
"storagegrid_private_ade_received_messages_total",
"storagegrid_private_ade_sent_messages_total",
"storagegrid_private_audit_commit_messages_retried",
"storagegrid_private_audit_export_records_committed",
"storagegrid_private_audit_messages_committed",
"storagegrid_private_audit_messages_generated",
"storagegrid_private_audit_messages_queued",
"storagegrid_private_audit_messages_received",
"storagegrid_private_cassandra_commitlog_corruption_timestamp",
"storagegrid_private_cassandra_commitlog_performance_average_commit_duration_ms",
"storagegrid_private_cassandra_commitlog_performance_average_duration_over_target_ms",
"storagegrid_private_cassandra_commitlog_performance_number_over_target",
"storagegrid_private_cassandra_commitlog_performance_total_commits",
"storagegrid_private_cassandra_connection_timeouts",
"storagegrid_private_cassandra_delete_consistency_unavailable_total",
"storagegrid_private_cassandra_delete_request_time_ms_bucket",
"storagegrid_private_cassandra_delete_request_time_ms_count",
"storagegrid_private_cassandra_delete_request_time_ms_sum",
"storagegrid_private_cassandra_fifteen_minute_request_rate",
"storagegrid_private_cassandra_five_minute_request_rate",
"storagegrid_private_cassandra_ingest_consistency_unavailable_total",
"storagegrid_private_cassandra_ingest_time_ms_bucket",
"storagegrid_private_cassandra_ingest_time_ms_count",
"storagegrid_private_cassandra_ingest_time_ms_sum",
"storagegrid_private_cassandra_list_objects_prefix_count",
"storagegrid_private_cassandra_list_objects_prefix_cycles",
"storagegrid_private_cassandra_list_objects_result_common_prefix_count",
"storagegrid_private_cassandra_list_objects_result_key_count",
"storagegrid_private_cassandra_mean_request_rate",
"storagegrid_private_cassandra_one_minute_request_rate",
"storagegrid_private_cassandra_pending_connection_request_timeouts",
"storagegrid_private_cassandra_prefix_cache_insert_new",
"storagegrid_private_cassandra_prefix_cache_size",
"storagegrid_private_cassandra_received_merkle_tree",
"storagegrid_private_cassandra_repair_session_duration_ms",
"storagegrid_private_cassandra_repair_session_finished",
"storagegrid_private_cassandra_repair_session_started",
"storagegrid_private_cassandra_request_latency_max",
"storagegrid_private_cassandra_request_latency_mean",
"storagegrid_private_cassandra_request_latency_median",
"storagegrid_private_cassandra_request_latency_min",
"storagegrid_private_cassandra_request_latency_percentile_75th",
"storagegrid_private_cassandra_request_latency_percentile_95th",
"storagegrid_private_cassandra_request_latency_percentile_98th",
"storagegrid_private_cassandra_request_latency_percentile_999th",
"storagegrid_private_cassandra_request_latency_percentile_99th",
"storagegrid_private_cassandra_request_latency_stddev",
"storagegrid_private_cassandra_request_time_ms",
"storagegrid_private_cassandra_request_time_ms_count",
"storagegrid_private_cassandra_request_time_ms_sum",
"storagegrid_private_cassandra_request_timeouts",
"storagegrid_private_cassandra_requesting_merkle_trees",
"storagegrid_private_cassandra_sending_merkle_tree",
"storagegrid_private_cassandra_tombstone_warning_live_cells",
"storagegrid_private_cassandra_tombstone_warning_tombstones",
"storagegrid_private_cassandra_total_connections",
"storagegrid_private_cgrscanner_buffer_external_capacity",
"storagegrid_private_cgrscanner_buffer_external_elements",
"storagegrid_private_cgrscanner_buffer_scanner_capacity",
"storagegrid_private_cgrscanner_buffer_scanner_elements",
"storagegrid_private_cgrscanner_emit_completed_requests",
"storagegrid_private_cgrscanner_emit_in_progress_requests",
"storagegrid_private_cgrscanner_emit_processes",
"storagegrid_private_client_connections",
"storagegrid_private_config_generation_state_timestamp",
"storagegrid_private_core_timestamp",
"storagegrid_private_cpu_usage_seconds",
"storagegrid_private_direct_packet_source_read_size_bytes_bucket",
"storagegrid_private_direct_packet_source_read_size_bytes_count",
"storagegrid_private_direct_packet_source_read_size_bytes_sum",
"storagegrid_private_direct_packet_source_read_time_ms_bucket",
"storagegrid_private_direct_packet_source_read_time_ms_count",
"storagegrid_private_direct_packet_source_read_time_ms_sum",
"storagegrid_private_direct_packet_source_total_process_time_ms_bucket",
"storagegrid_private_direct_packet_source_total_process_time_ms_count",
"storagegrid_private_direct_packet_source_total_process_time_ms_sum",
"storagegrid_private_disk_io_usage_tracking_service_rchar",
"storagegrid_private_disk_io_usage_tracking_service_read_bytes",
"storagegrid_private_disk_io_usage_tracking_service_syscr",
"storagegrid_private_disk_io_usage_tracking_service_syscw",
"storagegrid_private_disk_io_usage_tracking_service_wchar",
"storagegrid_private_disk_io_usage_tracking_service_write_bytes",
"storagegrid_private_disk_io_usage_tracking_subservice_rate_rchar",
"storagegrid_private_disk_io_usage_tracking_subservice_rate_read_bytes",
"storagegrid_private_disk_io_usage_tracking_subservice_rate_syscr",
"storagegrid_private_disk_io_usage_tracking_subservice_rate_syscw",
"storagegrid_private_disk_io_usage_tracking_subservice_rate_wchar",
"storagegrid_private_disk_io_usage_tracking_subservice_rate_write_bytes",
"storagegrid_private_disk_queue_acknowledge_time_ms_bucket",
"storagegrid_private_disk_queue_acknowledge_time_ms_count",
"storagegrid_private_disk_queue_acknowledge_time_ms_sum",
"storagegrid_private_disk_queue_dequeue_time_ms_bucket",
"storagegrid_private_disk_queue_dequeue_time_ms_count",
"storagegrid_private_disk_queue_dequeue_time_ms_sum",
"storagegrid_private_disk_queue_enqueue_time_ms_bucket",
"storagegrid_private_disk_queue_enqueue_time_ms_count",
"storagegrid_private_disk_queue_enqueue_time_ms_sum",
"storagegrid_private_ec_chunksource_processing_time_ms_bucket",
"storagegrid_private_ec_chunksource_processing_time_ms_count",
"storagegrid_private_ec_chunksource_processing_time_ms_sum",
"storagegrid_private_ec_deletes",
"storagegrid_private_ec_groupmonitor_ecgroup_count",
"storagegrid_private_ec_packetSource_processing_time_ms_bucket",
"storagegrid_private_ec_packetSource_processing_time_ms_count",
"storagegrid_private_ec_packetSource_processing_time_ms_sum",
"storagegrid_private_ec_packetsource_buffer_pool_allocation_bytes",
"storagegrid_private_ec_packetsource_invocations",
"storagegrid_private_ec_packetsource_read_bytes",
"storagegrid_private_ec_pipeline_async_read_bytes_bucket",
"storagegrid_private_ec_pipeline_async_read_bytes_count",
"storagegrid_private_ec_pipeline_async_read_bytes_sum",
"storagegrid_private_ec_pipeline_processing_time_ms_bucket",
"storagegrid_private_ec_pipeline_processing_time_ms_count",
"storagegrid_private_ec_pipeline_processing_time_ms_sum",
"storagegrid_private_ec_stripesource_processing_time_ms_bucket",
"storagegrid_private_ec_stripesource_processing_time_ms_count",
"storagegrid_private_ec_stripesource_processing_time_ms_sum",
"storagegrid_private_ec_writes",
"storagegrid_private_event_time_ms_bucket",
"storagegrid_private_event_time_ms_count",
"storagegrid_private_event_time_ms_sum",
"storagegrid_private_grid_management_account_usage_total_duration_ms_bucket",
"storagegrid_private_grid_management_account_usage_total_duration_ms_count",
"storagegrid_private_grid_management_account_usage_total_duration_ms_sum",
"storagegrid_private_grid_management_account_usage_total_request_total",
"storagegrid_private_identity_service_failed_get_config_requests",
"storagegrid_private_identity_service_failed_post_config_requests",
"storagegrid_private_identity_service_get_config_time_ms_bucket",
"storagegrid_private_identity_service_get_config_time_ms_count",
"storagegrid_private_identity_service_get_config_time_ms_sum",
"storagegrid_private_identity_service_post_config_time_ms_bucket",
"storagegrid_private_identity_service_post_config_time_ms_count",
"storagegrid_private_identity_service_post_config_time_ms_sum",
"storagegrid_private_identity_service_total_get_config_requests",
"storagegrid_private_identity_service_total_post_config_requests",
"storagegrid_private_ilm_active_location_removals",
"storagegrid_private_ilm_active_location_replications",
"storagegrid_private_ilm_active_metadata_purges",
"storagegrid_private_ilm_awaiting_delete_objects",
"storagegrid_private_ilm_cloc_remove_time_ms_bucket",
"storagegrid_private_ilm_cloc_remove_time_ms_count",
"storagegrid_private_ilm_cloc_remove_time_ms_sum",
"storagegrid_private_ilm_configured_leak_cleanup_ratio",
"storagegrid_private_ilm_configured_scan_rate_objects_per_second",
"storagegrid_private_ilm_delete_cassandra_metadata_time_ms_bucket",
"storagegrid_private_ilm_delete_cassandra_metadata_time_ms_count",
"storagegrid_private_ilm_delete_cassandra_metadata_time_ms_sum",
"storagegrid_private_ilm_deleted_object_time_ms_bucket",
"storagegrid_private_ilm_deleted_object_time_ms_count",
"storagegrid_private_ilm_deleted_object_time_ms_sum",
"storagegrid_private_ilm_evaluation_time_ms_bucket",
"storagegrid_private_ilm_evaluation_time_ms_count",
"storagegrid_private_ilm_evaluation_time_ms_sum",
"storagegrid_private_ilm_ilm_driven_delete_time_ms_bucket",
"storagegrid_private_ilm_ilm_driven_delete_time_ms_count",
"storagegrid_private_ilm_ilm_driven_delete_time_ms_sum",
"storagegrid_private_ilm_leaked_uuid_cleanup_total_bytes",
"storagegrid_private_ilm_leaked_uuid_cleanup_total_count",
"storagegrid_private_ilm_nameless_delete_time_ms_bucket",
"storagegrid_private_ilm_nameless_delete_time_ms_count",
"storagegrid_private_ilm_nameless_delete_time_ms_sum",
"storagegrid_private_ilm_objects_retrieved",
"storagegrid_private_ilm_retrieval_time_ms_bucket",
"storagegrid_private_ilm_retrieval_time_ms_count",
"storagegrid_private_ilm_retrieval_time_ms_sum",
"storagegrid_private_ilm_service_query_time_ms_bucket",
"storagegrid_private_ilm_service_query_time_ms_count",
"storagegrid_private_ilm_service_query_time_ms_sum",
"storagegrid_private_ilm_total_async_client_deletes",
"storagegrid_private_ilm_total_deletes",
"storagegrid_private_ilm_total_ilm_driven_deletes",
"storagegrid_private_ilm_total_sync_client_deletes",
"storagegrid_private_ilm_unused_metadata",
"storagegrid_private_improved_availability_enabled",
"storagegrid_private_ingest_ec_writer_buffer_pool_allocation_bytes",
"storagegrid_private_ingest_ec_writer_buffer_pool_allocation_time_milliseconds_bucket",
"storagegrid_private_ingest_ec_writer_buffer_pool_allocation_time_milliseconds_count",
"storagegrid_private_ingest_ec_writer_buffer_pool_allocation_time_milliseconds_sum",
"storagegrid_private_ingest_ec_writer_buffer_pool_request_bytes_total",
"storagegrid_private_ingest_ec_writer_buffer_pool_response_bytes_total",
"storagegrid_private_ingest_failed_retains_total",
"storagegrid_private_ingest_obtain_rrtc_microseconds",
"storagegrid_private_ingest_obtain_rrtc_microseconds_count",
"storagegrid_private_ingest_obtain_rrtc_microseconds_sum",
"storagegrid_private_ingest_retained_total",
"storagegrid_private_load_balancer_storage_early_reset_upstream_nodes",
"storagegrid_private_load_balancer_storage_internal_request_accept_count",
"storagegrid_private_load_balancer_storage_internal_request_all_upstream_nodes_failed",
"storagegrid_private_load_balancer_storage_internal_request_count",
"storagegrid_private_load_balancer_storage_internal_request_failed_count",
"storagegrid_private_load_balancer_storage_internal_request_retry_count",
"storagegrid_private_load_balancer_storage_internal_request_rx_bytes",
"storagegrid_private_load_balancer_storage_internal_request_time",
"storagegrid_private_load_balancer_storage_internal_request_too_many_upstream_retries",
"storagegrid_private_load_balancer_storage_internal_request_tx_bytes",
"storagegrid_private_load_balancer_storage_internal_request_upstream_time",
"storagegrid_private_load_balancer_storage_internal_upstream_error",
"storagegrid_private_load_balancer_storage_internal_upstream_marked_offline",
"storagegrid_private_load_balancer_storage_request_accept_count",
"storagegrid_private_load_balancer_storage_request_all_upstream_nodes_failed",
"storagegrid_private_load_balancer_storage_request_body_bytes_bucket",
"storagegrid_private_load_balancer_storage_request_body_bytes_count",
"storagegrid_private_load_balancer_storage_request_body_bytes_sum",
"storagegrid_private_load_balancer_storage_request_count",
"storagegrid_private_load_balancer_storage_request_delayed_all_upstream_nodes_failed",
"storagegrid_private_load_balancer_storage_request_incoming_count",
"storagegrid_private_load_balancer_storage_request_rejected_count",
"storagegrid_private_load_balancer_storage_request_rejected_time",
"storagegrid_private_load_balancer_storage_request_retry_count",
"storagegrid_private_load_balancer_storage_request_time",
"storagegrid_private_load_balancer_storage_request_too_many_upstream_retries",
"storagegrid_private_load_balancer_storage_request_upstream_time",
"storagegrid_private_load_balancer_storage_rx_bytes",
"storagegrid_private_load_balancer_storage_tenant_manager_s3_console_request_count",
"storagegrid_private_load_balancer_storage_tx_bytes",
"storagegrid_private_load_balancer_storage_upstream_error",
"storagegrid_private_load_balancer_storage_upstream_marked_offline",
"storagegrid_private_load_balancer_upstream_requests_completed",
"storagegrid_private_load_balancer_upstream_requests_started",
"storagegrid_private_location_reader_invocations",
"storagegrid_private_location_reader_processing_time_ms_bucket",
"storagegrid_private_location_reader_processing_time_ms_count",
"storagegrid_private_location_reader_processing_time_ms_sum",
"storagegrid_private_mgmtapi_auth_failure_count",
"storagegrid_private_mgmtapi_auth_success_count",
"storagegrid_private_object_actions_time_ms_bucket",
"storagegrid_private_object_actions_time_ms_count",
"storagegrid_private_object_actions_time_ms_sum",
"storagegrid_private_object_actions_total",
"storagegrid_private_object_ingest_request_detail_time_ms_bucket",
"storagegrid_private_object_ingest_request_detail_time_ms_count",
"storagegrid_private_object_ingest_request_detail_time_ms_sum",
"storagegrid_private_object_ingest_request_time_ms_bucket",
"storagegrid_private_object_ingest_request_time_ms_count",
"storagegrid_private_object_ingest_request_time_ms_sum",
"storagegrid_private_object_ingest_total_requests",
"storagegrid_private_object_retain_request_client_rrtc_time_ms_bucket",
"storagegrid_private_object_retain_request_client_rrtc_time_ms_count",
"storagegrid_private_object_retain_request_client_rrtc_time_ms_sum",
"storagegrid_private_object_retain_request_client_stor_time_ms_bucket",
"storagegrid_private_object_retain_request_client_stor_time_ms_count",
"storagegrid_private_object_retain_request_client_stor_time_ms_sum",
"storagegrid_private_object_retain_request_detail_time_ms_bucket",
"storagegrid_private_object_retain_request_detail_time_ms_count",
"storagegrid_private_object_retain_request_detail_time_ms_sum",
"storagegrid_private_object_retain_request_time_ms_bucket",
"storagegrid_private_object_retain_request_time_ms_count",
"storagegrid_private_object_retain_request_time_ms_sum",
"storagegrid_private_osl_async_request_time_ms_bucket",
"storagegrid_private_osl_async_request_time_ms_count",
"storagegrid_private_osl_async_request_time_ms_sum",
"storagegrid_private_prefixsplitting_active_keys_in_prefix_bucket",
"storagegrid_private_prefixsplitting_active_keys_in_prefix_count",
"storagegrid_private_prefixsplitting_active_keys_in_prefix_sum",
"storagegrid_private_prefixsplitting_candidate_prefixes_scanned",
"storagegrid_private_prefixsplitting_child_prefixes_split_into_bucket",
"storagegrid_private_prefixsplitting_child_prefixes_split_into_count",
"storagegrid_private_prefixsplitting_child_prefixes_split_into_sum",
"storagegrid_private_prefixsplitting_counted_below_threshold_prefixes_scanned",
"storagegrid_private_prefixsplitting_deleted_keys_in_prefix_bucket",
"storagegrid_private_prefixsplitting_deleted_keys_in_prefix_count",
"storagegrid_private_prefixsplitting_deleted_keys_in_prefix_sum",
"storagegrid_private_prefixsplitting_inactive_prefixes_scanned",
"storagegrid_private_prefixsplitting_prefix_count_time_ms_bucket",
"storagegrid_private_prefixsplitting_prefix_count_time_ms_count",
"storagegrid_private_prefixsplitting_prefix_count_time_ms_sum",
"storagegrid_private_prefixsplitting_prefix_enumeration_idle_time_ms",
"storagegrid_private_prefixsplitting_prefix_enumeration_time_ms",
"storagegrid_private_prefixsplitting_prefix_split_time_ms_bucket",
"storagegrid_private_prefixsplitting_prefix_split_time_ms_count",
"storagegrid_private_prefixsplitting_prefix_split_time_ms_sum",
"storagegrid_private_prefixsplitting_resumed_candidate_prefixes_scanned",
"storagegrid_private_prefixsplitting_secondary_candidate_prefixes_scanned",
"storagegrid_private_prefixsplitting_total_prefix_splits",
"storagegrid_private_prefixsplitting_total_prefixes_scanned",
"storagegrid_private_prefixsplitting_uncounted_below_threshold_prefixes_scanned",
"storagegrid_private_process_open_fds",
"storagegrid_private_s3_request_detail_time_ms_bucket",
"storagegrid_private_s3_request_detail_time_ms_count",
"storagegrid_private_s3_request_detail_time_ms_sum",
"storagegrid_private_s3_request_time_ms_bucket",
"storagegrid_private_s3_request_time_ms_count",
"storagegrid_private_s3_request_time_ms_sum",
"storagegrid_private_s3_total_requests",
"storagegrid_private_serialization_active_proxy_delete_object_requests",
"storagegrid_private_serialization_delete_object_client_duration_seconds",
"storagegrid_private_serialization_delete_object_client_duration_seconds_count",
"storagegrid_private_serialization_delete_object_client_duration_seconds_sum",
"storagegrid_private_serialization_delete_object_failure_total",
"storagegrid_private_serialization_delete_object_proxy_duration_seconds",
"storagegrid_private_serialization_delete_object_proxy_duration_seconds_count",
"storagegrid_private_serialization_delete_object_proxy_duration_seconds_sum",
"storagegrid_private_serialization_delete_object_total",
"storagegrid_private_serialization_mapping_update_committed_total",
"storagegrid_private_serialization_mapping_update_proposals_accepted_total",
"storagegrid_private_serialization_mapping_update_proposals_committed_total",
"storagegrid_private_serialization_mapping_update_proposals_total",
"storagegrid_private_serialization_mapping_update_proposed_total",
"storagegrid_private_service_uptime_seconds",
"storagegrid_private_sstable_data_age_in_seconds",
"storagegrid_private_sstable_data_age_in_seconds_mean",
"storagegrid_private_sstable_data_age_in_seconds_median",
"storagegrid_private_sstable_data_percentage_younger_than",
"storagegrid_private_sstable_data_row_size",
"storagegrid_private_storage_active_cloc_removes",
"storagegrid_private_storage_verifier_status",
"storagegrid_private_storage_writer_gauge",
"storagegrid_private_storage_writer_time_ms_bucket",
"storagegrid_private_storage_writer_time_ms_count",
"storagegrid_private_storage_writer_time_ms_sum",
"storagegrid_private_traces_disabled",
"storagegrid_private_traces_sampled",
"storagegrid_private_unexpected_reboot_timestamp",
"storagegrid_private_usage_sum_up_failures",
"storagegrid_private_usage_total_up_failures",
"storagegrid_release_version",
"storagegrid_s3_data_transfers_bytes_ingested",
"storagegrid_s3_data_transfers_bytes_retrieved",
"storagegrid_s3_operations_failed",
"storagegrid_s3_operations_successful",
"storagegrid_s3_operations_unauthorized",
"storagegrid_s3_requests_cancelled_time_ms_bucket",
"storagegrid_s3_requests_cancelled_time_ms_count",
"storagegrid_s3_requests_cancelled_time_ms_sum",
"storagegrid_s3_requests_cancelled_total",
"storagegrid_s3_requests_time_ms_bucket",
"storagegrid_s3_requests_time_ms_count",
"storagegrid_s3_requests_time_ms_sum",
"storagegrid_s3_requests_total",
"storagegrid_s3_select_bytes_processed_total",
"storagegrid_s3_select_bytes_returned_total",
"storagegrid_s3_select_bytes_scanned_total",
"storagegrid_s3_select_executor_limit",
"storagegrid_s3_select_executors_running",
"storagegrid_s3_select_jobs_cancelled_total",
"storagegrid_s3_select_jobs_errored_total",
"storagegrid_s3_select_jobs_queued",
"storagegrid_s3_select_jobs_running",
"storagegrid_s3_select_jobs_started_total",
"storagegrid_s3_select_jobs_successful_total",
"storagegrid_s3_select_records_processed_total",
"storagegrid_s3_select_records_returned_total",
"storagegrid_s3_select_runtime_seconds_total",
"storagegrid_servercertificate_management_interface_cert_expiry_days",
"storagegrid_servercertificate_storage_api_endpoints_cert_expiry_days",
"storagegrid_service_cpu_seconds",
"storagegrid_service_load",
"storagegrid_service_memory_usage_bytes",
"storagegrid_service_network_received_bytes",
"storagegrid_service_network_transmitted_bytes",
"storagegrid_service_restarts",
"storagegrid_service_runtime_seconds",
"storagegrid_service_uptime_seconds",
"storagegrid_storage_pool_utilization_data_bytes",
"storagegrid_storage_pool_utilization_usable_space_bytes",
"storagegrid_storage_state_current",
"storagegrid_storage_state_desired",
"storagegrid_storage_utilization_data_bytes",
"storagegrid_storage_utilization_ec_data_bytes",
"storagegrid_storage_utilization_metadata_allowed_bytes",
"storagegrid_storage_utilization_metadata_bytes",
"storagegrid_storage_utilization_metadata_max_bytes",
"storagegrid_storage_utilization_metadata_record_count",
"storagegrid_storage_utilization_metadata_remaining_object_count",
"storagegrid_storage_utilization_metadata_reserved_bytes",
"storagegrid_storage_utilization_total_space_bytes",
"storagegrid_storage_utilization_usable_space_bytes",
"storagegrid_storage_volume_available_space",
"storagegrid_storage_volume_custom_soft_readonly_watermark",
"storagegrid_storage_volume_ec_data",
"storagegrid_storage_volume_maximum_optimized_soft_readonly_watermark",
"storagegrid_storage_volume_minimum_optimized_soft_readonly_watermark",
"storagegrid_storage_volume_object_data_percent",
"storagegrid_storage_volume_replication_data",
"storagegrid_storage_volume_status",
"storagegrid_storage_volume_total_space",
"storagegrid_storageverifier_average_stat_time",
"storagegrid_storageverifier_corrupt_objects_identified",
"storagegrid_storageverifier_corrupt_objects_unidentified",
"storagegrid_storageverifier_data_verification_rate",
"storagegrid_storageverifier_data_verified",
"storagegrid_storageverifier_missed_objects",
"storagegrid_storageverifier_object_verification_rate",
"storagegrid_storageverifier_object_verified",
"storagegrid_storageverifier_percent_completed",
"storagegrid_storageverifier_quarantined_objects",
"storagegrid_swift_data_transfers_bytes_ingested",
"storagegrid_swift_data_transfers_bytes_retrieved",
"storagegrid_swift_operations_failed",
"storagegrid_swift_operations_successful",
"storagegrid_swift_operations_unauthorized",
"storagegrid_swift_requests_total",
"storagegrid_tenant_usage_data_bytes",
"storagegrid_tenant_usage_object_count",
"storagegrid_tenant_usage_quota_bytes",
"storagegrid_tmp_dir_avail_bytes",
"storagegrid_tmp_dir_total_bytes",
"storagegrid_volume_device_free_space",
"storagegrid_volume_device_size",
"storagegrid_volume_device_status",
"storagegrid_volume_device_write_cache_status",
"storagegrid_volume_update_timestamp_seconds",