System Metrics

PerfCounterMonitor.exe (Graphite.System)

Introduction

Besides profiling and instrumenting your code manually you can report various system parameters to StatsD and Graphite, too. This can be accomplished with the Graphite.System block, respectively PerfCounterMonitor.exe.

Graphite.System enables reporting values from the following sources:

• Performance counters
• Event log
• IIS Application Pools

All settings for Graphite.System can be changed and configured in the PerfCounterMonitor.exe.config file. The graphite-client-config-gen tool can help generate a configuration file.

Installation and Usage

PerfCounterMonitor.exe can be run as a standalone console application or installed as a Windows service.

Installation as Windows service

C:\Windows\Microsoft.NET\Framework\v4.0.30319\installutil.exe PerfCounterMonitor.exe
net start GraphiteSystemMonitoring

Remarks

For some Performance Counters it might be required to run PerfCounterMonitor.exe with administrator rights.

Performance Counters

Performance counters are classified by the following three properties: category, counter and instance.

Exploring Performance Counters

To explore all available performance counters in your system you can call PerfCounterMonitor.exe with the -e parameter.

Additional to the -e paramater a category and instance name can be specified to show more details:

PerfCounterMonitor.exe -e [category [instance]]

Example

Show all performance counters for category "Processor" and instance "_Total":

Command:

PerfCounterMonitor.exe -e Processor "_Total"

Result:

Category: Processor
Instance: _Total

Counters:
    % Processor Time (Timer100NsInverse)
    % User Time (Timer100Ns)
    % Privileged Time (Timer100Ns)
    Interrupts/sec (RateOfCountsPerSecond32)
    % DPC Time (Timer100Ns)
    % Interrupt Time (Timer100Ns)
    DPCs Queued/sec (RateOfCountsPerSecond32)
    DPC Rate (NumberOfItems32)
    % Idle Time (Timer100Ns)
    % C1 Time (Timer100Ns)
    % C2 Time (Timer100Ns)
    % C3 Time (Timer100Ns)
    C1 Transitions/sec (RateOfCountsPerSecond64)
    C2 Transitions/sec (RateOfCountsPerSecond64)
    C3 Transitions/sec (RateOfCountsPerSecond64)

Configuration

PerfCounterMonitor.exe can be configured to periodically report values of any Performance Counter through the PerfCounterMonitor.exe.config file:

<graphite.system>
    <counters>
      <clear />
      <add 
        key="processor.time" 
        category="Processor" instance="_Total" counter="% Processor Time" 
        type="gauge" target="statsd" interval="30" />
      <add 
        key="processor.dpc" 
        category="Processor" instance="_Total" counter="DPC Rate" 
        type="gauge" target="statsd" interval="5" />
    </counters>
  </graphite.system>

Configuration options

Required

• key - the metric key.
• category - the performance counter category name.
• instance - the performance counter instance name. For counters without an instance this attribute must be set to "" (e.g. Memory\Available MBytes).
• counter - the performance counter name.
• type - the metric type (counter, timing or gauge).
• target - the monitoring target (statsd or graphite).

Optional

• interval - the interval in seconds this metric should be read/reported (defaults to 30 seconds).
• sampling - for StatsD counter types to sample values before submission (reduces number of submitted packets, see http://github.com/etsy/statsd for more details)
• retry - retry creation counter listener, if it fails on start (true or false).

Event log

With PerfCounterMonitor.exe it is possible to listen for new entries in the event log.

For each new entry matching the configured properties, a metric with specified value is reported.

Configuration

<eventlog>
  <clear />
  <add
    key="admin.errors"
    protocol="Application" entrytypes="Error"
    type="gauge" target="graphite" />
  <add
    key="admin.dotnet.exceptions"
    protocol="Application" source=".NET Runtime" entrytypes="Error"
    type="gauge" target="graphite" />
  <add
    key="admin.dotnet.warnings"
    protocol="Application" source=".NET Runtime" entrytypes="Warning"
    type="gauge" target="graphite" />
</eventlog>

Configuration options

Required

• key - the metric key.
• protocol - the event log protocol (e.g. "Application", "System", "Security", etc.).
• source - the log source value.
• entrytypes - type of event log entries to listen for (Error, Warning, Information, SuccessAudit and FailureAudit). Multiple types can be seperated by ";" or ",".
• type - the metric type (counter, timing or gauge).
• target - the monitoring target (statsd or graphite).

Optional

• category - the category of the messages.
• value - the reported value (defaults to 1).
• sampling - for StatsD counter types to sample values before submission (reduces number of submitted packets, see http://github.com/etsy/statsd for more details).

IIS Application Pools

The "working set" (i.e. used memory) of IIS Application Pools can be reported by PerfCounterMonitor.exe.

Configuration

The application pool names must be specified in PerfCounterMonitor.exe.config:

<appPool>
  <add key="admin.appPool.default" appPoolName="DefaultApplicationPool" workingSet="true"
    type="gauge" target="statsd" />
  <add key="admin.appPool.asp4" appPoolName="ASP.NET v4.0" category="process" counter="Working Set"
    type="gauge" target="statsd" />
</appPool>

As seen in the example the AppPool listeners can be configured in two different ways: through the "shortcut" workingSet="true" or by specifying a category and a counter manually.

Configuration options

Required

• key - the metric key.
• appPoolName - the application pool name.
• type - the metric type (counter, timing or gauge).
• target - the monitoring target (statsd or graphite).

Optional

• interval - the interval in seconds this metric should be read/reported (defaults to 30 seconds).
• workingSet - must be true to act as a shortcut for Process\Working Set.
• category - the performance counter category name (usually this should be "Process", by because the AppPool name is resolved to a process id/name.
• counter - the performance counter name.