Skip to content

telemetry-js/telemetry

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Telemetry

Modular metrics for Node.js. Collect, process and publish metrics, picking only the metrics that you need.

npm status node Test JavaScript Style Guide

Table of Contents

Click to expand

Highlights ✨

  • Plugin-based and evented
  • A plugin serves one of 4 roles:
    1. Collector: emits metrics (when "pinged")
    2. Processor: decorates or combines metrics
    3. Schedule: pings other plugins on an interval
    4. Publisher: publishes metrics
  • Add custom metrics sourced from a function, counter or your own plugin
  • Add tags (a.k.a. dimensions) to metrics
  • Includes plugins for automatic tags like instanceid on EC2
  • Locally aggregate before publishing metrics, to save bandwidth
  • Publish to CloudWatch, AppOptics, Logz.io Metrics, stdio or your own publisher.

Usage

To get started you'll need 3 things.

1. The telemetry module (this)

This module controls what and when to collect and publish metrics. It groups plugins into "tasks". One task might publish a memory metric with a name tag to CloudWatch every 5 minutes, while another task publishes the same metric with a name, instanceid and region tag to AppOptics every minute. If the desired tags and interval are the same for both publishers, you can use a single task.

Tasks are also useful to manage the lifetime of metrics. Say we have an application with long-running background jobs that we want metrics on. Each job can have its own telemetry task and add for example a job_id tag to its metrics.

2. At least one collector and publisher

Without a collector, there are no metrics. Without a publisher, the metrics don't go anywhere.

3. A schedule like @telemetry-js/schedule-simple.

Most collectors follow a pull-based model: they must be "pinged" before they emit metrics. A schedule does just that, typically pinging your plugins on a fixed interval.

Examples

Basic

This example collects Node.js memory metrics every 30 seconds and sends them to CloudWatch with a name tag.

const telemetry = require('@telemetry-js/telemetry')()
const mem = require('@telemetry-js/collector-nodejs-memory')
const simple = require('@telemetry-js/schedule-simple')
const tag = require('@telemetry-js/processor-tag')
const cloudwatch = require('@telemetry-js/publisher-cloudwatch')

telemetry.task()
  .collect(mem)
  .schedule(simple, { interval: '30s' })
  .process(tag, { name: 'my-app' })
  .publish(cloudwatch)

// Start tasks
await telemetry.start()

Aggregation

This example collects metrics every 30 seconds, locally aggregates them, and sends summary metrics to CloudWatch and AppOptics every 5 minutes. Those summary metrics have a min, max, sum and count of recorded values, and are called "statistic sets" in CloudWatch.

const telemetry = require('@telemetry-js/telemetry')()
const fn = require('@telemetry-js/collector-function')
const counter = require('@telemetry-js/collector-counter')
const disk = require('@telemetry-js/collector-disk')
const simple = require('@telemetry-js/schedule-simple')
const summarize = require('@telemetry-js/processor-summarize')
const tag = require('@telemetry-js/processor-ec2-instance-id')
const cloudwatch = require('@telemetry-js/publisher-cloudwatch')
const appoptics = require('@telemetry-js/publisher-appoptics')

// Example of custom metric that takes value from a function
const rand = fn.sync('myapp.random.count', { unit: 'count' }, Math.random)
const errors = counter.delta('myapp.errors.delta')

telemetry.task()
  .collect(rand)
  .collect(errors)
  .collect(disk, { metrics: ['*.percent'] })
  .schedule(simple, { interval: '30s' })
  .process(summarize, { window: '5m' })
  .process(tag)
  .publish(cloudwatch)
  .publish(appoptics, { token: '***' })

await telemetry.start()

// Elsewhere in your app
errors.increment(1)

But I Just Want To Publish A One-Time Metric

Got you:

const appoptics = require('@telemetry-js/publisher-appoptics')
const single = require('@telemetry-js/metric').single

const publisher = appoptics({ token: '***' })
const metric = single('myapp.example.count', { unit: 'count', value: 2 })

publisher.publish(metric)

await publisher.flush()

Available Plugins

Collectors

Name Description
disk Free, available and total disk space
net TCP, UDP, ICMP, IP metrics
sockstat /proc/net/sockstat metrics
nodejs-gc Node.js garbage collection duration
nodejs-memory Node.js memory (RSS, heap, external)
nodejs-event-loop-duration Node.js event loop duration
nodejs-event-loop-lag Node.js event loop lag
osmem Free, used and total memory
counter A counter incremented by you
function Collect metric value from a function
redis Redis metrics
incidental Record incidental values
stopwatch Record durations
aws-lb AWS LB node count
dmesg Count certain kernel messages

Schedules

Name Description
simple Collect metrics on a fixed interval

Processors

Name Description
summarize Locally summarize metrics within a time window
tag Add your own tags
ecs-tags Add common tags for ECS container
ec2-instance-id Add instanceid tag, fetched from metadata
ec2-instance-tags Copy all instance tags, fetched from EC2 API
ec2-instance-name Copy only the name tag (if set)
ec2-instance-region Add region tag, fetched from metadata
debug Log metrics and task lifecycle events with debug

Publishers

Name Description
appoptics AppOptics
cloudwatch CloudWatch
logzio-metrics Logz.io Metrics
ndjson Write NDJSON to a stream
debug Log metrics and task lifecycle events with debug

Naming Guide

Metric Names

Lowercase, namespaced by dots (e.g. myapp.foo.bar.count), prefixed with project (myapp.), suffixed with unit (.count).

Metric names from Telemetry plugins are prefixed with telemetry. Custom metrics of your app should be prefixed with your appname. If however, your custom metric is not app-specific, then you could instead use metrics tags to differentiate apps and/or runtime contexts. You might as well write a Telemetry plugin at that point. Do it!

Metric Tags

Lowercase, only a-z. E.g. instanceid, name, project, environment.

Plugin Package Names

Follow the format <role>-<name>, where:

  • <role> is one of collector, processor, publisher, schedule, or, if the plugin serves multiple roles, then simply plugin
  • <name> typically matches the metric name (for collectors), purpose (for processors) or publisher name. If the plugin is a collector that emits multiple metrics (e.g. disk.free.bytes, disk.total.bytes) then use the longest common prefix as <name> (e.g. disk).

API

Yet to document.

Install

With npm do:

npm install @telemetry-js/telemetry

Acknowledgements

This project is kindly sponsored by Reason Cybersecurity Ltd.

reason logo

License

MIT © Vincent Weevers