New Relic Kubernetes Operator

• Overview
• Quick Start
• Using the Operator to Provision Resources
  • Create a New Relic alert policy
  • Create a New Relic NRQL alert condition
• Development

Overview

The newrelic-kubernetes-operator is a Kubernetes Operator that facilitates management of New Relic resources from within your Kubernetes configuration. Managing New Relic resources via custom Kubernetes objects can be done the same way you manage built-in Kubernetes objects.

Currently the operator supports managing the following resources:

• Alert Policies
• NRQL Alert Conditions.
• Alert Conditions for APM, Browser and mobile
• Alert Channels

Quick Start

Note: These quick start instructions do not require you to clone the repo.

Running Kubernetes in a Docker container locally with kind

1. Install docker, kubectl, kustomize, and kind

   brew cask install docker
   brew install kubernetes-cli kustomize kind

2. Create a test cluster with kind

   kind create cluster --name newrelic
   kubectl cluster-info

3. Install cert-manager

   kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.yaml

   Note: This takes a minute or two to finish so wait a minute before going on to the next step.

   You can also confirm it's running with the command kubectl rollout status deployment -n cert-manager cert-manager-webhook

4. Install the operator in the test cluster.

   kustomize build https://github.com/newrelic/newrelic-kubernetes-operator/config/default | kubectl apply -f -

   Note: This will install operator on whatever kubernetes cluster kubectl is configured to use.

Using a custom container

If you want to deploy the operator in a custom container you can override the image name with a kustomize.yaml file.

1. Create a new kustomize.yaml file

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   namespace: newrelic-kubernetes-operator-system
   resources:
     - github.com/newrelic/newrelic-kubernetes-operator/config/default
   images:
     - name: newrelic/kubernetes-operator:snapshot
       newName: <CUSTOM_IMAGE>
       newTag: <CUSTOM_TAG>

2. The apply the file with:

   kustomize build . | kubectl apply -f -

Provision New Relic resources with the operator

Once you've completed the Quick Start section, you can start provisioning New Relic resources with our New Relic Kubernetes objects.

Create a New Relic alert policy with NRQL alert conditions

1. We'll be using the following example policy configuration file. You will need to update the api_key field with your New Relic personal API key.
   

   Once you've added your API key, we can apply it your local cluster.

   kubectl apply -f examples/example_policy.yaml

   Note: You can also use a Kubernetes secret for providing your API key. We've provided an example secret configuration file in case you want to use this method. You'll need to replace api_key with api_key_secret.

2. See your configured policies with the following command.

   kubectl describe alertspolicies.nr.k8s.newrelic.com

   Note: You should also see the newly created policy within your New Relic account.

The operator will create and update alert policies and NRQL alert conditions as needed by applying your configuration files with kubectl apply -f <filename>

Create a NRQL alert condition and add it to an existing alert policy

1. We'll be using the following example NRQL alert condition configuration file. You will need to update the api_key field with your New Relic personal API key.
   

Create an Alerts Channel

1. We'll be using the following example alerts channel configuration file. You will need to update the api_key field with your New Relic personal API key.
   

   Note: The New Relic Alerts API does not allow updating Alerts Channels. In order to change a channel, you will need to either rename the k8s AlertsChannel object to create a new one and delete the old one or manually delete the k8s AlertsChannel object and create a new one.

Monitoring the New Relic Operator

The New Relic Operator uses the New Relic Go Agent to report monitoring statistics. This can be activated by adding a Kubernetes Secret and an option ConfigMap to configure the agent. We'll be using Example_New_Relic_Agent_Config

apiVersion: v1
kind: Secret
metadata:
  name: nr-agent
  namespace: newrelic-kubernetes-operator-system
type: Opaque
stringData:
  license-key: <your New Relic license key, base64 encoded>
  
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nr-agent
  namespace: newrelic-kubernetes-operator-system
data:
  # defaults to New Relic Kubernetes Operator
  app-name: <your desired New Relic App name for the Operator>
  # Defaults to collector.newrelic.com
  host: <New Relic endpoint>
  # Defaults to false
  log-level: debug

Note: If the agent isn't reporting, make sure to check your base64 encoding didn't include a /n character.

Uninstall the operator

The Operator can be removed with the reverse of installation, namely building the kubernetes resource files with kustomize and running kubectl delete

kustomize build github.com/newrelic/newrelic-kubernetes-operator/config/default | kubectl delete -f -

Development

This section should get you set up properly for doing development on the operator.

Requirements

• Go v1.13+
• Docker (with Kubernetes enabled)
• kubectl
• kustomize
• kubebuilder

Code

1. Clone the repo

   git clone git@github.com:newrelic/newrelic-kubernetes-operator.git

2. Install kubebuilder following the instructions for your operating system. This installation will also get etcd and kube-apiserver which are needed for the tests.
   

   Note: Do not install kubebuilder with brew. Homebrew's kubebuilder package will not provide all the necessary dependencies for running the tests.

3. Run the test suite, which uses the Ginkgo testing framework. Using the make targets is the quickest way to get started with testing.

   • Running tests with make

     make test              # runs all tests
     make test-unit         # only runs unit tests
     make test-integration  # only runs integration tests

   • Linting the codebase

     make lint

4. Perform the steps from the Quick Start section, except use the following instead of step 4 to install with the :snapshot image:

   kustomize build config/development | kubectl apply -f -

5. Confirm your configuration was deployed to your local Kubernetes cluster (the one that we created with kind).
   

   • Show your namespaces. You should see newrelic-kubernetes-operator-system in the list of namespaces.

     kubectl get namespaces

   • Show the nodes within the newrelic-kubernetes-operator-system namespace.

     kubectl get nodes -n newrelic-kubernetes-operator-system

     You should see something similar to the following output:

     NAME                     STATUS   ROLES    AGE    VERSION
     newrelic-control-plane   Ready    master   163m   v1.18.2
     

6. Personalize your development alert policy

   Edit examples/development/personal* with your name and API key.

   Next, create your customized alert policy:

   kustomize build examples | kubectl apply -f -

   To delete your customized policy:

   kustomize build examples | kubectl delete -f -

Helpful commands

# Describe the currently configured policies.
kubectl describe alertspolicies.nr.k8s.newrelic.com

# Describe the currently configured alert conditions.
kubectl describe alertsnrqlconditions.nr.k8s.newrelic.com

# Get the node being used for the newrelic operator.
kubectl get nodes -n newrelic-kubernetes-operator-system

# Describe the node being used for the newrelic operator.
kubectl describe node <your-node-name>

# Tail logs of the operator's manager container (useful during development).
# Use the `describe node` command above to locate your manager controller.
kubectl logs -f -n newrelic-kubernetes-operator-system -c manager newrelic-kubernetes-operator-controller-manager-<hash from>