Input/Output parameter processing

[christian-kreuzberger-dtx]

Assuming you have this incoming CloudEvent:

{
  "data": {
    "configurationChange": {
      "values": {
        "image": "docker.io/keptnexamples/carts:0.13.3"
      }
    },
    "deployment": {
      "deploymentURIsLocal": ["http://carts.sockshop-staging:80"],
      "deploymentURIsPublic": ["http://carts.sockshop-staging.127.0.0.1.nip.io:80"],
    },
    "message": "",
    "project": "sockshop",
    "result": "pass",
    "service": "carts",
    "stage": "staging",
    "status": "succeeded",
  },
  "source": "shipyard-controller",
  "type": "sh.keptn.event.test.triggered",
}

And you want to achieve the following using job-executor-service:

• access data.deployment.dataURIsLocal[0]
• extract protocol, hostname and port (e.g., http, carts.sockshop-staging, port)

What would be the desired way of achieving this? Please comment and like/dislike the options below!

[christian-kreuzberger-dtx]

Option 1

Write your own Docker Container and parse the input variable within a binary/script within this container, e.g.:

apiVersion: v2
actions:
  - name: "Run JMeter"
    events:
      - name: "sh.keptn.event.test.triggered"
        jsonpath:
          property: "$.data.test.teststrategy"
          match: "performance"
    tasks:
      - name: "Run jmeter smoke tests"
        files:
          - jmeter/load.jmx
        image: "docker.io/christiankreuzbergerdtx/my-jmeter-job:1.2.3+MODIFIED"
        cmd: /jmeter
        args:
          - '--uri'
          - $(URI)
        env:
          - name: URI
            value: "$.data.deployment.deploymentURIsLocal[0]"
            valueFrom: event

Note: This option should already work, but requires the user to modify the docker image

[botchk]

Going with this option would also allow a transition into a future where we could pass output of tasks as input to other tasks. In the following a short pseudo example.

actions:
  - name: "Run JMeter"
    events:
      - name: "sh.keptn.event.test.triggered"
        jsonpath:
          property: "$.data.test.teststrategy"
          match: "performance"
    tasks:
      - name: "Prepare args"
        image: "docker.io/christiankreuzbergerdtx/my-jmeter-prepare-args:1.2.3"
        args:
          - '--uri'
          - $(URI)
        env:
          - name: URI
            value: "$.data.deployment.deploymentURIsLocal[0]"
            valueFrom: event
        output:
          - name: PROTOCOL
          - name: PORT
          - name: HOST
      - name: "Run jmeter smoke tests"
        files:
          - jmeter/load.jmx
        image: "docker.io/christiankreuzbergerdtx/my-jmeter-job:1.2.3"
        cmd: /jmeter
        args:
          - '--protocol'
          - $(PROTOCOL)
          - '--port'
          - $(PORT)
          - '--host'
          - $(HOST)
        env:
          - name: PROTOCOL
            valueFrom: task ("Prepare args")
          - name: PORT
            valueFrom: task ("Prepare args")
          - name: HOST
            valueFrom: task ("Prepare args")

The first task takes the URI as input and splits it into the 3 outputs (Basically what your modified image does). The outputs could then be stored as e.g. k8s secrets for the second task which just looks them up and sets them as env as usual. Here the original jmeter image could be used without any additional changes.

Although I am not sure how exactly the part of passing output parameters to the job-executor would look like and probably something like passing on files would also be interesting.

[christian-kreuzberger-dtx]

I like that @yeahservice - just would change the last part to

        env:
          - name: PROTOCOL
            valueFrom: task(task-id)
            value: "$.outputs.PROTOCOL"
          - name: PORT
            valueFrom: task(task-id)
            value: "$.outputs.PORT"
          - name: HOST
            valueFrom: task(task-id)
            value: "$.outputs.HOST"

to stay in line with our existing valueFrom: event syntax

[botchk]

👍 Before starting something like this we should probably decide what to do with #68 which redesigns the env block to be like k8s.

Then it could like this:

env:
  - name: PROTOCOL
    valueFrom:
      taskKeyRef:
        name: task-id
        key: "$.outputs.PROTOCOL"

[christian-kreuzberger-dtx]

We are looking into #68 before we do the other changes :)

Going forward with valueFrom syntax, I like the taskKeyRef example provided here. This is how the full example would look like:

actions:
  - name: "Run JMeter"
    events:
      - name: "sh.keptn.event.test.triggered"
        jsonpath:
          property: "$.data.test.teststrategy"
          match: "performance"
    tasks:
      - name: "Prepare args"
        image: "docker.io/christiankreuzbergerdtx/my-jmeter-prepare-args:1.2.3"
        args:
          - '--uri'
          - $(URI)
        env:
          - name: URI
            valueFrom:                                                  <-- FIRST CHANGE (valueFrom + eventKeyRef)
              eventKeyRef:
                key: "$.data.deployment.deploymentURIsLocal[0]"

        output:                                                         <-- SECOND CHANGE: DEFINE OUTPUT
          - name: PROTOCOL
          - name: PORT
          - name: HOST

      - name: "Run jmeter smoke tests"
        files:
          - jmeter/load.jmx
        image: "docker.io/christiankreuzbergerdtx/my-jmeter-job:1.2.3"
        cmd: /jmeter
        args:
          - '--protocol'
          - $(PROTOCOL)
          - '--port'
          - $(PORT)
          - '--host'
          - $(HOST)
        env:
          - name: PROTOCOL                                        <-- THIRD CHANGE: RE-USING OUTPUT AS ENV
            valueFrom:
              taskKeyRef:
                name: "Prepare args"                        <-- REFERENCE TO THE TASK
                key: "$.outputs.PROTOCOL"                   <-- REFERENCE TO THE OUTPUT VARIABLE
          - name: PORT
              taskKeyRef:
                name: "Prepare args"
                key: "$.outputs.PORT"
          - name: HOST
              taskKeyRef:
                name: "Prepare args"
                key: "$.outputs.HOST"

FYI @didiladi @yeahservice @thschue any feedback on this last example?

[christian-kreuzberger-dtx]

Option 2

Provide some way of processing strings within job-executor-service, with some helper functions (e.g., bash scripts, golang templating, ...)

apiVersion: v2
actions:
  - name: "Run JMeter"
    events:
      - name: "sh.keptn.event.test.triggered"
        jsonpath:
          property: "$.data.test.teststrategy"
          match: "performance"
    tasks:
      - name: "Run jmeter smoke tests"
        files:
          - jmeter/load.jmx
        image: "docker.io/christiankreuzbergerdtx/my-jmeter-job:1.2.3"
        cmd: /jmeter
        args:
          - '--protocol'
          - $(PROTOCOL)
          - '--port'
          - $(PORT)
          - '--host'
          - $(HOST)
        env:
          - name: PROTOCOL
            value: "getProtocol($.data.deployment.deploymentURIsLocal[0])"
            valueFrom: event
          - name: PORT
            value: "getPort($.data.deployment.deploymentURIsLocal[0])"
            valueFrom: event
          - name: HOST
            value: "getHost($.data.deployment.deploymentURIsLocal[0])"
            valueFrom: event

[christian-kreuzberger-dtx]

Option 3

Allow the user to modify/overwrite the entrypoint script

apiVersion: v2
actions:
  - name: "Run JMeter"
    events:
      - name: "sh.keptn.event.test.triggered"
        jsonpath:
          property: "$.data.test.teststrategy"
          match: "performance"
    tasks:
      - name: "Run jmeter smoke tests"
        files:
          - jmeter/load.jmx
        image: "docker.io/christiankreuzbergerdtx/my-jmeter-job:1.2.3"
        entrypoint: |
            #!/bin/bash
            
            # extract the protocol
            proto="$(echo $1 | grep :// | sed -e's,^\(.*://\).*,\1,g')"
            # remove the protocol
            url="$(echo ${1/$proto/})"
            # extract the user (if any)
            user="$(echo $url | grep @ | cut -d@ -f1)"
            # extract the host and port
            hostport="$(echo ${url/$user@/} | cut -d/ -f1)"
            # by request host without port    
            host="$(echo $hostport | sed -e 's,:.*,,g')"
            # by request - try to extract the port
            port="$(echo $hostport | sed -e 's,^.*:,:,g' -e 's,.*:\([0-9]*\).*,\1,g' -e 's,[^0-9],,g')"
            # extract the path (if any)
            path="$(echo $url | grep / | cut -d/ -f2-)"
            
            echo "url: $url"
            echo "  proto: $proto"
            echo "  user: $user"
            echo "  host: $host"
            echo "  port: $port"
            echo "  path: $path"

            bash -c "$cmd --host $host --port $port --protocol $proto"

        cmd: /jmeter
        args:
          - $(URI)

        env:
          - name: URI
            value: "$.data.deployment.deploymentURIsLocal[0]"
            valueFrom: event

[christian-kreuzberger-dtx]

I also just noticed that job-executor-service supports sending "start" and "end" in case of a test.triggered event. Implementation:

job-executor-service/pkg/eventhandler/eventhandlers.go

Lines 231 to 239 in 38c9f19

	if isTestTriggeredEvent(myKeptn.CloudEvent.Type()) && !data.start.IsZero() && !data.end.IsZero() {
	event := &keptnv2.TestFinishedEventData{
	Test: keptnv2.TestFinishedDetails{
	Start: data.start.Format(time.RFC3339),
	End: data.end.Format(time.RFC3339),
	},
	EventData: *eventData,
	}
	_, err = myKeptn.SendTaskFinishedEvent(event, serviceName)

job-executor-service/pkg/eventhandler/eventhandlers.go

Lines 134 to 136 in 38c9f19

	additionalFinishedEventData := dataForFinishedEvent{
	start: time.Now(),
	}

job-executor-service/pkg/eventhandler/eventhandlers.go

Line 186 in 38c9f19

additionalFinishedEventData.end = time.Now()