Skip to content

OpenAF/oJob-common

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

oJob-common

version

A set of base oJobs as common building blocks for custom oJobs.

Check the documentation for some of them:

oJob-common Description
oJobBasics Basic init/done logging, sh command execution, etc.
oJobEmail Send emails.
oJobSSH SSH to execute commands and upload/download files.
oJobSQL Query or execute SQL to a JDBC database.
oJobES Logging to ElasticSearch
oJobNet Testing network connectivity
oJob Building a simple HTTP(s) server
oJobRest Building a simple REST server
oJobOPack Simplified OPack creation
oJobRAID Simplified RAID AF operation execution.
oJobWatchDog Helps build a cron-based process "watchdog".

oJobBasics

oJobBasics jobs
oJob sh
oJob background processes
oJob From global
oJob Args from JSON
oJob Args from YAML
oJob Send email
oJob Jobs Report

oJob sh

This job runs a local shell command and accepts the following arguments:

Argument Type Mandatory Description
cmd String/Array Yes The command to execute (or an array of commands)
quiet Boolean No Determines if the stdout should be visible or not (default is false)
directory String No Sets the working directory for the command
stdin String No Provide any stdin needed
exitcode Number No Determines what exitcode should be consider success (default is 0)

Example:

include:
  - oJobBasics.yaml

jobs:
  # Example to show 123
  #
  - name: Example Echo 123
    to  : oJob sh
    args:
      cmd: echo 123

  # Example to show how you can combine multiple commands
  #
  - name: Example with multiple commands
    to  : oJob sh
    args:
      cmd:
        - echo -- [You are in `pwd`] -------------

        - >-
          echo -- [Previous directory] ----------- &&
          cd .. &&
          ls -1

        - >-
          echo -- [Current directory] ------------ &&
          cd . &&
          ls -1

  # Example to parse output
  #
  - name: Example to parse output
    from: oJob sh
    args:
      quiet: true
      cmd  : >-
         curl -X GET "https://httpbin.org/json" -H "accept: application/json"

    exec: |
      print("STDOUT   = " + stringify(jsonParse(args.stdout)));
      print("STDERR   = " + args.stderr);
      print("EXITCODE = " + args.exitcode);

  # Example to prepare cmd
  #
  - name: Example to prepare cmd
    to  : oJob sh
    exec: |
      args.cmd = "echo " + new Date();

todo:
  - Example Echo 123
  - Example with multiple commands
  - Example to parse output
  - Example to prepare cmd

oJob background processes

These jobs will run a command in background and wait for all to finish if needed. oJob Process Launch expects:

Argument Type Mandatory Description
cmd String/Array Yes The command to execute (or an array of commands) in background.
console Boolean No Defines if stdout/stderr should be printed or not (defaults to true)
success String No Code to execute as a function in case of success. Receives a "res" map from executing a sh function.
error String No Code to execute as a function in case of error. Receives a "e" exception and a "cmd" with the original cmd argument.

Example:

include:
  - oJobBasics.yaml

todo:
  - Launch proc 1
  - Launch proc 2
  - oJob Process Wait

jobs:
  - name: Launch proc 1
    to  :
      - oJob Process Launch
    args:
      cmd    : "myProc1.sh"
      success: "log(stringify(res, void 0, ''));"

  - name: Launch proc 2
    to  :
      - oJob Process Launch
    args:
      cmd : "myProc2.sh"

oJob From global

This job will reset or merge a global variable map. Expects:

Argument Type Mandatory Description
global String Yes Load arguments from the global variable specified.

oJob Args from JSON

This job will load the args map from a JSON file. Expects:

Argument Type Mandatory Description
file String Yes The filepath to read the JSON file from.
global String No Alternatively load to the global variable specified.

oJob Args from YAML

This job will load the args map from a YAML file. Expects:

Argument Type Mandatory Description
file String Yes The filepath to read the YAML file from.
global String No Alternatively load to the global variable specified.

oJob Jobs Report

This job prints a job execution report on shutdown.

Argument Type Mandatory Description
format String No Output in json, yaml or table format (defaults to table)

Note: you can also use the job "oJob Jobs Final Report" to output the report on shutdown.

include:
  - oJobBasics.yaml

jobs:
  # some jobs

todo:
  - oJob Jobs Final Report
  # some todos

oJobEmail

oJobEmail jobs
oJob Send email

oJob Send email

This job tries to send an email. Expects:

Argument Type Mandatory Description
server String Yes The email server to use.
port Number No The email server port to use.
from String Yes The email from address.
to Array Yes The email to addresses.
cc Array No The email cc addresses.
bcc Array No The email bcc addresses.
isHTML Boolean No Specifies if the email is in HTML format.
subject String Yes The email subject (a hbs template using args as data).
output String Yes The email body message (a hbs template using args as data).
altOutput String No The email body alternative message (defaults to message).
credentials Map No The email server credentials (user and pass).
useSSL Boolean No If the email server uses SSL.
useTLS Boolean No If the email server uses TLS.
embedFiles Array No Array of maps (with file and name) to embeded on the email.
addAttachments Array No Array of maps (with file, isInLine, name) to attach on the email.
addImages Array No Array of urls to images (only available if isHTML = true)-
embedURLs Array No Array of maps (with url and name) to embeded on the email.
debug Boolean No Determines if it should debug the process.

Example:

smtp-config.yaml

from       : my.email@some.domain
server     : my.smtp.server
credentials:
  user: user1
  pass: pass1
useSSL     : true

sendEmail.yaml

include:
  - oJobEmail.yaml

jobs:
  - name: Send email test
    from: oJob Args from YAML
    to  : oJob Send email
    args:
      file   : smtp-config.yaml
      to     :
        - email1@some.domain
      subject: Test email
      output : My test email

todo:
  - Send email test

oJobSSH

SSH Exec

Executes commands on a SSH connection. The expected arguments are:

Argument Type Mandatory Description
cmd String/Array Yes A SSH command-line to execute or and array of it (keep in mind that this isn't bash)
stdin String No An optional SSH stdin for the command-line to execute
chHosts String No A channel with hosts configurations to use instead of individual config
host String No The SSH host
port Number No The SSH port (defaults to 22)
login String No The SSH login
pass String No The SSH pass
key String No The path to a SSH key file (optional)
exitcode Number No Determines what exitcode should be consider success (default is 0)
sudo String No Sudo's to the corresponding user to executing the command-line
quiet Boolean No Determines if no output of the command(s) execution should be provided (default to false)

Example:

hosts.yaml

- name : my host 1
  host : host1.local
  login: user1
  pass : pass1234
  key  : key.rsa

- name : my host 2
  host : host2.local
  login: user1
  pass : pass1234
  key  : key.rsa

example.yaml

include:
  - oJobSSH.yaml

ojob:
  sequential: true

jobs:
  # Example to show 123
  #
  - name: Example Echo 123
    deps:
      - SSH Load hosts
    to  : SSH Exec
    args:
      cmd: echo 123

  # Example to show how you can combine multiple commands
  #
  - name: Example with multiple commands
    deps:
      - SSH Load hosts
    to  : SSH Exec
    args:
      cmd:
        - echo -- [You are in `pwd`] -------------

        - >-
          echo -- [Previous directory] ----------- &&
          cd .. &&
          ls -1

        - >-
          echo -- [Current directory] ------------ &&
          cd . &&
          ls -1

  # Example to parse output
  #
  - name: Example to parse output
    deps:
      - SSH Load hosts
    from: SSH Exec
    args:
      quiet: true
      cmd  : >-
         curl -X GET "https://httpbin.org/json" -H "accept: application/json"

    exec: |
      print("STDOUT   = " + stringify(jsonParse(args.stdout)));
      print("STDERR   = " + args.stderr);
      print("EXITCODE = " + args.exitcode);

todo:
  - name: SSH Load hosts
    args:
      chHosts: myhosts
      file   : hosts.yaml
  
  - name: Example Echo 123
    args:
      chHosts: myhosts

  - name: Example with multiple commands
    args:
      chHosts: myhosts

  - name: Example to parse output
    args:
      chHosts: myhosts

oJobSQL

Allows for easy SQL query or executiong in any JDBC database.

Basic example:

consts:
  dbJDBC: &jdbcurl  jdbc:oracle:thin:@1.2.3.4:1521:ORCL
  dbUser: &jdbcuser scott
  dbPass: &jdbcpass tiger

include:
  - ojobSQL.yaml

jobs:
  ########################
  - name: Get current date
    from: SQL
    args:
      DBURL : *jdbcurl
      DBUser: *jdbcuser
      DBPass: *jdbcpass
      sql   : select current_date cd, sysdate sd from dual
    exec: |
      tprint("Current date = {{CD}}", args.output[0]);
      tprint("System date  = {{SD}}", args.output[0]);

  ##########################
  - name: Get generated data
    from: SQL RAID
    args:
      DBURL : *jdbcurl
      DBUser: *jdbcuser
      DBPass: *jdbcpass
      format : table
      sql    : |
        SELECT level, current_date, sysdate
        FROM   dual
        CONNECT BY level <= 10

todo:
  - Get current date
  - Get generated data

Example using RAID:

consts:
  raidURL: &raidurl http://user:pass@1.2.3.4:1234/xdt
  raidDB : &raiddb  Dat

include:
  - ojobSQL.yaml

jobs:
  ########################
  - name: Get current date
    from: SQL RAID
    args:
      raidURL: *raidurl
      raidDB : *raiddb
      sql    : select current_date cd, sysdate sd from dual
    exec: |
      tprint("Current date = {{CD}}", args.output[0]);
      tprint("System date  = {{SD}}", args.output[0]);

  ##########################
  - name: Get generated data
    from: SQL RAID
    args:
      raidURL: *raidurl
      raidDB : *raiddb
      format : table
      sql    : |
        SELECT level, current_date, sysdate
        FROM   dual
        CONNECT BY level <= 10

todo:
  - Get current date
  - Get generated data

oJobES

Start Log to ES

Starts logging to ElasticSearch. Expects:

Argument Type Description
url String The ElasticSearch cluster URL
index String An ElasticSearch index name (it will be suffixed automatically with the current date)
format String The format of the date if different from day
host String A LogStash like host to identify in each entry
user String A user name
pass String A password (encrypted or not)

Example:

include:
  - oJobES.yaml

# [...]

# Log to ElasticSearch
todo:
  - name: Start Log to ES
    args:
      url  : http://my.es.cluster
      index: mylogs
      host : myjob

Example by week:

include:
  - oJobES.yaml

# [...]

# Log to ElasticSearch
todo:
  - name: Start Log to ES
    args:
      url   : http://my.es.cluster
      index : mylogs
      format: yyyy.ww
      host  : myjob

Stop ES Logging

Stops logging to ElasticSearch.

Example:

# Stop logging to ElasticSearch
todo:
  - Stop ES logging

oJobNet

tbc

oJobHTTPd

Simplifies the creation of one or more HTTP(s) server where you just provide which functions to run for each URI on a http(s) server. The function will receive all the requests parameters and return the content for the browser. If you wish to return JSON please check oJobRest.

It's composed of 3 jobs:

  • HTTP Start Server
  • HTTP Stop Server
  • HTTP Service
  • HTTP File Browse

The job "HTTP Start Server" expects:

Argument Type Mandatory Description
port Number No The port number where to assign the HTTP(s) server (defaults to 8091)
keystore String No The keystore for the SSL certificates to create a HTTPS server
pass String No The password for the keystore to create a HTTPS server
host String No The ip address of the local network interface to which to bind this HTTP server (defaults to 0.0.0.0)
cp String No Provide a folder where the keystore file is to include it on the current classpath (Java requires for keystores to be on the execution classpath)
hs HTTPServer object No An already created HTTPServer to which to bind the HTTP services
mapLibs Boolean No Map internal OpenAF libs like JQuery, highlight css, etc.

The job "HTTP Stop Server" expects:

Argument Type Mandatory Description
port Number No The port number where to assign the HTTP(s) server (defaults to 8091)

The job "HTTP service" expects:

Argument Type Mandatory Description
port Number No The port number where to assign the HTTP(s) server (defaults to 8091)
uri String No The URI where the HTTP(s) service will be available.
execURI String Yes The code to execute whenever the uri is requested. The code is included into a function that receives the arguments: request and server. "request" is a map containing all the request properties. "server" is the HTTPServer object for which you should use replyOKText, replyOKJSON, etc.

The job "HTTP File Browse" expects:

Argument Type Mandatory Description
port Number No The port number where to assign the HTTP(s) server (defaults to 8091)
uri String No The URI where the HTTP(s) service will be available.
path String Yes The canonical path to the file path that will be made available for browsing.
browse String No If "false" no browsing interface will be included.

Example:

include:
  - oJobHTTPd.yaml

ojob:
  daemon: true

jobs:
  - name: Hello world
    to  : HTTP Service
    args:
      port   : 8080
      uri    : /
      execURI: return server.replyOKText("Hello world!");

  - name: Echo
    to  : HTTP Service
    args:
      port   : 8080
      uri    : /echo
      execURI: return server.replyOKJSON(stringify(request));
 
  - name: Browser
    to  : HTTP File Browse
    args:
      port: 8080
      uri : /browser
      path: .

  - name: README
    to  : HTTP Service
    args:
      port   : 8080
      uri    : /README
      execURI: return ow.server.httpd.replyFileMD(server, ".", "/README", request.uri, void 0, [ "README.md" ]); 

todo: 

  # Starts the server
  - name: HTTP Start Server
    args:
      port   : 8080
      mapLibs: true

  # Sets a shutdown job once the everything is stopped.
  - name: HTTP Stop Server 
    args:
      port: 8080
    
  # Sets for every URI to return Hello world
  - Hello world

  # Sets that /echo will return the actual request map
  - Echo

  # Sets that /README shows this README.md file
  - README

  # Sets that /browser shows a simple browse interface for the current directory.
  - Browser

oJobREST

In the same line as oJobHTTPd simplifies the specific creation of REST HTTP(s) servers.

It's composed of 3 jobs:

  • REST Start Server
  • REST Stop Server
  • REST Service

The job "REST Start Server" expects:

Argument Type Mandatory Description
port Number No The port number where to assign the HTTP(s) server (defaults to 8091)
keystore String No The keystore for the SSL certificates to create a HTTPS server
pass String No The password for the keystore to create a HTTPS server
host String No The ip address of the local network interface to which to bind this HTTP server (defaults to 0.0.0.0)
cp String No Provide a folder where the keystore file is to include it on the current classpath (Java requires for keystores to be on the execution classpath)
hs HTTPServer object No An already created HTTPServer to which to bind the HTTP services
mapLibs Boolean No Map internal OpenAF libs like JQuery, highlight css, etc.

The job "REST Stop Server" expects:

Argument Type Mandatory Description
port Number No The port number where to assign the HTTP(s) server (defaults to 8091)

The job "REST service" expects:

Argument Type Mandatory Description
port Number No The port number where to assign the HTTP(s) server (defaults to 8091)
uri String No The URI where the HTTP(s) service will be available.
execGET String Yes The code to execute whenever the uri is requested with a GET verb. The code is included into a function that receives the argument: idxs. "idxs" is a map containing all the parameters from the URL.
execPOST String Yes The code to execute whenever the uri is requested with a POST verb. The code is included into a function that receives the arguments: idxs and data. "idxs" is a map containing all the parameters from the URL. "data" is a map containing all the parameters passed on the request body.
execPUT String Yes The code to execute whenever the uri is requested with a PUT verb. The code is included into a function that receives the arguments: idxs and data. "idxs" is a map containing all the parameters from the URL. "data" is a map containing all the parameters passed on the request body.
execDELETE String Yes The code to execute whenever the uri is requested with a DELETE verb. The code is included into a function that receives the argument: idxs. "idxs" is a map containing all the parameters from the URL.
returnWithParams Boolean No Changes the behaviour of the return of each exec* function to use a map to force mimetype, http code, etc. (see help ow.server.rest.reply for more details)
error Boolean No If true and an error occurs in execGET, execPOST, execPUT or execDELETE will return a map with the exception in a map with a key __error (defaults to true)

oJobOPack

Simplifies the creation of oPack files based on an url or oPack name.

The job "oPack Pack external" expects:

Argument Type Mandatory Description
name String Yes The name or URL to an oPack
tmpDir String No The temporary folder to use during the process (defaults to ./tmp)
outputDir String No Output folder where the oPack will be placed (defaults to .)

Example:

include:
  - oJobOPack.yaml

todo:
  - name: oPack Pack external
    args:
      - name  : https://raw.githubusercontent.com/OpenAF/nAttrMon/master
        tmpDir: nAttrMon

      - name  : APIs
        tmpDir: APIs

      - name  : GoogleCompiler
        tmpDir: GoogleCompiler
        
      - name  : GooglePhoneNumber
        tmpDir: GooglePhoneNumber

oJobRAID

Simplified RAID AF operation execution.

Expects:

Argument Type Mandatory Description
raidURL String Yes A RAID AF connection URL.
operation String Yes The RAID AF operation to execute.
input Map No The RAID AF operation input map.
format String No If not quiet it displays the result on a format you can choose between "prettyprint" (default), "pmap", "parametermap", "yaml" or "json"
quiet Boolean No If true no output will be displayed.
include:
  - oJobRAID.yaml

todo:
  - name: RAID AF
    args:
      raidURL  : http://user:pass@127.0.0.1:8090/#/web/guest/home
      operation: Ping
      input    :
        test: 123
      format   : prettyprint

oJobWatchDog

Helps build a cron-based process "watchdog" for checking if a daemon process isn't running, has a "fatal" error on the log or any custom way to check the responsive of the daemon process.

Expects:

Argument Type Mandatory Description
checks Map No Map of checks to determine if the daemon process should be restarted or not.
checks.pid.file String No If a pid file location is provided it will check if the corresponding pid is running. If not it sets to trigger a stop and start operation.
checks.log.folder String No Folder where the log files to check are located.
checks.log.fileRE String No Checks for files matching fileRE choosing the latest by modified date.
checks.log.stringRE String No Array of regular expressions strings to look for. If found assumes a restart is needed.
checks.log.histFile String No The file where to store the history of findings on the file to avoid duplicate findings.
checks.log.olderMin Number No Checks if the latest log file (fileRE) by modified date is older than x minutes and if yes assumes it needs to be restarted.
checks.custom.exec String No Executes the corresponding code in a function and passes if returns true or fails assuming a restart is needed if returns false.
quiet Boolean No If true will only produce logging if something is not right (default is true)
cmdToStop String No If defined will execute the command on the stop event.
execToStop String No If defined will execute the code on the stop event.
jobToStop String No If defined will execute the job on the stop event.
waitAfterStop Number No Number of ms to wait after stopping.
workDirStop String No The working directory to use for the stop command.
timeoutStop Number No Timeout waiting for cmdToStop to exit.
exitCodeStop Number No If defined the cmdToStop exitcode must be this value.
cmdToStart String No Command to startup on the start event (make sure it exits after starting).
execToStart String No If defined will execute the code on the start event.
jobToStart String No If defined will execute the job on the start event.
waitAfterStart Number No Number of ms to wait after starting.
workDirStart String No The working directory to use for startup command.
timeoutStart Number No Timeout waiting for cmdToStart to exit.
exitCodeStart Number No If defined the cmdToStart exitcode must be this value.

Example:

include:
  - oJobWatchDog.yaml

ojob:
  logToFile   :
    logToFolder          : /some/path/watchdog.logs
    HKhowLongAgoInMinutes: 11152
    setLogOff            : false
  logToConsole: false
  logJobs     : false
  unique      :
    pidFile     : /some/path/watchdog.pid
    killPrevious: false
  checkStall  :
    everySeconds    : 1
    killAfterSeconds: 60

jobs:
  - name: Watch my logya daemon
    to  : oJob WatchDog
    args:
      checks  :
        pid:
          file: /some/path/a.pid
        log   :
          folder  : /some/path/logya
          fileRE  : log-\d+-\d+-\d+.log 
          histFile: /some/path/logya/logya.json
          stringRE: OutOfMemory
        custom:
          exec: |
            print(123);
            return false;

      cmdToStart    : start ojob /some/path/a.yaml
      workDirStart  : /some/path/
      waitAfterStart: 5000

      execToStop    : |
        pidKill(io.readFileString("/some/path/a.pid"), true);

      quiet         : false

todo:
  - nAttrMon watchdog

Shortcuts examples:

oJobTest.yaml

include:
- oJobTest.yaml

todo:
- (test      ): oJob::a
  ((job     )): a
- (test      ): oJob::b
  ((job     )): b
- (test      ): Script::Test
  ((func    )): |
    sleep(1500, true)
    ow.test.assert(1, 1, "Problem with assert in script test")
- (testAssert): Problem with a and b
  ((a       )): 123
  ((b       )): 124
- (testGenMD ): __
  ((file    )): results.md

jobs:
- name: a
  exec: |
    sleep(1500, true)

- name: b
  exec: |
    throw "MY error!"

oJobHTTPd.yaml

todo:
# Starts the server
- (httpdStart  ): &PORT 12345

# Setups the default answer, /healthz, /livez and /metrics
- (httpdDefault): *PORT
  ((uri       )): /
- (httpdHealthz): *PORT 
- (httpdMetrics): *PORT
  ((prefix    )): mytest

# Allows browsing of files
- (httpdFileBrowse): *PORT
  ((uri          )): /browse
  ((path         )): .

# Allows for the upload of files
- (httpdUpload): *PORT
  ((uri      )): /upload
  ((path     )): .

# Adds a custom metric
- (httpdAddMetric): global-counter
  ((fn          )): | 
    // Sets an atomic counter if one does not exist and returns a counter increment
    if (isUnDef(global.counter)) global.counter = $atomic()
    return global.counter.inc()

# /test calls the 'test' job
- (httpdService): *PORT
  ((uri       )): /test
  ((execURI   )): |
    // Shows all request components for debug
    cprint(request)
    // Returns the result of calling the job 'test' passing the request and expecting an ANSWER to be returned
    return ow.server.httpd.reply($job("test", request).ANSWER)

jobs:
# ---------------------------------
# Job test is written in shell code
- name: test
  lang: shell
  exec: |
    # Sets ANSWER in shell script
    ANSWER="Echo from the shell (a: {{params.a}})"

    # return ANSWER

# Includes the http server functionality
include:
- oJobHTTPd.yaml

# Makes sures it runs forever and oJob-common is included
ojob:
  daemon: true
  opacks:
    oJob-common

About

A set common building blocks for custom OpenAF's oJobs.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published