Skip to content

πŸ’» CLI for reporting events to Faros platform

License

Notifications You must be signed in to change notification settings

faros-ai/faros-events-cli

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

πŸ’» Faros Events CLI CI Latest Release

CLI for reporting events to Faros platform.

Installation

Using Docker

Requirements: docker

docker pull farosai/faros-events-cli:v0.6.11 && docker run farosai/faros-events-cli:v0.6.11 help

Using Bash

Requirements: curl, jq (1.6+), sed, awk (we recommend gawk).

Either download the script manually or invoke the script directly with curl:

bash <(curl -s https://raw.githubusercontent.com/faros-ai/faros-events-cli/v0.6.11/faros_event.sh) help

Instrumenting CI pipelines

Report CI events to the Faros platform if you would like to analyze success/failure rates of your CI pipelines and how long different stages take.

This CI event reports a successful build event where an artifact is built from a commit:

./faros_event.sh CI -k "<faros_api_key>" \
    --commit "<commit_source>://<commit_organization>/<commit_repository>/<commit_sha>" \
    --artifact "<artifact_source>://<artifact_organization>/<artifact_repository>/<artifact_id>" \
    --run "<run_source>://<run_organization>/<run_pipeline>/<run_id>" \
    --run_status "Success" \
    --run_start_time "2021-07-20T18:05:46.019Z" \
    --run_end_time "2021-07-20T18:08:42.024Z"

Example usage:

./faros_event.sh CI -k "<faros_api_key>" \
    --commit "GitHub://faros-ai/faros-events-cli/4414ad2b3b13b17055171678437a92e5d788cad1" \
    --artifact "Docker://farosai/faros-events-cli/v0.6.11" \
    --run "Jenkins://faros-ai/faros-events-cli/168_1700016590" \
    --run_status "Success" \
    --run_start_time "2023-11-14T18:05:46.019Z" \
    --run_end_time "2023-11-14T18:08:42.024Z"

❗ The run_status is an enum. Read the documentation on arguments here for accepted values.

❗ If your CI pipeline does not build artifacts, omit the --artifact parameter, and be sure to add the --no-artifact flag.

Reporting builds and build steps in parts

In addition to tracking build outcomes, you can also instrument specific steps in your build processes, and report on information in parts, as it becomes available.

For example, after reporting the start of a build:

./faros_event.sh CI -k "<faros_api_key>" \
    --commit "<commit_source>://<commit_organization>/<commit_repository>/<commit_sha>" \
    --artifact "<artifact_source>://<artifact_organization>/<artifact_repository>/<artifact_id>" \
    --run "<run_source>://<run_organization>/<run_pipeline>/<run_id>" \
    --run_start_time "Now"

You can report the start of a specific build step:

./faros_event.sh CI -k "<faros_api_key>" \
    --run "<run_source>://<run_organization>/<run_pipeline>/<run_id>" \
    --run_step_id "<run_step_id>" \
    --run_step_start_time "Now"

Then report its outcome and end time:

./faros_event.sh CI -k "<faros_api_key>" \
    --run "<run_source>://<run_organization>/<run_pipeline>/<run_id>" \
    --run_step_id "<run_step_id>" \
    --run_step_status "Success" \
    --run_step_end_time "Now"

Don't forget to report the end of the build itself!

./faros_event.sh CI -k "<faros_api_key>" \
    --run "<run_source>://<run_organization>/<run_pipeline>/<run_id>" \
    --run_status "Success" \
    --run_end_time "Now"

Reporting test execution results

Use this event type if you would like to analyze success/failure rates and execution times of Test Suites

❗ --full flag must be provided with TestExecution event

This event reports a successful test suite invocation:

./faros_event.sh TestExecution -k "<faros_api_key>" \
    --commit "<commit_source>://<commit_organization>/<commit_repository>/<commit_sha>" \
    --test_id "<test_id>" \
    --test_source "<test_source>" \
    --test_type "Functional" \
    --test_status "Success" \
    --test_suite "<test_suite_id>" \
    --test_stats "failure=0,success=18,skipped=3,unknown=0,custom=2,total=23" \
    --test_start_time "2021-07-20T18:05:46.019Z" \
    --test_end_time "2021-07-20T18:08:42.024Z" \
    --full

Reporting deployments

Send CD events to the Faros platform if you would like to analyze your deploy frequency and lead time metrics.

Option 1: If information about the specific commit that is being deployed is available at the time of deployment, use this CD event to report the successful deployment of an application to an environment:

./faros_event.sh CD -k "<faros_api_key>" \
    --commit "<commit_source>://<commit_organization>/<commit_repository>/<commit_sha>" \
    --deploy "<deploy_source>://<deploy_application>/<deploy_environment>/<deploy_id>" \
    --deploy_status "Success" \
    --deploy_start_time "2021-07-20T18:05:46.019Z" \
    --deploy_end_time "2021-07-20T18:08:42.024Z"

Option 2: If commit information is not readily available at the time of deployment, but you do have artifact information, you can reference the artifact instead of the commit. In such a scenario, you must also spearately report CI events as described above, and the Faros Platform will do the work of figuring out what commit got deployed.

./faros_event.sh CD -k "<faros_api_key>" \
    --artifact "<artifact_source>://<artifact_organization>/<artifact_repository>/<artifact_id>" \
    --deploy "<deploy_source>://<deploy_application>/<deploy_environment>/<deploy_id>" \
    --deploy_status "Success" \
    --deploy_start_time "2021-07-20T18:05:46.019Z" \
    --deploy_end_time "2021-07-20T18:08:42.024Z"

❗ If choosing Option 2 to report your deployment events, the --artifact parameter in the CD event should exactly match the artifact parameter in the CI event.

❗ The deploy_status is an enum. Read the documentation on arguments here for accepted values.

❗ The deploy_environment is also an enum. Read the documentation on arguments here for accepted values.

Arguments

Passing arguments: flags or environment variables

There are two ways that arguments can be passed into the script. The first, is via flags. The second is via environment variables. You may use a combination of these two options. If both are set, flags will take precedence over environment variables.

πŸ“ Note: By convention, you can switch between using a flag or an environment variable by simply capitalizing the argument name and prefixing it with FAROS_. For example, --commit becomes FAROS_COMMIT, --artifact becomes FAROS_ARTIFACT.

General arguments

Argument Description Required Default
-k, --api_key Your Faros API key. See the documentation for more information on obtaining an api key. Yes (not required when --community_edition flag is present)
-u, --url The Faros API url to send the event to. https://prod.api.faros.ai (http://localhost:8080 for Faros Community Edition)
-g, --graph The graph(s) that the event should be sent to. If specifying more than one graph, they should be provided as a comma separated array (e.g. graph_1,graph_2) "default"
--validate_only Event will not be consumed but instead will only be validated against event schema.
--dry_run Print the event instead of sending.
--community_edition Events will be formatted and sent to Faros Community Edition.

CI arguments

Argument Description Dependency
--run The URI of the job run that built the code. (<source>://<organization>/<pipeline>/<run_id>)
--run_status The status of the job run that built the code. (Allowed values: Success, Failed, Canceled, Queued, Running, Unknown, Custom) --run
--run_status_details Any extra details about the status of the job run. --run
--run_start_time The start time of the job run in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --run
--run_end_time The end time of the job run in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --run
--run_name The name of the job run that built the code. --run
--commit The URI of the commit. (<commit_source>://<commit_organization>/<commit_repository>/<commit_sha>) --run
--artifact The URI of the artifact. (<source>://<organization>/<repository>/<artifact_id>) --commit
--pull_request_number The pull request number of the commit. (e.g. 123). --commit
--run_step_id The id of the job run step. (e.g. 123). --run
--run_step_name The name of the job run step (e.g. Lint). --run
--run_step_status The status of the job run step. (Allowed values: Success, Failed, Canceled, Queued, Running, Unknown, Custom) --run
--run_step_status_details Any extra details about the status of the job run step. --run
--run_step_type The type of the job run step. (Allowed values: Script, Manual, Custom) --run
--run_step_type_details Any extra details about the type of the job run step. --run
--run_step_start_time The start time of the job run step in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --run
--run_step_end_time The end time of the job run step in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --run
--run_step_command The command executed by the job run step. --run
--run_step_url The url to the job run step. --run

CD arguments

Argument Description Dependency
--deploy The URI of the deployment. (<deploy_source>://<application>/<environment>/<deploy_id>) (<environment> allowed values: Prod, Staging, QA, Dev, Sandbox, Canary, Custom)
--deploy_start_time The start time of the deployment in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --deploy
--deploy_end_time The end time of the deployment in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --deploy
--deploy_requested_at The time the deployment was requested in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --deploy
--deploy_status The status of the deployment. (Allowed values: Success, Failed, Canceled, Queued, Running, RolledBack, Custom) --deploy
--deploy_status_details Any extra details about the status of the deployment. --deploy
--deploy_url The url of the deployment. --deploy
--deploy_app_platform The compute platform that runs the application. --deploy
--deploy_app_tags A comma separated array of key:value application tags. (e.g. key1:value1,key2:value2) --deploy
--deploy_app_paths A comma separated array of application slash separated paths. (e.g. aws/us-east-1/eks-001,aws/us-west-2/eks-002) --deploy
--deploy_env_details Any extra details about the deployment environment. --deploy
--commit The URI of the commit. If you specify --artifact in your CI events, you should use --artifact in your CD events. Otherwise, use --commit. (<source>://<organization>/<repository>/<commit_sha>) --deploy
--artifact The URI of the artifact. If you specify --artifact in your CI events, you should use --artifact in your CD events. Otherwise, use --commit. (<artifact_source>://<artifact_organization>/<artifact_repository>/<artifact_id>) --deploy
--pull_request_number The pull request number of the commit. (e.g. 123). Used only if --commit is included --commit
--run The URI of the job run executing the deployment. (<source>://<organization>/<pipeline>/<run_id> e.g. Jenkins://faros-ai/my-pipeline/1234)
--run_status The status of the job run executing the deployment. (Allowed values: Success, Failed, Canceled, Queued, Running, Unknown, Custom) --run
--run_status_details Any extra details about the status of the job run executing the deployment. --run
--run_start_time The start time of the job run in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --run
--run_end_time The end time of the job run in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --run

Test Execution arguments

Argument Description Required
--commit The URI of the commit. (<source>://<organization>/<repository>/<commit_sha> e.g. GitHub://faros-ai/my-repo/da500aa4f54cbf8f3eb47a1dc2c136715c9197b9) Yes
--pull_request_number The pull request number of the commit. (e.g. 123).
--test_id The unique identifier of the test within the test source system. Yes
--test_source The test source system. (e.g. Jenkins) Yes
--test_type The type of the test that was executed: (Allowed values: Functional, Integration, Manual, Performance, Regression, Security, Unit, Custom) Yes
--test_type_details Additional details about the type of the test that was executed.
--test_status The outcome status of the test execution. (Allowed values: Success, Failure, Custom, Skipped, Unknown) Yes
--test_status_details Additional details about the status of the outcome status of the test.
--test_suite The name of the test suite. Yes
--test_stats The stats of the test outcome as a string of comma separated key=value pairs. (e.g. failure=0,success=18,skipped=3,unknown=0,custom=2,total=23)
--test_start_time The start time of the test in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z)
--test_end_time The end time of the test in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z)
--test_tags A string of comma separated tags to associate with the test. (e.g. tag1,tag2)
--environments A string of comma separated environments to associate with the test. (e.g. env1,env2)
--device_name The name of the device on which the test was executed. (e.g. MacBook)
--device_os The operating system of the device on which the test was executed. (e.g. OSX)
--device_browser The browser on which the test was executed. (e.g. Chrome)
--device_type The type of the device on which the test was executed.
--task_source The type of TMS (Task Management System). (e.g. Jira, Shortcut, GitHub) If --test_task, --defect_task, --test_suite_task, or --test_execution_task provided
--test_task A comma separated array of one or many unique identifiers of test tasks within the TMS (Task Management System). The outcome of a specific test for this execution can be provided as a key=value pair (e.g. TEST-123=Success,TEST-456=Failure with allowed statuses: Success, Failure, Custom, Skipped, Unknown )
--defect_task The unique identifier of the defect task within the TMS (Task Management System).
--test_suite_task The unique identifier of the test suite task within the TMS (Task Management System).
--test_execution_task The unique identifier of the test execution task within the TMS (Task Management System).
--run The URI of the job run executing the test. (<source>://<organization>/<pipeline>/<run_id> e.g. Jenkins://faros-ai/my-pipeline/1234)
--run_status The status of the job run executing the test. (Allowed values: Success, Failed, Canceled, Queued, Running, Unknown, Custom) --run
--run_status_details Any extra details about the status of the job run executing the test. --run
--run_start_time The start time of the job run in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --run
--run_end_time The end time of the job run in milliseconds since the epoch, ISO-8601, or Now. (e.g. 1626804346019, 2021-07-20T18:05:46.019Z) --run

URI arguments alternative

Sometimes using the URI format required by --run, --commit, --artifact, or --deploy arguments gets in the way. For example when your commit repository has a / in the name. Here is how you can supply the required information as individual fields instead. Each alternative requires all listed fields.

--run argument alternative (all listed fields are required)

Argument Description
--run_id The id of the run
--run_pipeline The pipeline of the run
--run_org The organization of the run
--run_source The source system of the run

--deploy argument alternative (all listed fields are required)

Argument Description
--deploy_id The id of the deployment
--deploy_env The environment of the deployment (allowed values: Prod, Staging, QA, Dev, Sandbox, Canary, Custom)
--deploy_app The application being deployed
--deploy_source The source system of the deployment

--commit argument alternative (all listed fields are required)

Argument Description
--commit_sha The SHA of the commit
--commit_repo The repository of the commit
--commit_org The organization of the commit
--commit_source The source system storing the commit

--artifact argument alternative (all listed fields are required)

Argument Description
--artifact_id The id of the artifact
--artifact_repo The repository of the artifact
--artifact_org The organization of the artifact
--artifact_source The source system storing the artifact

Additional arguments

Argument Description Default
--origin The origin of the event that is being sent to Faros. "Faros_Script_Event"
--full The event being sent should be validated as a full event.
--silent Unexceptional output will be silenced.
--debug Helpful information will be printed.
--skip_saving_run Do not include cicd_Build in the event.
--no_lowercase_vcs Do not lowercase commit_organization and commit_repo.
--hasura_admin_secret The Hasura Admin Secret. Only used with ‑‑community_edition flag. "admin"
--max_time The time in seconds allowed for each retry attempt. 10
--retry The number of allowed retry attempts. 3
--retry_delay The delay in seconds between each retry attempt. 1
--retry_max_time The total time in seconds the request with retries can take. 40

Tips

Validating your command

As you are iterating on instrumentation you can use the --validate_only flag to test before you are ready to send actual data:

./faros_event.sh <...your command arguments...> --validate_only

Usage with Faros Community Edition

❗ Sending events in parts is not currently supported ❗ Build steps in CI events are not currently supported ❗ Test Execution events are not currently supported

When using Faros Community Edition, you can use the tool in exactly the same way as described above. Just include the --community_edition flag. The Faros API key is not needed, since the tool will call your locally deployed Hasura to perform mutations derived from the events. See the Faros Community Edition repo for more details.

./faros_event.sh <...your command arguments...> --community_edition

πŸ”¨ Development

βœ… Testing & Checking for Bugs

We use ShellSpec to test our scripts and ShellCheck to check for potential bugs.

Install using Homebrew

brew tap shellspec/shellspec
brew install shellspec shellcheck

Running the tests

Move to the /test directory and execute shellspec

cd test && shellspec

Checking for bugs

Go to root directory and execute:

shellcheck -s bash faros_event.sh