[IBCDPE-1007] Templatizing items out to a root level

[BryanFauble]

Problem:

1. As noted by @BWMac in:

• [IBCDPE-1007] Monitoring and security scanning #14 (comment)
• [IBCDPE-1007] Monitoring and security scanning #14 (comment)
• [IBCDPE-1007] Monitoring and security scanning #14 (comment)
• [IBCDPE-935] VPC Updates & VPC CNI Exploration #13 (comment)

1. There are a bunch of areas that are not templatized, and are using static values that we had set up for development.

Solution:

1. Setting up more variable and environment value passing between the stacks. Allowing us to define all of the levers within the root main.tf and deployments/main.tf
2. Removing the dev directory and instead allowing us to create different dev/prod stacks from the same terraform scripts depending on how we are defining it's usage in deployments/main.tf

Testing:

1. Tested the deployment of the dev stack and verified that it came up healthy.

[BryanFauble]

Swapping over to this is to resolve a race condition issue. Since the VMServiceScrape kind is a resource that depends on a CRD (Custom Resource Definition) being installed already on the cluster the terraform plan stage was failing with:

│ Error: API did not recognize GroupVersionKind from manifest (CRD may not be installed)
│ 
│   with module.trivy-operator.kubernetes_manifest.vmservicescrape,
│   on .terraform/modules/trivy-operator/main.tf line 20, in resource "kubernetes_manifest" "vmservicescrape":
│   20: resource "kubernetes_manifest" "vmservicescrape" {
│ 
│ no matches for kind "VMServiceScrape" in group
│ "operator.victoriametrics.com"

The suggestion was to try out a different provider (https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/kubectl_manifest) to get around this issue. This allows the resources to be created as expected in the right order.

[BWMac]

Interesting that this resource has you pass a whole YAML configuration to the yaml_body arg instead of something more similar to how kubernetes_manifest is configured

[BWMac]

Is pinning this tag better than using the default as was done before?

[BryanFauble]

There was a bug that I had reported around this. Basically a '+' sign that was getting included in the tag image that prevented the cleanup job from running. I updated the helm chart to the newer version so we don't need to set this tag manually anymore. I will remove