[IBCDPE] Split out and start creating items as individual modules

[BryanFauble]

Problem:

1. Spotocean controller V2 is not ready, use are using it wrong, or the documentation is incorrect.
2. We need to start thinking about how we'll organize both modules going into stacks, and the spacelift stacks themselves.

Solution:

1. Revert to Spotocean controller v1
2. Start splitting out the spotocean autoscaler to it's own module
3. Split out airflow into it's own module
4. Start to create some of the scaffolding for programmatically creating spacelift resources like policy, stack, and policy attachment

Testing:

1. Will be verifying that I am able to register these modules in the spacelift terraform registry
2. Will verify that the spot ocean autoscaler correct deploys

[thomasyu888]

🔥 Thanks for working on this - just going to pre-approve, I just have minor comments.

[thomasyu888]

This is nice - thanks for adding this!

[thomasyu888]

nit: should these be named sage instead of bfauble?

[BryanFauble]

When we tear down the existing spacelift stacks and move to a totally programmatic approach yes. I named the first stack 'bfauble', and even if you update the display name the ID of the stack is sticky and does not change.

[thomasyu888]

What is this for?

[BryanFauble]

I left this in as scaffolding for how I am thinking we'll organize our spacelift resources. Basically we need:

1. Modules that define what we can create
2. Another set of resources that combine those modules in a meaningful way (ie: This is what I am intending this dpe-prod/main.tf to do. This is where we create the individual "stacks" we'll be deploying out.
3. Another set of spacelift specific resources (Like policies, stacks, contexts) and their relationships. When a spacelift stack is created it requires that you are pointing at a specific VCS location where it's pulling terraform resources from. That VCS location is defining the resources that "this" spacelift stack i going to deploy.

[thomasyu888]

Do you have a list of questions you'd like spot to answer?

[BryanFauble]

1. What terraform modules are required to install the V2 spot ocean components into our EKS cluster?
2. What dependencies need to be completed, and operational before attempting to install any spot ocean terraform resources?
3. What does the removal of ocean's terraform resources entail, is everything being properly reset back to the "before" state if removed?

[thomasyu888]

Are we currently using these modules?

[BryanFauble]

No. This is planned future work to start programmatically creating the entire stack within spacelift

[thomasyu888]

This is still something I'd like to figure out, spot claimed on the call that we didn't need a managed node group.

[BryanFauble]

So the problem is that in order for the add ons to be installed in the first place EKS has to have at least 1 node running as that is where the tools are installed to. This means that in order for the EKS cluster to be "green" it has to have at least a single node.

After this point we can install spot ocean.

Even their docs here: https://github.com/spotinst/terraform-spotinst-ocean-eks?tab=readme-ov-file#following-modules-should-be-used-instead--

Specify that we should:

1. Use the AWS EKS provider to create an EKS cluster and it's resources
2. Then install the ocean controllers/k8s