Figure out orchestration of capepy deployment and updating

[mehalter]

Currently we have the capepy library built and included in the infrastructure source code. This makes it hard to rapidly iterate and improve capepy. We should sit down and figure out how we are going to do deployments decoupled from the infrastructure itself but have the infrastructure go and use the existing packages.

Things to consider:

• Lambda Layers can be considered "persistent" even across deployment/destruction of the infrastructure. We could add part of the deployment process in capepy to deploy a new Lambda Layer outside of this
  • Lambda layers are versioned and would allow us to decide in the infrastructure code which version to use at the moment
• capepy follows semantic versioning so we should ideally allow new features and bug fixes automatically during deployment but require manual modification for major release versions.
• It is nice that the current implementation guarantees that Glue jobs and Lambda functions all get the same capepy version. I wonder if this is a good idea or not given capepy will evolve and the code running might be from different times. This will only be a problem with breaking releases (major releases)

Possible plan:

• Make capepy version configurable in the meta configuration table
• capepy is in PyPI now so AWS Glue could just pull from there rather than shipping the wheel file
• capepy could publish it's own Lambda Layers during the release phase of the repo and keep them versioned
• when building the infrastructure it looks for the latest lambda layer for the required version and glue pulls the capepy from PyPI