Skip to content

Commit

Permalink
Add OcpSandbox kind (#55)
Browse files Browse the repository at this point in the history 
The provided change is a set of modifications to add the OcpSandbox resource kind to the Sandbox API.

New types:
* `OcpSharedClusterConfiguration`: Shared cluster configuration (with virt or not) that is used by the sandbox API to schedule and create resources for OcpSandboxes.
* `OcpSandbox`: Basically a namespace + service account

**How sandboxes are scheduled**
1. use annotations to filter the cluster in the request.   OcpCluster are created with annotations. When requesting, provide `cloud_selector` to match the annotations of the desired cluster(s). Behind the scene that uses postgresql `@>` operator to filter the OcpClusters. Example of request in agnosticV:
    ```yaml
     __meta__:
       sandboxes:
         - kind: OcpSandbox
           cloud_selector:
             virt: enable
             region: na
    ```
2. look at resources (CPU/Memory) on the clusters before electing
3. If cluster has room, then create the resources (namespace, serviceaccount)


**Go Source Code Changes:**

In `account_handlers.go`, significant changes include the introduction of handling OpenShift (OCP) accounts along with AWS accounts. This includes creating a new handler for OCP accounts, adjusting existing functions to manage accounts by their kind (e.g., AWS or OCP), and logging enhancements.

Adjustments in error handling, HTTP status codes, and addition of comments for clarity.
In `handlers.go`, the creation of placements now supports OCP resources, error handling improvements, and changes to ensure resources are correctly cleaned up in case of errors during placement creation.

**Enhanced Annotations Handling:**

The handling of annotations within placement requests has been refined. This includes the implementation of a Merge function for annotations, allowing for the combination of annotations from different sources. This change ensures that annotations provided at different levels (e.g., placement request level vs. individual resource request level) are properly consolidated, enhancing flexibility and the ability to pass and utilize metadata throughout the system.

The AWS account management logic within internal/dynamodb/accounts.go has been updated to support annotations. This includes the ability to store and retrieve annotations associated with AWS accounts, allowing for richer metadata management associated with AWS resources. This enhancement supports more nuanced account management and allocation strategies, catering to specific needs or criteria defined via annotations.

For example, with this change,  in agnosticv it'll now be possible to do: 
```yaml
__meta__:
  sandboxes:
    - kind: OcpSandbox
      var: ocp_account
      annotations:
        purpose: webapp
    - kind: AwsSandbox
      var: aws_account
      annotations:
        purpose: storage
    - kind: AwsSandbox
      var: aws_account2
      annotations:
        purpose: automation
```
And in Anarchy use the annotations to request the different accounts and pass that information so the accounts can be identified when retrieving them

```json
{
  "service_uuid": "{{uuid}}",
  "resources": [
    {"kind": "OcpSandbox", "annotations":{"var":"ocp_account", "purpose": "webapp"}},
    {"kind": "AwsSandbox", "annotations":{"var":"aws_account", "purpose": "storage"}},
    {"kind": "AwsSandbox", "annotations":{"var":"aws_account2", "purpose": "automation"}},
  ],
  "annotations": {
    "guid": "...",
    "owner": "...",
  }
}
```
That is done in PR rhpds/babylon_anarchy_governor#83

**SQL Migration Scripts:**

The addition of `005_ocp_sandbox.up.sql` and `005_ocp_sandbox.down.sql` for managing OpenShift providers in the database. This includes creating a new table ocp_providers and updating the resources table to accommodate OCP-specific data.

**Swagger API Documentation (swagger.yaml):**

Updated the API documentation to reflect new endpoints and parameters related to the handling of different kinds of sandbox accounts, specifically the inclusion of OCP alongside AWS.

**Dependency and Module Changes (go.mod, go.sum):**

Updated various dependencies, including the Kubernetes client libraries (k8s.io/*) to newer versions. These updates support the handling of OCP resources.
General updates to dependencies and removal of unused ones.

Overall, these changes aim to expand the project's capabilities by introducing support for OpenShift accounts alongside AWS accounts, improving error handling and logging, and updating dependencies to support new features and ensure compatibility.

**Makefile Changes:** Introduced a new section in the migrate target to print the database URL without exposing the password, and ask for confirmation before proceeding.

Co-authored-by: Guillaume Core <gucore@redhat.com>
  • Loading branch information
@agonzalezrh @fridim
agonzalezrh and fridim authored Apr 5, 2024
1 parent c2e5307 commit 901aec9
Show file tree
Hide file tree
Showing 28 changed files with 3,434 additions and 289 deletions.
46 changes: 46 additions & 0 deletions .air.toml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
root = "."
testdata_dir = "testdata"
tmp_dir = "tmp"

[build]
args_bin = []
bin = "./build/sandbox-api"
cmd = "make sandbox-api"
delay = 1000
exclude_dir = ["assets", "tmp", "vendor", "testdata"]
exclude_file = []
exclude_regex = ["_test.go"]
exclude_unchanged = false
follow_symlink = false
full_bin = ""
include_dir = []
include_ext = ["go", "tpl", "tmpl", "html"]
include_file = []
kill_delay = "0s"
log = "build-errors.log"
poll = false
poll_interval = 0
post_cmd = []
pre_cmd = []
rerun = false
rerun_delay = 500
send_interrupt = false
stop_on_error = false

[color]
app = ""
build = "yellow"
main = "magenta"
runner = "green"
watcher = "cyan"

[log]
main_only = false
time = false

[misc]
clean_on_exit = false

[screen]
clear_on_rebuild = false
keep_scroll = true
20 changes: 14 additions & 6 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ VERSION ?= $(shell git describe --tags 2>/dev/null | cut -c 2-)
COMMIT ?= $(shell git rev-parse HEAD 2>/dev/null)
DATE ?= $(shell date -u)

build: sandbox-list sandbox-metrics sandbox-api sandbox-issue-jwt sandbox-dynamodb-rotate-vault
build: sandbox-list sandbox-metrics sandbox-api sandbox-issue-jwt sandbox-rotate-vault

test:
@echo "Running tests..."
Expand All @@ -26,13 +26,21 @@ rm-local-pg:
run-local-pg: .dev.pg_password rm-local-pg
@echo "Running local postgres..."
@podman run -p 5432:5432 --name localpg -e POSTGRES_PASSWORD=$(shell cat .dev.pg_password) -d postgres:16-bullseye
# See full list of parameters here:
# https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS
# See full list of parameters here:
# https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS

issue-jwt: .dev.jwtauth_env
@. ./.dev.pgenv && . ./.dev.jwtauth_env && go run ./cmd/sandbox-issue-jwt

migrate: .dev.pgenv
# Print a message with the database URL and ask for confirmation
# Remove password from the URL before printing
@. ./.dev.pgenv && echo "Database URL: $$(echo $${DATABASE_URL} | sed -E 's/:[^@]+@/:<password>@/g')"
@read -p "Are you sure [y/n]? " -n 1 -r; \
if [[ ! $$REPLY =~ ^[Yy]$$ ]]; then \
echo "Aborting."; \
exit 1; \
fi
@echo "Running migrations..."
@. ./.dev.pgenv && migrate -database "$${DATABASE_URL}" -path db/migrations up

Expand All @@ -55,8 +63,8 @@ sandbox-issue-jwt:
sandbox-replicate:
CGO_ENABLED=0 go build -o build/sandbox-replicate ./cmd/sandbox-replicate

sandbox-dynamodb-rotate-vault:
CGO_ENABLED=0 go build -ldflags="-X 'main.Version=$(VERSION)' -X 'main.buildTime=$(DATE)' -X 'main.buildCommit=$(COMMIT)'" -o build/sandbox-dynamodb-rotate-vault ./cmd/sandbox-dynamodb-rotate-vault
sandbox-rotate-vault:
CGO_ENABLED=0 go build -ldflags="-X 'main.Version=$(VERSION)' -X 'main.buildTime=$(DATE)' -X 'main.buildCommit=$(COMMIT)'" -o build/sandbox-rotate-vault ./cmd/sandbox-rotate-vault


push-lambda: deploy/lambda/sandbox-replicate.zip
Expand All @@ -65,7 +73,7 @@ push-lambda: deploy/lambda/sandbox-replicate.zip
fmt:
@go fmt ./...

.PHONY: sandbox-api sandbox-issue-jwt sandbox-list sandbox-metrics sandbox-dynamodb-rotate-vault run-api sandbox-replicate migrate fixtures test run-local-pg push-lambda clean fmt
.PHONY: sandbox-api sandbox-issue-jwt sandbox-list sandbox-metrics sandbox-rotate-vault run-api sandbox-replicate migrate fixtures test run-local-pg push-lambda clean fmt

clean: rm-local-pg
rm -f build/sandbox-*
Expand Down
155 changes: 104 additions & 51 deletions cmd/sandbox-api/account_handlers.go
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
package main

import (
"context"
"encoding/json"
"fmt"
"net/http"
"context"

"github.com/jackc/pgx/v4"
"github.com/rhpds/sandbox/internal/api/v1"
Expand All @@ -18,38 +18,74 @@ import (

type AccountHandler struct {
awsAccountProvider models.AwsAccountProvider
OcpSandboxProvider models.OcpSandboxProvider
}

func NewAccountHandler(awsAccountProvider models.AwsAccountProvider) *AccountHandler {
func NewAccountHandler(awsAccountProvider models.AwsAccountProvider, OcpSandboxProvider models.OcpSandboxProvider) *AccountHandler {
return &AccountHandler{
awsAccountProvider: awsAccountProvider,
OcpSandboxProvider: OcpSandboxProvider,
}
}

// GetAccountsHandler returns all accounts
// GET /accounts
// GetAccountsHandler returns all accounts by kind
// GET /accounts/{kind}
func (h *AccountHandler) GetAccountsHandler(w http.ResponseWriter, r *http.Request) {
enc := json.NewEncoder(w)
enc.SetIndent("", " ")

kind := chi.URLParam(r, "kind")
serviceUuid := r.URL.Query().Get("service_uuid")

// Get available from Query
available := r.URL.Query().Get("available")

var (
accounts []models.AwsAccount
err error
)
if serviceUuid != "" {
// Get the account from DynamoDB
accounts, err = h.awsAccountProvider.FetchAllByServiceUuid(serviceUuid)

} else {
var err error
var accountlist []interface{}
switch kind {
case "AwsSandbox", "aws":
var (
accounts []models.AwsAccount
)
if serviceUuid != "" {
// Get the account from DynamoDB
accounts, err = h.awsAccountProvider.FetchAllByServiceUuid(serviceUuid)
} else {
if available != "" && available == "true" {
accounts, err = h.awsAccountProvider.FetchAllAvailable()
} else {
accounts, err = h.awsAccountProvider.FetchAll()
}
}
accountlist = make([]interface{}, len(accounts))
for i, acc := range accounts {
accountlist[i] = acc
}
case "OcpSandbox", "ocp":
var (
accounts []models.OcpSandbox
)
if available != "" && available == "true" {
accounts, err = h.awsAccountProvider.FetchAllAvailable()
// Account are created on the fly, so this request doesn't make sense
// for OcpSandboxes
// Return bad request
w.WriteHeader(http.StatusBadRequest)
enc.Encode(v1.Error{
HTTPStatusCode: http.StatusBadRequest,
Message: "Bad request, Ocp Account are created on the fly",
})
return
}
if serviceUuid != "" {
// Get the account from DynamoDB
accounts, err = h.OcpSandboxProvider.FetchAllByServiceUuid(serviceUuid)
} else {
accounts, err = h.awsAccountProvider.FetchAll()
accounts, err = h.OcpSandboxProvider.FetchAll()
}

accountlist = make([]interface{}, len(accounts))
for i, acc := range accounts {
accountlist[i] = acc
}
}

Expand All @@ -63,15 +99,14 @@ func (h *AccountHandler) GetAccountsHandler(w http.ResponseWriter, r *http.Reque
})
return
}

if len(accounts) == 0 {
if len(accountlist) == 0 {
w.WriteHeader(http.StatusNotFound)
} else {
w.WriteHeader(http.StatusOK)
}

// Print accounts using JSON
if err := enc.Encode(accounts); err != nil {
if err := enc.Encode(accountlist); err != nil {
log.Logger.Error("GET accounts", "error", err)
w.WriteHeader(http.StatusInternalServerError)
enc.Encode(v1.Error{
Expand All @@ -84,47 +119,66 @@ func (h *AccountHandler) GetAccountsHandler(w http.ResponseWriter, r *http.Reque
// GetAccountHandler returns an account
// GET /accounts/{kind}/{account}
func (h *AccountHandler) GetAccountHandler(w http.ResponseWriter, r *http.Request) {
enc := json.NewEncoder(w)
enc.SetIndent("", " ")

// Grab the parameters from Params
accountName := chi.URLParam(r, "account")
kind := chi.URLParam(r, "kind")

// We don't need 'kind' param for now as it is checked and validated
// by the swagger openAPI spec.
switch kind {
case "AwsSandbox", "aws":

// Get the account from DynamoDB
sandbox, err := h.awsAccountProvider.FetchByName(accountName)
if err != nil {
if err == models.ErrAccountNotFound {
log.Logger.Warn("GET account", "error", err)
w.WriteHeader(http.StatusNotFound)
enc.Encode(v1.Error{
HTTPStatusCode: http.StatusNotFound,
Message: "Account not found",
// Get the account from DynamoDB
sandbox, err := h.awsAccountProvider.FetchByName(accountName)
if err != nil {
if err == models.ErrAccountNotFound {
log.Logger.Warn("GET account", "error", err)
w.WriteHeader(http.StatusNotFound)
render.Render(w, r, &v1.Error{
HTTPStatusCode: http.StatusNotFound,
Message: "Account not found",
})
return
}
log.Logger.Error("GET account", "error", err)

w.WriteHeader(http.StatusInternalServerError)
render.Render(w, r, &v1.Error{
HTTPStatusCode: 500,
Message: "Error reading account",
})
return
}
log.Logger.Error("GET account", "error", err)
// Print account using JSON
w.WriteHeader(http.StatusOK)
render.Render(w, r, &sandbox)
return
case "OcpSandbox", "ocp":
// Get the account from DynamoDB
sandbox, err := h.OcpSandboxProvider.FetchByName(accountName)
if err != nil {
if err == models.ErrAccountNotFound {
log.Logger.Warn("GET account", "error", err)
w.WriteHeader(http.StatusNotFound)
render.Render(w, r, &v1.Error{
HTTPStatusCode: http.StatusNotFound,
Message: "Account not found",
})
return
}
log.Logger.Error("GET account", "error", err)

w.WriteHeader(http.StatusInternalServerError)
enc.Encode(v1.Error{
HTTPStatusCode: 500,
Message: "Error reading account",
})
w.WriteHeader(http.StatusInternalServerError)
render.Render(w, r, &v1.Error{
HTTPStatusCode: 500,
Message: "Error reading account",
})
return
}
// Print account using JSON
w.WriteHeader(http.StatusOK)
render.Render(w, r, &sandbox)
return
}
// Print account using JSON
if err := enc.Encode(sandbox); err != nil {
log.Logger.Error("GET account", "error", err)
w.WriteHeader(http.StatusInternalServerError)
enc.Encode(v1.Error{
HTTPStatusCode: 500,
Message: "Error reading account",
})
}
}

func (h *AccountHandler) CleanupAccountHandler(w http.ResponseWriter, r *http.Request) {
// Grab the parameters from Params
accountName := chi.URLParam(r, "account")
Expand Down Expand Up @@ -355,7 +409,7 @@ func (h *BaseHandler) DeleteAccountHandler(w http.ResponseWriter, r *http.Reques
w.WriteHeader(http.StatusConflict)
render.Render(w, r, &v1.Error{
HTTPStatusCode: http.StatusConflict,
Message: "Cleanup must be attempted at least 3 times before deletion",
Message: "Cleanup must be attempted at least 3 times before deletion",
})
return
}
Expand All @@ -364,12 +418,11 @@ func (h *BaseHandler) DeleteAccountHandler(w http.ResponseWriter, r *http.Reques
w.WriteHeader(http.StatusConflict)
render.Render(w, r, &v1.Error{
HTTPStatusCode: http.StatusConflict,
Message: "Cleanup is in progress",
Message: "Cleanup is in progress",
})
return
}


// Close the AWS account using CloseAccount
if err := sandbox.CloseAccount(); err != nil {
log.Logger.Error("Error closing account", "error", err)
Expand Down
Loading

0 comments on commit 901aec9

Please sign in to comment.