chore: prepare deployment, update README.md, sanitise application of …

…keys (#42) * feat(deployment): terraform, workflow, kube * feat(deployment): change terraform config * feat(terraform-rds): init rds files for terraform * chore(frontend): clean up * feat(frontend): deploy * fix(client): added validation logic to flashcards * fix(workflow): removed branch * fix(client): added null checks to cards * feat(terraform): deployment of resources done * fix(client): fixed the ui for flashcard and MCQ at the generation page * feat(marketplace): added tooltip * fix(viewmcq): enhanced UI design * feat(workflow): created workflows for each service * fix(kong): reverted dockerfile * fix(mcq): fixed issue where only 1 MCQ was being generated by gpt * fix(client): minor qol fixes * fix(mcq): added additonal fixes * fix(profile): fixed cards * fix(subscribe): added blue box * feat(MCQ): added option to reorder, delete or add options * Feat/build push (#34) * feat(deployment): build push simple services * feat(workflow): created workflows for each service * feat(deployment): build push simple services * feat(deployment): kube files for all simple services * Feat/deployment (#28) * fix(mcq): fixed issue where only 1 MCQ was being generated by gpt * fix(client): minor qol fixes * fix(mcq): added additonal fixes * fix(profile): fixed cards * fix(subscribe): added blue box * feat(MCQ): added option to reorder, delete or add options --------- Co-authored-by: neilscallywag <neil.sharma.2022@scis.smu.edu.sg> Co-authored-by: Louis <64778758+iamlouisteo@users.noreply.github.com> * chore(upload-note): added env file * fix(docker): fixed docker file to add env file for upload note * chore(deployment): build push complex service images * feat(kubernetes): finished manifest * chore(kong): build and push kong image * chore(kong): added health check path * fix(user-storage): rebuild image * fix(subscriptions): rebuild image * fix(subscriptions): rebuild image * fix(payment): rebuild image * fix(contents): rebuild image * fix(notes & fileprocessor): rebuild image * fix(deployment): rebuild image * fix(handletempcontent): rebuild image * fix(makepayment & verifyuser): rebuild image * fix(kong): rebuild image * fix(userstorage & kong): temp fix * fix(contents): fix redis connection * fix(process-chunks): fix connection to aws mq * fix(deploy): fix connection * fix * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(upload-notes): fixed rabbit connection * fix(contents): fix redis connection * feat(terraform): resources creation done * feat(kubernetes): done * fix(user-storage): singass auth use random email * fix(user-storage): singass auth use random email --------- Co-authored-by: neilscallywag <neil.sharma.2022@scis.smu.edu.sg> Co-authored-by: Louis <64778758+iamlouisteo@users.noreply.github.com> * chore(frontend): adjust colours, remove unused methods * chore(upload-notes): fix concurrent uploads * 30 prod generated content can be viewed and edited by anyone with the redis note (#35) * fix(view-tc): no longer allows users to view other users temporary content * fix(docker-compose): uncommented ngrok * chore(deploy): build push new images * chore(deploy): build push new images * chore(deploy): build push new images * chore(deploy): build push new images * chore(deploy): build push new images * chore(deploy): build push new images * fix(kubernetes): fixed deployment for process chunks * feat(argocd): created application yaml files for all services * chore(process-chunks): add retry logic for connecting to temporary queue * chore(process-chunks): add retry logic for connecting to temporary queue (#36) * fix(view-notes): update image * chore(upload-notes): add endpoint for view authz * chore: remove non-ready notes from view, fix created notes pagination * chore: fix some stuff hehe (#38) * chore(process-chunks): add retry logic for connecting to temporary queue * chore(upload-notes): add endpoint for view authz * chore: remove non-ready notes from view, fix created notes pagination * fix(htc): rebuild fixed * feat(pdf-reader): added text stemming and lemmatisation to reduce token count * fix(pdf-reader): cleaned up dead code from dockerfile * chore(frontend): tweak colour schemes, intuitiveness of navbar * fix(frontend): remove duplicate identifier * chore(workflow): test build push and update value * Chore/build push (#41) * chore: lemmenisation, frontend tweaks (#39) * chore(process-chunks): add retry logic for connecting to temporary queue * chore(upload-notes): add endpoint for view authz * chore: remove non-ready notes from view, fix created notes pagination * feat(pdf-reader): added text stemming and lemmatisation to reduce token count * fix(pdf-reader): cleaned up dead code from dockerfile * chore(frontend): tweak colour schemes, intuitiveness of navbar * fix(frontend): remove duplicate identifier --------- Co-authored-by: neilscallywag <neil.sharma.2022@scis.smu.edu.sg> Co-authored-by: SONG JIHOON <69960711+EchoSkorJjj@users.noreply.github.com> * Feat/deployment (#40) * chore(process-chunks): add retry logic for connecting to temporary queue * chore(upload-notes): add endpoint for view authz * chore: remove non-ready notes from view, fix created notes pagination * feat(pdf-reader): added text stemming and lemmatisation to reduce token count * fix(pdf-reader): cleaned up dead code from dockerfile * chore(frontend): tweak colour schemes, intuitiveness of navbar * fix(frontend): remove duplicate identifier * chore(workflow): test build push and update value --------- Co-authored-by: ztdevelops <yuezhengting@gmail.com> Co-authored-by: neilscallywag <neil.sharma.2022@scis.smu.edu.sg> * chore(workflow): test build push and update value * chore(workflow): test build push and update value * chore(workflow): test build push and update value * chore(workflow): test build push and update value * chore(cicd): test full cicd flow --------- Co-authored-by: Yue Zheng Ting <87757354+ztdevelops@users.noreply.github.com> Co-authored-by: neilscallywag <neil.sharma.2022@scis.smu.edu.sg> Co-authored-by: ztdevelops <yuezhengting@gmail.com> * chore(readme): update * Update README.md * Update README.md * fix(frontend): added unsplash client id * fix(frontend): added unsplash client id * fix(frontend): updated payment price * fix(frontend): fixed navigation * chore: update README.md * chore: update README.md * chore: update README.md * chore: update README.md * chore: replace keys with placeholder values --------- Co-authored-by: Jihoon_Song <69960711+EchoSkorJjj@users.noreply.github.com> Co-authored-by: neilscallywag <neil.sharma.2022@scis.smu.edu.sg> Co-authored-by: Louis <64778758+iamlouisteo@users.noreply.github.com> Co-authored-by: Neil <64523806+neilscallywag@users.noreply.github.com>
EchoSkorJjj · May 3, 2024 · e64e0a3 · e64e0a3
1 parent e31eed5
commit e64e0a3
Show file tree

Hide file tree

Showing 220 changed files with 7,200 additions and 6,761 deletions.
diff --git a/.github/workflows/contents.yml b/.github/workflows/contents.yml
@@ -0,0 +1,55 @@
+name: Build and Push Contents Service image to ECR
+
+on:
+  push:
+    branches:
+      - main
+      - chore/build-push
+    paths:
+      - 'backend/simple/contents/**'
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Check out the code
+      uses: actions/checkout@v3
+
+    - name: Configure AWS credentials
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        aws-region: ${{ secrets.AWS_REGION }}
+
+    - name: Login to Amazon ECR
+      run: |
+        aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v1
+
+    - name: Build and push Docker image to ECR
+      uses: docker/build-push-action@v2
+      with:
+        context: ./backend/simple/contents/
+        file: ./backend/simple/contents/Contents.Dockerfile
+        push: true
+        tags: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/${{ secrets.ECR_REPOSITORY }}:contents-${{ github.sha }}
+        platforms: linux/arm64
+
+    - name: Clone deployment repository
+      uses: actions/checkout@v3
+      with:
+        repository: ${{ secrets.DEPLOYMENT_REPO }}
+        token: ${{ secrets.DEPLOYMENT_REPO_TOKEN }}
+        path: 'deployment-repo'
+
+    - name: Update Kubernetes deployment image
+      run: |
+        cd deployment-repo
+        sed -i 's|image: .*|image: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/${{ secrets.ECR_REPOSITORY }}:contents-${{ github.sha }}|' ./kubernetes/simple/contents/contents-deployment.yaml
+        git config --local user.email "action@github.com"
+        git config --local user.name "GitHub Action"
+        git commit -am "Update image tag to ${{ github.sha }}"
+        git push
diff --git a/.github/workflows/dockerhub-image-pusher.yml b/.github/workflows/dockerhub-image-pusher.yml
@@ -3,7 +3,6 @@ on:
   push:
     branches:
       - main
-      - chore/cicd
 
 env:
   DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}

diff --git a/.github/workflows/fileprocessor.yml b/.github/workflows/fileprocessor.yml
@@ -0,0 +1,56 @@
+name: Build and Push File Processor Service image to ECR
+
+on:
+  push:
+    branches:
+      - main
+      - chore/build-push
+    paths:
+      - 'backend/simple/fileprocessor/**'
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Check out the code
+      uses: actions/checkout@v3
+
+    - name: Configure AWS credentials
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        aws-region: ${{ secrets.AWS_REGION }}
+
+    - name: Login to Amazon ECR
+      run: |
+        aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v1
+
+    - name: Build and push Docker image to ECR
+      uses: docker/build-push-action@v2
+      with:
+        context: ./backend/simple/fileprocessor/
+        file: ./backend/simple/fileprocessor/Fileprocessor.Dockerfile
+        push: true
+        tags: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/${{ secrets.ECR_REPOSITORY }}:fileprocessor-${{ github.sha }}
+        platforms: linux/arm64
+
+    - name: Clone deployment repository
+      uses: actions/checkout@v3
+      with:
+        repository: ${{ secrets.DEPLOYMENT_REPO }}
+        token: ${{ secrets.DEPLOYMENT_REPO_TOKEN }}
+        path: 'deployment-repo'
+
+    - name: Update Kubernetes deployment image
+      run: |
+        cd deployment-repo
+        sed -i 's|image: .*|image: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/${{ secrets.ECR_REPOSITORY }}:fileprocessor-${{ github.sha }}|' ./kubernetes/simple/fileprocessor/fileprocessor-deployment.yaml
+        git config --local user.email "action@github.com"
+        git config --local user.name "GitHub Action"
+        git commit -am "Update image tag to ${{ github.sha }}"
+        git push
+  
diff --git a/.github/workflows/frontend-client.yml b/.github/workflows/frontend-client.yml
@@ -0,0 +1,44 @@
+name: Build and Push Client Production Build to s3
+
+on:
+  push:
+    branches:
+      - main
+      - feat/deployment
+    paths:
+      - 'client/**'
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Check out the code
+      uses: actions/checkout@v3
+
+    - name: Configure AWS credentials
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        aws-region: ${{ secrets.AWS_REGION }}
+
+    - name: Install Node.js
+      uses: actions/setup-node@v2
+      with:
+        node-version: '18'
+
+    - name: Build the client
+      env:
+        VITE_STORAGE_KEY: ${{ secrets.VITE_STORAGE_KEY }}
+        VITE_API_BASE_URL: ${{ secrets.VITE_API_BASE_URL }}
+        VITE_API_PROD_BASE_URL: ${{ secrets.VITE_API_PROD_BASE_URL }}
+        VITE_GOOGLE_CLIENT_ID: ${{ secrets.VITE_GOOGLE_CLIENT_ID }}
+        VITE_NODE_ENV: ${{ secrets.VITE_NODE_ENV }}
+        VITE_UNSPLASH_CLIENT_ID: ${{ secrets.VITE_UNSPLASH_CLIENT_ID }}
+      run: |
+        cd client
+        npm install
+        npm run build
+  
+    - name: Sync dist folder to S3
+      run: aws s3 sync ./client/dist/ s3://${{ secrets.AWS_FRONTEND_S3_BUCKET }}/
diff --git a/.github/workflows/handle-temporary-contents.yml b/.github/workflows/handle-temporary-contents.yml
@@ -0,0 +1,55 @@
+name: Build and Push Handle Temporary Contents Service image to ECR
+
+on:
+  push:
+    branches:
+      - main
+      - chore/build-push
+    paths:
+      - 'backend/complex/handle-temporary-contents/**'
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Check out the code
+      uses: actions/checkout@v3
+
+    - name: Configure AWS credentials
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        aws-region: ${{ secrets.AWS_REGION }}
+
+    - name: Login to Amazon ECR
+      run: |
+        aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v1
+
+    - name: Build and push Docker image to ECR
+      uses: docker/build-push-action@v2
+      with:
+        context: ./backend/complex/handle-temporary-contents/
+        file: ./backend/complex/handle-temporary-contents/HTC.Dockerfile
+        push: true
+        tags: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/${{ secrets.ECR_REPOSITORY }}:handle-temporary-contents-${{ github.sha }}
+        platforms: linux/arm64
+
+    - name: Clone deployment repository
+      uses: actions/checkout@v3
+      with:
+        repository: ${{ secrets.DEPLOYMENT_REPO }}
+        token: ${{ secrets.DEPLOYMENT_REPO_TOKEN }}
+        path: 'deployment-repo'
+
+    - name: Update Kubernetes deployment image
+      run: |
+        cd deployment-repo
+        sed -i 's|image: .*|image: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/${{ secrets.ECR_REPOSITORY }}:handle-temporary-contents-${{ github.sha }}|' ./kubernetes/complex/handle-temporary-contents/handle-temporary-contents-deployment.yaml
+        git config --local user.email "action@github.com"
+        git config --local user.name "GitHub Action"
+        git commit -am "Update image tag to ${{ github.sha }}"
+        git push
diff --git a/.github/workflows/kong-gateway.yml b/.github/workflows/kong-gateway.yml
@@ -0,0 +1,55 @@
+name: Build and Push Kong Gateway image to ECR
+
+on:
+  push:
+    branches:
+      - main
+      - chore/build-push
+    paths:
+      - 'backend/kong-gateway/**'
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Check out the code
+      uses: actions/checkout@v3
+
+    - name: Configure AWS credentials
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        aws-region: ${{ secrets.AWS_REGION }}
+
+    - name: Login to Amazon ECR
+      run: |
+        aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v1
+
+    - name: Build and push Docker image to ECR
+      uses: docker/build-push-action@v2
+      with:
+        context: ./backend/kong-gateway/
+        file: ./backend/kong-gateway/Kong.Dockerfile
+        push: true
+        tags: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/${{ secrets.ECR_REPOSITORY }}:kong-gateway-${{ github.sha }}
+        platforms: linux/arm64
+
+    - name: Clone deployment repository
+      uses: actions/checkout@v3
+      with:
+        repository: ${{ secrets.DEPLOYMENT_REPO }}
+        token: ${{ secrets.DEPLOYMENT_REPO_TOKEN }}
+        path: 'deployment-repo'
+
+    - name: Update Kubernetes deployment image
+      run: |
+        cd deployment-repo
+        sed -i 's|image: .*|image: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/${{ secrets.ECR_REPOSITORY }}:kong-gateway-${{ github.sha }}|' ./kubernetes/kong-gateway/kong-gateway-deployment.yaml
+        git config --local user.email "action@github.com"
+        git config --local user.name "GitHub Action"
+        git commit -am "Update image tag to ${{ github.sha }}"
+        git push
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
diff --git a/.github/workflows/make-payment.yml b/.github/workflows/make-payment.yml
@@ -0,0 +1,55 @@
+name: Build and Push Make Payment Service image to ECR
+
+on:
+  push:
+    branches:
+      - main
+      - chore/build-push
+    paths:
+      - 'backend/complex/make-payment/**'
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Check out the code
+      uses: actions/checkout@v3
+
+    - name: Configure AWS credentials
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        aws-region: ${{ secrets.AWS_REGION }}
+
+    - name: Login to Amazon ECR
+      run: |
+        aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v1
+
+    - name: Build and push Docker image to ECR
+      uses: docker/build-push-action@v2
+      with:
+        context: ./backend/complex/make-payment/
+        file: ./backend/complex/make-payment/MakePayment.Dockerfile
+        push: true
+        tags: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/${{ secrets.ECR_REPOSITORY }}:make-payment-${{ github.sha }}
+        platforms: linux/arm64
+
+    - name: Clone deployment repository
+      uses: actions/checkout@v3
+      with:
+        repository: ${{ secrets.DEPLOYMENT_REPO }}
+        token: ${{ secrets.DEPLOYMENT_REPO_TOKEN }}
+        path: 'deployment-repo'
+
+    - name: Update Kubernetes deployment image
+      run: |
+        cd deployment-repo
+        sed -i 's|image: .*|image: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/${{ secrets.ECR_REPOSITORY }}:make-payment-${{ github.sha }}|' ./kubernetes/complex/make-payment/make-payment-deployment.yaml
+        git config --local user.email "action@github.com"
+        git config --local user.name "GitHub Action"
+        git commit -am "Update image tag to ${{ github.sha }}"
+        git push
-Original file line number
+Diff line change
@@ Expand Up / @@ -3,7 +3,6 @@ on: @@
       push:
         branches:
           - main
-          - chore/cicd
     env:
       DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
@@ Expand Down @@