Clean CICD template

vectornguyen76 · Dec 1, 2024 · 2f32e13 · 2f32e13
1 parent 898616a
commit 2f32e13
Show file tree

Hide file tree

Showing 6 changed files with 202 additions and 230 deletions.
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
@@ -1,161 +1,129 @@
-# CI/CD Deploy
+# CI/CD Pipeline Documentation
 
-## Set up
+This document describes the Continuous Integration and Continuous Deployment (CI/CD) pipeline setup for the project.
 
-### Create Secrets on Github
+## Overview
 
-1. **AWS**
+The project uses GitHub Actions for automated testing, building, and deployment with a blue-green deployment strategy. There are three main environments:
 
-   - AWS_ACCESS_KEY_ID: access token
-   - AWS_SECRET_ACCESS_KEY: secret access
-   - SSH_PRIVATE_KEY: ssh key pair
+- Development (develop branch)
+- Staging (staging branch)
+- Production (master branch)
 
-2. **Dockerhub**
+## Required Secrets and Variables
 
-   - DOCKERHUB_USERNAME: username
-   - DOCKERHUB_PASSWORD: password
+### GitHub Secrets
 
-3. **Chat Service**
+1. **AWS Credentials**
 
-   - OPENAI_API_KEY: your openai api key
+   - `AWS_ACCESS_KEY_ID`: AWS access key
+   - `AWS_SECRET_ACCESS_KEY`: AWS secret key
+   - `SSH_PRIVATE_KEY`: SSH key pair for EC2 access
 
-4. **Frontend**
+2. **Docker Hub**
 
-   - GOOGLE_CLIENT_ID: google client id
-   - GOOGLE_CLIENT_SECRET: google client password
-   - NEXTAUTH_SECRET: nextauth secret
+   - `DOCKERHUB_USERNAME`: Docker Hub username
+   - `DOCKERHUB_PASSWORD`: Docker Hub password
 
-5. **Backend**
-   _Config env email_
+3. **Frontend Authentication**
+   - `GOOGLE_CLIENT_ID`: Google OAuth client ID
+   - `GOOGLE_CLIENT_SECRET`: Google OAuth client secret
+   - `NEXTAUTH_SECRET`: NextAuth secret key
 
-   - MAIL_HOST
-   - MAIL_PORT
-   - MAIL_USER
-   - MAIL_PASS
-   - MAIL_SENDER
+### GitHub Variables
 
-   _Config env JWT_
-
-   - AT_SECRET
-   - RT_SECRET
-
-   _Database url_
-
-   - DATABASE_URL
-
-6. **Database**
-   - POSTGRES_USER: user name
-   - POSTGRES_PASSWORD: password
-   - POSTGRES_DB: database name
-
-### Create Variables on Github
-
-1. **AWS**
-
-   - TAGS: Tag for resources
-
-   Example:
-
-   ```sh
-   [{ "Key": "ApplicationName", "Value": "Omni Assistant" },
-   { "Key": "Purpose", "Value": "Learning" },
-   { "Key": "Project", "Value": "Omni Assistant" },
-   { "Key": "ProjectID", "Value": "Omni Assistant" },
-   { "Key": "Creator", "Value": "VectorNguyen" },
-   { "Key": "OwnerService", "Value": "VectorNguyen" }
+1. **AWS Resource Tags**
+   - `TAGS`: JSON array of AWS resource tags
+   ```json
+   [
+     { "Key": "ApplicationName", "Value": "Search Engine" },
+     { "Key": "Purpose", "Value": "Learning" },
+     { "Key": "Project", "Value": "Search Engine" },
+     { "Key": "Creator", "Value": "VectorNguyen" }
    ]
    ```
 
-2. **Backend**
-   _Config env Rate Limitting_
-
-   - RL_TTL
-   - RL_LIMIT
-
-   _Config refresh token, token time_
-
-   - EXP_AT
-   - EXP_RT
-
-   _Config env API for AI_
-
-   - ENDPOINT_AI
-
-   _Config CORS Socket_
+## Pipeline Workflows
 
-   - FRONTEND_URL
+### 1. Development Pipeline
 
-   _Redis_
+**File:** [development_pipeline.yml](development_pipeline.yml)
 
-   - REDIS_HOST
-   - REDIST_PORT
+- **Trigger:** Push to `develop` branch
+- **Jobs:**
+  - Run code quality checks (ruff)
+  - Run unit tests
+  - Build Docker images
 
-   _Config port backend_
+### 2. Staging Pipeline
 
-   - PORT
+**File:** [staging_pipeline.yml](staging_pipeline.yml)
 
-## Workflows
+- **Trigger:** Push to `staging` branch
+- **Jobs:**
+  - Run CI checks
+  - Deploy to staging environment
+  - Automatic rollback on failure
 
-### Development - Build and Unittest
+### 3. Production Pipeline
 
-#### File: [development_pipeline.yml](development_pipeline.yml)
+**File:** [production_pipeline.yml](production_pipeline.yml)
 
-**Event:** On Commit or Pull Request → any branch into develop
+- **Trigger:** Pull request to `master` branch
+- **Jobs:**
+  - Run CI checks
+  - Deploy to production environment
+  - Automatic rollback on failure
 
-**Jobs:**
+## Deployment Process (CD Pipeline)
 
-- Install dependencies (caches)
-- Run isort
-- Run black
-- Run flake8
+The CD pipeline implements blue-green deployment using AWS infrastructure:
 
-**Description:**
-This workflow is triggered on Pull Requests into the develop branch. It ensures a clean and standardized codebase by installing dependencies, checking code formatting with isort, black, and flake8, and finally building and pushing Docker images to Docker Hub.
+1. **Infrastructure Creation**
 
-### Staging - CI/CD Pipeline
+   - Creates VPC, subnets, security groups
+   - Launches EC2 instance
+   - Sets up Application Load Balancer
+   - Configures SSL certificate
 
-#### File: [staging_pipeline.yml](staging_pipeline.yml)
+2. **Application Deployment**
 
-**Event:** On Pull Request → any branch into staging
+   - Builds and pushes Docker images
+   - Configures EC2 instance using Ansible
+   - Deploys applications using Docker Compose
 
-**Jobs:**
+3. **Traffic Switch**
 
-- Install dependencies (caches)
-- Run isort
-- Run black
-- Run flake8
-- Build images (caches)
-- Push images to Docker Hub
-- Create infrastructure
-- Configure infrastructure
-- Deploy application using Docker Compose
-- Clean up following the concept of A/B deploy
+   - Performs health checks
+   - Updates Route53 DNS records
+   - Switches traffic to new environment
 
-**Description:**
-This pipeline is designed for the staging environment and is triggered on Pull Requests into the staging branch. It includes steps to ensure code quality, build and push Docker images, create and configure necessary infrastructure, and deploy the application using Docker Compose. The cleanup process follows the A/B deployment concept.
+4. **Cleanup**
+   - Removes old infrastructure after successful deployment
 
-### Production - CI/CD Pipeline
+## Rollback Process
 
-#### File: [production_pipeline.yml](production_pipeline.yml)
+The rollback workflow ([rollback.yml](rollback.yml)) is triggered automatically if deployment fails:
 
-**Event:** On Pull Request → any branch into master
+1. Identifies failed deployment stack
+2. Removes newly created infrastructure
+3. Traffic remains routed to previous stable environment
 
-**Jobs:**
+## Infrastructure as Code
 
-- Install dependencies (caches)
-- Run isort
-- Run black
-- Run flake8
-- Build images (caches)
-- Push images to Docker Hub
-- Create infrastructure
-- Configure infrastructure
-- Deploy application using Docker Compose
-- Clean up following the concept of A/B deploy
+The infrastructure is defined using AWS CloudFormation:
 
-**Description:**
-The production pipeline is triggered on Pull Requests into the master branch, indicating changes are ready for deployment to the production environment. It follows a similar process to the staging pipeline but is specifically tailored for the production environment. The cleanup process adheres to the A/B deployment concept, ensuring a smooth transition between versions.
+- **Template:** [server.yml](cloudformations/server.yml)
+- **Parameters:** Configurable via pipeline inputs
+- **Resources:**
+  - VPC and networking components
+  - EC2 instances
+  - Load balancer
+  - SSL certificate
+  - DNS configuration
 
 ## References
 
-- [Reusing workflows](https://docs.github.com/en/actions/sharing-automations/reusing-workflows)
+- [GitHub Actions Documentation](https://docs.github.com/en/actions)
+- [AWS CloudFormation Documentation](https://docs.aws.amazon.com/cloudformation/)
+- [Blue-Green Deployment](https://martinfowler.com/bliki/BlueGreenDeployment.html)
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
@@ -0,0 +1,61 @@
+name: build_images
+
+on:
+  workflow_call:
+    secrets:
+      DOCKERHUB_USERNAME:
+        required: true
+      DOCKERHUB_PASSWORD:
+        required: true
+
+jobs:
+  build-images:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_PASSWORD }}
+
+      - name: Build and push image search
+        uses: docker/build-push-action@v4
+        with:
+          context: ./image_search
+          push: true
+          tags: ${{ secrets.DOCKERHUB_USERNAME }}/image_search:latest
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Build and push text search
+        uses: docker/build-push-action@v4
+        with:
+          context: ./text_search
+          push: true
+          tags: ${{ secrets.DOCKERHUB_USERNAME }}/text_search:latest
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Build and push backend
+        uses: docker/build-push-action@v4
+        with:
+          context: ./backend
+          push: true
+          tags: ${{ secrets.DOCKERHUB_USERNAME }}/backend_search_engine:latest
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Build and push frontend
+        uses: docker/build-push-action@v4
+        with:
+          context: ./frontend
+          push: true
+          tags: ${{ secrets.DOCKERHUB_USERNAME }}/frontend_search_engine:latest
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -4,52 +4,7 @@ on:
   workflow_call:
 
 jobs:
-  build-test-image-search:
-    runs-on: ubuntu-latest
-    services:
-      qdrant:
-        image: qdrant/qdrant:v1.9.0
-        ports:
-          - 6334:6334
-    steps:
-      - name: Checkout Repository
-        uses: actions/checkout@v4
-
-      - name: Set up Python
-        uses: actions/setup-python@v4
-        with:
-          python-version: "3.9"
-          cache: "pip"
-
-      - name: Install Dependencies
-        working-directory: image_search
-        run: |
-          pip install -r requirements.txt
-
-      - name: Run ruff
-        run: ruff check image_search
-
-  build-test-text-search:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout Repository
-        uses: actions/checkout@v4
-
-      - name: Set up Python
-        uses: actions/setup-python@v4
-        with:
-          python-version: "3.9"
-          cache: "pip"
-
-      - name: Install Dependencies
-        working-directory: text_search
-        run: |
-          pip install -r requirements.txt
-
-      - name: Run ruff
-        run: ruff check text_search
-
-  build-test-backend:
+  lint-test:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout Repository
@@ -62,14 +17,16 @@ jobs:
           cache: "pip"
 
       - name: Install Dependencies
-        working-directory: backend
         run: |
-          pip install -r requirements.txt
+          pip install ruff
 
       - name: Run ruff
-        run: ruff check backend
+        run: |
+          ruff check image_search
+          ruff check text_search
+          ruff check backend
 
-  build-test-frontend:
+  build-frontend:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout Repository
@@ -79,10 +36,12 @@ jobs:
         uses: actions/setup-node@v4
         with:
           node-version: "18"
+          cache: "npm"
+          cache-dependency-path: frontend/package-lock.json
 
       - name: Install dependencies
         working-directory: frontend
-        run: npm install
+        run: npm ci
 
       - name: Build frontend
         working-directory: frontend