microservice-seatgeek

Setup in Google Cloud (DEV ONLY) - Development in the cloud (DitC)

Create a new GCP account at <cloud.google.com/free> to get a free $300 credit
Create a new project and a new cluster in kubernetes engine with static version: 1.15.*, 3 nodes
in Nodes, select series N1 and machine type g1-small, click create
Install Google Cloud SDK at https://cloud.google.com/sdk/docs/quickstarts
gcloud auth login
gloud init
gloud container clusters get-credentials <project-name-in-GCP>
In Docker-desktop, check if the cluster was added to the kubernetes context. Kuberbetes must be pointed to GCP k8s cluster
Install ingress-nginx on GCP, https://kubernetes.github.io/ingress-nginx/deploy/#gce-gke and GCP will create a load balancer automatically
In dev console, network services, load balancing, get the IP of the load balancer and add it to /etc/hosts on local as <IP> buytickets.dev
skaffold dev
Create a stripe account and an API key and add the secret to k8s secrets. https://stripe.com/docs/payments, https://stripe.com/docs/api
Create and apply secrets:

k create secret generic jwt-secret --from-literal JWT_KEY=ChangeThisSecretValue
k create secret generic stripe-secret --from-literal STRIPE_KEY=stripe_secret_key_value

visit https://buytickets.dev

Setup on local

Install skaffold dev
Install ingress-nginx https://kubernetes.github.io/ingress-nginx/deploy/
Get docker and k8s running
To restart ts-node-dev server, enter rs in the terminal window


brew update
brew install skaffold
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/cloud/deploy.yaml
skaffold dev
k create secret generic jwt-secret --from-literal=JWT_KEY=ChangeThisSecretValue

Flow

Signup and a JWT stored in a cookie will be returned in 1 go with Next.js
Create a ticket with a title and a price => emit ticket:created
Edit the ticket => emit ticket:updated
Create a order for a ticket => emit order:created
Ticket link with order will be locked/reserved for x mins so that no further editing can be done
After x mins, expiraion service expires the order and the ticket is unlocked
Expired order => emits order:cancelled
NEED TO COMPLETE THIS FLOW

Cutting some corners - FEATURES NOT YET IMPLEMENTED IN THIS REPO

1. Future Auth+Authz enhancement

In this project, authentication is done using a JWT that is passed around in the services.
Currently, each service can verify the validity of a JWT but cannot determine whether someone has the required Authorization level
Ideally, we should handle both Auth and Authz -> this will be done in the future
- Add expiration to JWT
- If JWT is expired, refresh by calling Auth service
- To fix authorization issue where some users might have their access removed but still have a valid JWT:
  - Add UserBanned event to EVENT_BUS and broadcast it to all services
  - All services will have a cache ideally using redis that will persist the Banned User for a duration of the expiration time of the JWT

2. Event failure and transaction rollback

After a service persist data into its own database, it will usually publish an event like ticket:created to notify other services
However, if the event fails to go through event with retries, like NATS-streaming-server going down permanently, other state of other services will not be updated
SOLUTION:
- Implement a transaction feature similar to a database transaction whereby if anything fails during the series of actions in the transaction, it is marked as failed and all previous state changes are rolled back.
- Service A stores both its data and the events it wants to publish in 2 separate collection
- An external listener is setup which get triggered when a new update is made to the events collection of service A
- The external listener then publishes the new event to NATS which publish them to any service listening to that topic
- The event is marked as success/failed by the external listener
- If failed, the whole transaction rolls back
Example: a user deposits $X to his account
- The records of his action is updated by the records service which then publish and event to update the account balance state/service
- NATS connection lost and event never goes through
- even though the record says the deposit got through, the user never sees an update to his account balance => BAD

3. Add HTTPS support

See https://cert-manager.io/docs/tutorials/acme/ingress/

4. Add Email Service to notify users

Send email to notify users on payment:complete
See Mailchimp/Sendgrid

5. Add `build` steps for prod cluster

Currently, the services are running in dev mode with ts-node-dev
Create additional Dockerfiles and npm commands to start service in prod mode

6. Add a Staging environment

Create a staging cluster on Digital Ocean
Add CICD pipelines from github to the Staging cluster

Note

This repo contains an e-commerce microservice platform built using a Reactive / event-driven approach. Transactions are made using Stripe.js
During dev on localhost, if chrome responds with a privacy security warning, type thisisunsafe. This page is thrown by the nginx engine used in this app because it might not be using https or it is using self-signed certs.
The private mongodb instances for every service will be created by k8s depl on skaffold dev
Password hashing with scrypt instead of bscrypt: https://stackoverflow.com/questions/1226513/whats-the-advantage-of-scrypt-over-bcrypt
- scrypt requires increasingly more ram and computational power
- scrypt is also a derivative of PBKDF2 inside the crypto native module in nodeJS, making module auditing easier
- bcrypt (and PBKDF2) use constant, and small, amounts of memory.
- scrypt require 4000x more resources to run than bcrypt

Server side rendering issues with NextJs

Usually, a regular react app has 3 stages:
1. Initial request to buytickets.dev: fetch boilerplate html
2. 2nd request: fetch react scripts with rendering logic
3. 3rd request: Auth + fetch data from resource API
NextJS will try to build the full html page with all the content and send it back with only a single initial request
- Client request buytickets.dev
- NextJS makes call to API to get data, builds the fully rendered html with content and returns it
Calling the API resources needs Auth+Authz with JWT, but we cannot attach any headers or body content with the initial request to auth the user in
- Solution: Attach JWT to a cookie in the initial request with npm module cookie-session
- Corner case solution: Use service-workers -> Requires major architectural change
Advantage of SSR:
- SEO: Search Engine Optimization
- Faster page load speed: better UX on older or mobile devices with low processing power

Cookies VS JWT

          Cookies                                     |         JWT
    Transport mechanism                               | Auth + Authz mechanism
  Moves any kind  of data from browser and server     | Stores any data we want
  Auto managed by browser                             | Managed manually by devs

How to create a secret for our JWT signing key

Create the secret k8s object inside our cluster. It will be available in our pods as ENV_VARIABLE inside the container running the specific service
kubectl create secret generic <secret-name> --from-literal=<key>=<value>
- e.g: k create secret generic jwt-secret --from-literal=JWT_KEY=ChangeThisSecretValue
- k get secrets to get all secrets

Concurrency issues with :updated events - Solution: Record Updates with Optimistic Concurrency Control

After the ticket and order services have been created and can now CRUD and persist their own and redundant data by emitting events, we observed concurrency issues where events are not always processed in order
Scenario: 4 instance of tickets and orders each, 200 requests doing the following:
- Create a ticket with price = 5
- Update the ticket to price = 10
- Update the ticket to price = 15
Expect: all tickets in the tickets db and the orders db have price = 15
Result: Some tickets have price = 10
Solution:
- Add a version flag to the tickets and use it to process events in order. If out of order, do NOT msg.ack() and let the event timeout. NATS will then retry to publish that event 5s later in the hope that the missing version has been processed
- The version flag __v from mongoose and mongoDB and be used to that end
- Lookup: Record Updates with Optimistic Concurrency Control
- Use npm package: mongoose-update-if-current: https://www.npmjs.com/package/mongoose-update-if-current
  - This package can implement Optimistic concurrency control with version number or timestamps
- In listener, query resource with both _id and version flag
  - If not found, we are processing ticket out of order
  - Reject event OR allow to timeout so that NATS can requeue and retry to send this event

Services overview

Auth

On signup, Auth will create a JWT with no expiration and return it in a cookie
Has signin and signout logic
Need to add JWT expiration and other features from Cutting some corners - FEATURES NOT YET IMPLEMENTED IN THIS REPO

Ticket

CRUD tickets
Events:
- ticket:created
- ticket:updated
Has its own db to store all tickets

Order

CRUD orders
Events:
- order:created
- order:cancelled
Associates and locks a ticket with an order with an orderId
Locks the order for 15mins to allow for payment and waits for expiration service to free up the ticket once the order is expired
Once a ticket is lock, it cannot be edited and can only be purchased or expired/cancelled

Expiration

Expires orders
Has a webserver to listen to order:created and emit expiration:complete
order:created event payload already has a expiresAt property that the exp service watches and emits on expiration
Expiration service has no logic to process order expiration -> only emits event, it is the order service that listens to expiration:complete and has expiration-specific logic
Uses Bull.js to create and manage job queues
Jobs are stored in redis
Jobs are separately processed on a worker server

Payment

Uses Stripe.js
- Backend: https://www.npmjs.com/package/stripe
- Frontend: https://www.npmjs.com/package/react-stripe-checkout
To test stripe:
- Test Token: tok_visa,
- VISA card: 4242424242424242
More on https://stripe.com/docs/testing

Nats Event Broker

Event broker using NATS-streaming-server
Makes use of queue group name to avoid duplicate event processing
Has event sourcing and redelivery for reliability and to allow new/restarted services to sync up their states

Responses

error-handler middleware takes in the error, formats them and sends them back. Any error specific logic is handled by the specific error superset in /src/errors

Error response format

400 Bad Request

Interface


{
  errors: {
    message: string,
    field?: string
  }[]
}

Example


{
  "errors": [
    {
      "message": "Email must be valid",
      "field": "email"
    },
    {
      "message": "Password must be between 4 and 30 characters",
      "field": "password"
    }
  ]
}

K8S namespaces

All services are created under the default namespace
To do inter-services communication inside the same cluster, we use ClusterIP
To do Cross-namespace communication, we need to reach the ingress-nginx load balancer first, then use its route rules to reach the desired service

k get namespace, returns ingress-nginx

k get services -n , eg: k get services -n ingress-nginx, returns ingress-nginx-controller

http://..svc.cluster.local/

http://ingress-nginx-controller.ingress-nginx.svc.cluster.local/api/user/currentuser

npm packages - how to build and publish the common library

Create an npm account and a public organisation
https://docs.npmjs.com/cli/version
https://www.npmjs.com/package/@jnch-microservice-tickets/common

  npm login
  npm run build
  git add .
  git commit -m <message>
  npm version patch/minor/major
  git push
  npm publish --access public

To update common module in other services

npm update @jnch-microservice-tickets/common

EVENT BUS with NATS streaming server

https://docs.nats.io/nats-streaming-server/changes
https://hub.docker.com/_/nats-streaming
This repo uses the nats-pod-name as the nats client-id in nats.connect(clusterId, clientId, { nats-cluster-ip }); as described in ticket-depl.yaml and /ticket/index.ts

Setup

Local

  npm i
  npm run publish
  npm run listen

Port Forwarding to make the pod accessible during dev in local/GCP K8S

  kubectl get pods
  kubectl port-forward <nats-pod-name> 4222:4222
  kubectl port-forward <nats-pod-name> 8222:8222

To access the monitoring page

goto localhost:8222/streaming

NATS server health check

In nats-depl.yaml k8s deployment file, we have 3 hb: heartbeat arguments in arg of
- -hbi: How often NATS will make a health request to each of its client
- -hbt: How long each client has to respond
- -hbf: How many times each client can fail until NATS assume they are down/unhealthy

Deployment and automation

Github repo approach

Mono Repo Approach
Automate workflow, test and build using Github Actions https://help.github.com/en/actions/reference/events-that-trigger-workflows
- On PR create/commits, run tests inside respective repo with new changes
Check .github/workflows/tests-*.yml

Full deployment on Digital Ocean => Cheaper at $40/month => Google search $100 credit coupon on new account

Look for a coupon and create a new Digital Ocean account
Create a cluster with $10/months, 3 nodes-cluster
Install doctl at https://github.com/digitalocean/doctl

brew install doctl
Go to Digital Ocean => API => Generate new token

doctl auth init
Enter the token
Go to Digital Ocean => Kubernetes => <Your_Cluster_Name>
Add the K8S context to your local kubectl

doctl kubernetes cluster kubeconfig save <Your_Cluster_Name>
List all K8S context

kubectl config view
Use a different context OR if on Mac, use the docker-desktop client => kubernetes => choose the context

kubectl config use-context <Context_Name>
Create github actions for continuous deployment of our services on PR merge to master
- Action Flow on master branch change:
  - Services: Build new image => Push to docker hub => update deployment
  - Infra: Apply all yaml files to our cluster on Digital Ocean
- Before creating the new workflows, Create a Github secret inside github to allow dockerhub to login
  - Github repo => settings => secrets => add secret
Add Digital Ocean API key and Cluster Name as secret in github
Add doctl to github container and point kubectl inside github to Digital Ocean => see ./github/workflows/deploy-*.yml
Create JWT_KEY and STRIPE_KEY secrets inside Digital Ocean
Setup Ingress-Nginx inside Digital Ocean https://kubernetes.github.io/ingress-nginx/deploy/#digital-ocean

Name		Name	Last commit message	Last commit date
Latest commit History 173 Commits
.github/workflows		.github/workflows
auth		auth
client		client
expiration		expiration
infra		infra
order		order
payment		payment
ticket		ticket
.gitignore		.gitignore
README.md		README.md
microservice-seatgeek-collection.postman_collection.json		microservice-seatgeek-collection.postman_collection.json
skaffold.yaml		skaffold.yaml

jeremyNCH/microservice-seatgeek

Folders and files

Latest commit

History

Repository files navigation

microservice-seatgeek

Setup in Google Cloud (DEV ONLY) - Development in the cloud (DitC)

Setup on local

Flow

Cutting some corners - FEATURES NOT YET IMPLEMENTED IN THIS REPO

1. Future Auth+Authz enhancement

2. Event failure and transaction rollback

3. Add HTTPS support

4. Add Email Service to notify users

5. Add build steps for prod cluster

6. Add a Staging environment

Note

Server side rendering issues with NextJs

Cookies VS JWT

How to create a secret for our JWT signing key

Concurrency issues with :updated events - Solution: Record Updates with Optimistic Concurrency Control

Services overview

Auth

Ticket

Order

Expiration

Payment

Nats Event Broker

Responses

Error response format

K8S namespaces

npm packages - how to build and publish the common library

EVENT BUS with NATS streaming server

Setup

Local

Port Forwarding to make the pod accessible during dev in local/GCP K8S

NATS server health check

Deployment and automation

Github repo approach

Full deployment on Digital Ocean => Cheaper at $40/month => Google search $100 credit coupon on new account

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

5. Add `build` steps for prod cluster

Packages