DevOps course, practices with Google Cloud Platform.
Docker helps to package the software into standardized units for development, shipment and deployment. Containers are lightweight by design and ideal for enabling microservices application development.
Use Docker to create instance in GCE
and publish docker image in Docker Hub.
- Configure
gcloud
and create newGCE
projectdocker
~$ gcloud init
~$ gcloud auth application-default login
- Create instance in
GCE
bydocker-machine
and change the environment variables for theDocker Client
. It should connect to remoteDocker Engine
.
~$ docker-machine create --driver google \
--google-project <project_id> \
--google-zone europe-west1-b \
--google-machine-type g1-small \
--google-machine-image $(gcloud compute images list --filter ubuntu-1604-lts --uri) \
<docker_instance_name>
~$ eval $(docker-machine env <docker_instance_name>)
- Display machines running Docker
~$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
<docker_instance_name> * google Running tcp://<host_ip>:2376 v17.09.0-ce
- Build docker image
~$ docker build -t reddit:latest ./
~$ docker images -a
- Run docker container
$ docker run --name reddit -d --network=host reddit:latest
- Create firewall rule for reddit app
~$ gcloud compute firewall-rules create reddit-app \
--allow tcp:9292 --priority=65534 \
--target-tags=docker-machine \
--description="Allow TCP connections" \
--direction=INGRESS
- Docker machine needs to have rules
docker-machines
andreddit-app
~$ gcloud compute firewall-rules list
NAME NETWORK DIRECTION PRIORITY ALLOW DENY
default-allow-icmp default INGRESS 65534 icmp
default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp
default-allow-rdp default INGRESS 65534 tcp:3389
default-allow-ssh default INGRESS 65534 tcp:22
docker-machines default INGRESS 1000 tcp:2376
reddit-app default INGRESS 65534 tcp:9292
- Open URL http://<host_ip>:9292 and test the app
- Connect to the remote instance by ssh
~$ docker-machine ssh <docker_instance_name>
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.10.0-37-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
...
docker-user@docker-host:~$ lsb_release -d
Description: Ubuntu 16.04.3 LTS
- Auth and push docker image to Docker Hub
~$ docker login
~$ docker tag reddit:latest <user-login>/otus-reddit:1.0
~$ docker push <user-login>/otus-reddit:1.0
The push refers to a repository [docker.io/dashishmakov/otus-reddit]
...
1.0: digest: sha256:6d60a3915efdd937a9ae0ad9e8c7b1fa2edc1a9d63000f9993cbcffa4fcca085 size: 3241
- Create instance in
GCE
bydocker-machine
and change the environment variables for theDocker Client
. It should connect to remoteDocker Engine
.
~$ docker-machine create --driver google \
--google-project <project_id> \
--google-zone europe-west1-b \
--google-machine-type g1-small \
--google-machine-image $(gcloud compute images list --filter ubuntu-1604-lts --uri) \
<docker_instance_name>
~$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
<docker_instance_name> - google Running tcp://<host_ip>:2376 v17.09.0-ce
~$ docker-machine ssh <docker_instance_name>
~$ eval $(docker-machine env <docker_instance_name>)
- Download the latest MongoDB image
~$ docker pull mongo:latest
- Build images and test them
~compose$ docker build -t dashishmakov/ui:1.0 -f ./ui/Dockerfile_1_0 ./ui/
~compose$ docker build -t dashishmakov/comment:1.0 ./comment/
~compose$ docker build -t dashishmakov/post:1.0 ./post-py/
~compose$ docker images -a | grep dashishmakov
dashishmakov/ui 1.0 0594004be8ef 3 minutes ago 779MB
dashishmakov/comment 1.0 e18cb5c57fe5 4 minutes ago 774MB
dashishmakov/post 1.0 1ab61dabb6fe 10 minutes ago 102MB
- Create bridge-network for reddit app
~$ docker network create reddit
~$ docker network ls
NETWORK ID NAME DRIVER SCOPE
cac29504a2ac bridge bridge local
cce3aff812f4 host host local
97112fc564cf none null local
d85b1da7d878 reddit bridge local
- Run containers, bind each of them to the network
reddit
and set network aliases
~$ docker run -d --network=reddit --network-alias=post_db --network-alias=comment_db mongo:latest
~$ docker run -d --network=reddit --network-alias=post dashishmakov/post:1.0
~$ docker run -d --network=reddit --network-alias=comment dashishmakov/comment:1.0
~$ docker run -d --network=reddit -p 9292:9292 dashishmakov/ui:1.0
~$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5ad6df21619c dashishmakov/ui:1.0 "puma" 27 seconds ago Up 25 seconds 0.0.0.0:9292->9292/tcp stoic_fermat
8e10a97ee1e6 dashishmakov/comment:1.0 "puma" 37 seconds ago Up 36 seconds pedantic_kalam
cc883b6c52ba dashishmakov/post:1.0 "python3 post_app.py" 47 seconds ago Up 46 seconds naughty_fermi
f8d620170918 mongo:latest "docker-entrypoint..." 56 seconds ago Up 55 seconds 27017/tcp modest_blackwell
- Open URL http://<host_ip>:9292 and test the app
- Stop running containers
~$ docker kill $(docker ps -q)
- If you want to change the network aliases used for communication between applications you should define new env values for each container
~$ docker run -d --network=reddit --network-alias=post_db --network-alias=net_comment_db \
-e POST_SERVICE_HOST='net_post' \
-e COMMENT_SERVICE_HOST='net_comment' \
-e COMMENT_DATABASE_HOST='net_comment_db' \
mongo:latest
~$ docker run -d --network=reddit --network-alias=net_post \
-e POST_SERVICE_HOST='net_post' \
-e COMMENT_SERVICE_HOST='net_comment' \
-e COMMENT_DATABASE_HOST='net_comment_db' \
dashishmakov/post:1.0
~$ docker run -d --network=reddit --network-alias=net_comment \
-e POST_SERVICE_HOST='net_post' \
-e COMMENT_SERVICE_HOST='net_comment' \
-e COMMENT_DATABASE_HOST='net_comment_db' dashishmakov/comment:1.0
~$ docker run -d --network=reddit \
-p 9292:9292 \
-e POST_SERVICE_HOST='net_post' \
-e COMMENT_SERVICE_HOST='net_comment' \
-e COMMENT_DATABASE_HOST='net_comment_db' \
dashishmakov/ui:1.0
-
Open URL http://<host_ip>:9292 and test the app
-
Docker images are big and usually much larger than they need to be. True OS image runs either real hardware or virtualized hardware, but not inside a Docker container. Inside a container you don't want a full system. Use other docker files to build image 2.0 (ubuntu:16.04), image 3.0 (alpine:3.6) and test them with 1.0 (ruby:2.2)
~compose$ docker build -t dashishmakov/ui:2.0 -f ./ui/Dockerfile_2_0 ./ui/
~compose$ docker build -t dashishmakov/ui:3.0 -f ./ui/Dockerfile_3_0 ./ui/
~compose$ docker images dashishmakov/ui
dashishmakov/ui 3.0 086371a13621 25 seconds ago 199MB
dashishmakov/ui 2.0 caa68976405a About an hour ago 454MB
dashishmakov/ui 1.0 0594004be8ef 3 days ago 779MB
- Stop running containers
~$ docker kill $(docker ps -q)
- Create docker volume to save the data after restarting containers and mount it to the
mongo:latest
~$ docker volume create reddit_db
~$ docker volume ls
~$ docker run -d --network=reddit --network-alias=post_db --network-alias=comment_db --mount source=reddit_db,target=/data/db mongo:latest
~$ docker run -d --network=reddit --network-alias=post dashishmakov/post:1.0
~$ docker run -d --network=reddit --network-alias=comment dashishmakov/comment:1.0
~$ docker run -d --network=reddit -p 9292:9292 dashishmakov/ui:1.0
- Start new container with volume
~$ docker kill <mongo_container_id>
~$ docker run -d --network=reddit --network-alias=post_db --network-alias=comment_db --mount source=reddit_db,target=/data/db mongo:latest
~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d25e4f7d898d dashishmakov/ui:1.0 "puma" 2 hours ago Up 15 seconds 0.0.0.0:9292->9292/tcp friendly_yonath
1bdfd93a23cc dashishmakov/comment:1.0 "puma" 2 hours ago Up 15 seconds agitated_jackson
76fbdb3c77e8 dashishmakov/post:1.0 "python3 post_app.py" 2 hours ago Up 14 seconds mystifying_hypatia
3530edb57307 mongo:latest "docker-entrypoint..." 2 hours ago Up 13 seconds 27017/tcp zen_hawking
- Open URL http://<host_ip>:9292, test the app and posts
- Create instance in
GCE
bydocker-machine
and change the environment variables for theDocker Client
. It should connect to remoteDocker Engine
.
~$ docker-machine create --driver google \
--google-project <project_id> \
--google-zone europe-west1-b \
--google-machine-type g1-small \
--google-machine-image $(gcloud compute images list --filter ubuntu-1604-lts --uri) \
docker-host
~$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
docker-host * google Running tcp://<host_ip>:2376 v17.10.0-ce
- Run container
net_test
withnone
network driver that has timer and will deleted after 100 sec. Container has Loopback network interface only.
~$ docker run --network none --rm -d --name net_test joffotron/docker-net-tools -c "sleep 100"
~$ docker exec -ti net_test ifconfig
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
- Run container
net_test
withhost
network driver and testifconfig
outputs
~$ docker run --network host --rm -d --name net_test joffotron/docker-net-tools -c "sleep 100"
~$ docker exec -ti net_test ifconfig
~$ docker-machine ssh docker-host ifconfig
- Create new
bridge
network (it is a default network in the docker and you no need to define key '--driver' explicitly, but just do it)
~$ docker network create reddit --driver bridge
~$ docker network ls
NETWORK ID NAME DRIVER SCOPE
e7a048591474 bridge bridge local
e5e41b5bb7b9 host host local
a9aca5eaa1f3 none null local
df49b0667c1c reddit bridge local
- Run containers, bind each of them to the network
reddit
and set network aliases because services search by DNS names from env variables
~$ docker run -d --network=reddit --network-alias=post_db --network-alias=comment_db mongo:latest
~$ docker run -d --network=reddit --network-alias=post dashishmakov/post:1.0
~$ docker run -d --network=reddit --network-alias=comment dashishmakov/comment:1.0
~$ docker run -d --network=reddit -p 9292:9292 dashishmakov/ui:1.0
-
Open URL http://<host_ip>:9292 and test the app
-
Right now all services use one network interface. Let's create different network interfaces and separate
ui
fromdb
~$ docker network create back_net --subnet=10.0.2.0/24
~$ docker network create front_net --subnet=10.0.1.0/24
~$ docker run -d --network=front_net -p 9292:9292 --name ui dashishmakov/ui:1.0
~$ docker run -d --network=back_net --name comment dashishmakov/comment:1.0
~$ docker run -d --network=back_net --name post dashishmakov/post:1.0
~$ docker run -d --network=back_net --name mongo_db --network-alias=post_db --network-alias=comment_db mongo:latest
~$ docker network connect front_net post
~$ docker network connect front_net comment
-
Open URL http://<host_ip>:9292 and test the app
-
Let's look at the current Linux network stack on virtual instance in
GCE
~$ docker-machine ssh docker-host
~$ sudo apt-get update && sudo apt-get install bridge-utils
~$ sudo docker network ls
NETWORK ID NAME DRIVER SCOPE
310a8bd102cf back_net bridge local
e7a048591474 bridge bridge local
593663a63d49 front_net bridge local
e5e41b5bb7b9 host host local
a9aca5eaa1f3 none null local
df49b0667c1c reddit bridge local
~$ ifconfig | grep br
br-310a8bd102cf Link encap:Ethernet HWaddr 02:42:83:55:32:14
br-593663a63d49 Link encap:Ethernet HWaddr 02:42:1d:7a:48:6c
br-df49b0667c1c Link encap:Ethernet HWaddr 02:42:96:6c:be:1d
~$ brctl show br-593663a63d49
bridge name bridge id STP enabled interfaces
br-593663a63d49 8000.02421d7a486c no veth3b6ae49
vethc70235b
vethcf4e66b
- Show iptables
~$ sudo iptables -nL -t nat
...
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9292 to:10.0.1.2:9292
- Find the
docker-proxy
process listening on TCP port 9292
~$ ps ax | grep docker-proxy
- Stop and remove running containers
~$ docker kill $(docker ps -q)
~$ docker rm $(docker ps -aq)
A service definition contains configuration which will be applied to each container started for that service,
much like passing command-line parameters to docker run
, docker network
, docker volume
- Set environment variables for
docker-compose
~compose$ export USERNAME=dashishmakov
~compose$ cp .env.example .env
~compose$ docker-compose up -d
~compose$ docker-compose ps
Name Command State Ports
---------------------------------------------------------------------------------------
microservices_comment_1 puma Up
microservices_mongo_db_1 docker-entrypoint.sh mongod Up 27017/tcp
microservices_post_1 python3 post_app.py Up
microservices_ui_1 puma Up 0.0.0.0:9292->9292/tcp
1.1) Prometheus is a powerful time-series monitoring service, providing a flexible platform for monitoring software products. Let's run and get acquainted with this product.
- create instance in
GCE
bydocker-machine
. Change the environment variables for theDocker Client
and connect to the remoteDocker Engine
$ docker-machine create --driver google \
--google-project <project_id> \
--google-machine-image https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/family/ubuntu-1604-lts \
--google-machine-type n1-standard-1 \
--google-zone europe-west1-b \
vm1
$ eval $(docker-machine env vm1)
- check firewall-rules:
$ gcloud compute firewall-rules list
NAME NETWORK DIRECTION PRIORITY ALLOW DENY
default-allow-icmp default INGRESS 65534 icmp
default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp
default-allow-rdp default INGRESS 65534 tcp:3389
default-allow-ssh default INGRESS 65534 tcp:22
docker-machines default INGRESS 1000 tcp:2376
prometheus-default default INGRESS 1000 tcp:9090
puma-default default INGRESS 1000 tcp:9292
- create if you do not have one of them
~$ gcloud compute firewall-rules create default-allow-ssh --allow tcp:22 --priority=65534 --description="Allow SSH connections" --direction=INGRESS
~$ gcloud compute firewall-rules create prometheus-default --allow tcp:9090
~$ gcloud compute firewall-rules create puma-default --allow tcp:9292
- run Prometheus from the DockerHub image
~compose$ docker run --rm -p 9090:9090 -d --name prometheus prom/prometheus
~compose$ docker ps
-
open URL http://<host_ip>:9090/targets and `http://<host_ip>:9090/metrics
-
stop docker instance Prometheus
~compose$ docker stop prometheus
1.2) Prometheus will monitor all microservices, so we need a container with Prometheus that could communicate
over the network with all other services and config it in docker-compose
file
- set environment variables for
docker-compose
~compose$ export USERNAME=<dockerhub_login>
~compose$ cp .env.example .env
- build custom images with Prometheus and other microservices
~compose/prometheus$ bash docker_build.sh
~compose/ui$ bash docker_build.sh
~compose/post-py$ bash docker_build.sh
~compose/comment$ bash docker_build.sh
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
dashishmakov/ui latest dfac89a27201 43 minutes ago 204MB
dashishmakov/comment latest ce03118d56db 2 hours ago 778MB
dashishmakov/post latest 8cafeba72426 2 hours ago 101MB
dashishmakov/prometheus latest 1c9abba20bb2 2 hours ago 80.2MB
- run all containers
~compose$ docker-compose up -d
~compose$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------
microservices_comment_1 puma Up
microservices_mongo_db_1 docker-entrypoint.sh mongod Up 27017/tcp
microservices_post_1 python3 post_app.py Up 5000/tcp
microservices_prometheus_1 /bin/prometheus -config.fi ... Up 0.0.0.0:9090->9090/tcp
microservices_ui_1 puma Up 0.0.0.0:9292->9292/tcp
- open URL http://<host_ip>:9292/targets, http://<host_ip>:9090/metrics, http://<host_ip>:9090/targets and test the app
<host_ip>
is an external host ip:docker-machine ip vm1
1.3) Healthcheck
is a part of metrics for each service and runs into each of them (part of source code).
Microservices are dependent from each other and healthcheck
indicates availability all endpoints for concrete service (1 = healthy, 0 = unhealthy)
- open URL [http://<host_ip>:9090] and find graph
ui_health
- stop
post
service and refresh graph
~compose$ docker-compose stop post
- open graphs
ui_health_post_availability
andui_health_comment_availability
and test each of them - start
post
service should fixHealthcheck
~compose$ docker-compose start post
1.4) Node exporter is a part of project Prometheus. It helps to collect hardware and *NIX kernels metrics of host machine (main virtual machine) for Prometheus; MongoDB exporter collects metrics about sharding, replication and storage engines
- rebuild Prometheus image if previous version don't have access to Node exporter, build MongoDB exporter image and restart microservices
~compose/prometheus$ bash docker_build.sh
~compose/mongodb-exporter$ bash docker_build.sh
~compose$ docker-compose down
~compose$ docker-compose up -d
~compose$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------------
microservices_comment_1 puma Up
microservices_mongo_db_1 docker-entrypoint.sh mongod Up 27017/tcp
microservices_mongodb-exporter_1 mongodb_exporter Up 9001/tcp
microservices_node-exporter_1 /bin/node_exporter --path. ... Up 9100/tcp
microservices_post_1 python3 post_app.py Up 5000/tcp
microservices_prometheus_1 /bin/prometheus --config.f ... Up 0.0.0.0:9090->9090/tcp
microservices_ui_1 puma Up 0.0.0.0:9292->9292/tcp
- open URL [http://<host_ip>:9090/targets] and test
node_cpu
metric - push all microservice images to Docker Hub repository
~compose$ docker login
~compose$ docker push $USERNAME/ui
~compose$ docker push $USERNAME/post
~compose$ docker push $USERNAME/comment
~compose$ docker push $USERNAME/prometheus
~compose$ docker push $USERNAME/mongodb_exporter
1.1) Docker's container monitoring. cAdvisor analyzes resource usage and performance characteristics of running containers. It collects, aggregates, processes, and might to exports information to various storages such as Prometheus, ElasticSearch, InfluxDB, Kafka and simple stdout.
- create instance in
GCE
bydocker-machine
. Change the environment variables for theDocker Client
and connect to the remoteDocker Engine
$ docker-machine create --driver google \
--google-project <project_id> \
--google-machine-image https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/family/ubuntu-1604-lts \
--google-machine-type n1-standard-1 \
--google-zone europe-west1-b \
--google-open-port 80/tcp \
--google-open-port 3000/tcp \
--google-open-port 8080/tcp \
--google-open-port 9090/tcp \
--google-open-port 9292/tcp \
vm1
$ eval $(docker-machine env vm1)
- new service
cadvisor
should be defined indocker-compose
file and Prometheus should get information from here - set environment variables for docker-compose
~compose$ export USERNAME=<dockerhub_login>
~compose$ cp .env.example .env
- rebuild Prometheus image and start microservices
~compose/prometheus$ bash docker_build.sh
~compose$ docker-compose up -d
~compose$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------------
microservices_cadvisor_1 /usr/bin/cadvisor -logtostderr Up 0.0.0.0:8080->8080/tcp
microservices_comment_1 puma Up
microservices_mongo_db_1 docker-entrypoint.sh mongod Up 27017/tcp
microservices_mongodb-exporter_1 mongodb_exporter Up 9001/tcp
microservices_node-exporter_1 /bin/node_exporter --path. ... Up 9100/tcp
microservices_post_1 python3 post_app.py Up 5000/tcp
microservices_prometheus_1 /bin/prometheus --config.f ... Up 0.0.0.0:9090->9090/tcp
microservices_ui_1 puma Up 0.0.0.0:9292->9292/tcp
- open URL [http://<host_ip>:8080/docker] and watch detailed information for each one; [http://<host_ip>:8080/metrics] defines metrics for Prometheus,
container_memory_usage_bytes
for example - open URL [http://<host_ip>:9090] and test metric
container_memory_usage_bytes
and others in Prometheus
1.2.1) Use Grafana for visualizing metrics data from docker containers
- new service
grafana
should be defined indocker-compose
file - start service
grafana
~compose$ docker-compose up -d grafana
~compose$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------------
microservices_cadvisor_1 /usr/bin/cadvisor -logtostderr Up 0.0.0.0:8080->8080/tcp
microservices_comment_1 puma Up
microservices_grafana_1 /run.sh Up 0.0.0.0:3000->3000/tcp
microservices_mongo_db_1 docker-entrypoint.sh mongod Up 27017/tcp
microservices_mongodb-exporter_1 mongodb_exporter Up 9001/tcp
microservices_node-exporter_1 /bin/node_exporter --path. ... Up 9100/tcp
microservices_post_1 python3 post_app.py Up 5000/tcp
microservices_prometheus_1 /bin/prometheus --config.f ... Up 0.0.0.0:9090->9090/tcp
microservices_ui_1 puma Up 0.0.0.0:9292->9292/tcp
- open URL [http://<host_ip>:3000/docker], login and add new data source, name:
Prometheus Server
, type:Prometheus
, url:http://<host_ip>:9090
- download shared dashboard
Docker and system monitoring
from Dashboards and import it
1.2.2) Let's create new dashboard UI Service Monitoring
for Grafana:
-
main menu
->dashboards
->new
graph
->edit panel title
->metrics
tab ->data source
: Prometheus Server -> select metricui_request_count
- tune graph: refresh
every 10 seconds
and showlast 15 minutes
general
tab ->Title
:UI HTTP Requests
-
save template with name:
Add UI HTTP Requests
-
add new row
Graph
for our custom dashboardgraph
->edit panel title
->metrics
tab ->data source
: Prometheus Server -> select metric patternrate(ui_request_count{http_status=~"^[45].*"}[5m])
- tune graph: refresh
every 10 seconds
and showlast 15 minutes
general
tab ->Title
:Rate of UI HTTP Requests with Error
- for test only open URL
http://<host_ip>:9292/nonexistent
and register exception with undefined endpoint
-
save template with name:
Add HTTP Requests with Error status code
-
add 3rd row
Graph
for our custom dashboardgraph
->edit panel title
->metrics
tab ->data source
: Prometheus Server -> select metric patternhistogram_quantile(0.95, sum(rate(ui_request_latency_seconds_bucket[5m])) by (le))
general
tab ->Title
:HTTP response time 95th percentile
-
save template with name:
Add HTTP response time 95th percentile
-
each dashboard has own
version history
of changes, look at configuration -
export dashbord to JSON file with name:
UI Service Monitoring
1.2.3) Let's create new one dashboard Business Logic Monitoring
:
- title:
Post count
, metrics query:rate(post_count[1h])
, save template - title:
Comment Count
, metrics query:rate(comment_count[1h])
, save template - export dashbord to JSON file
1.3) Alerts help to know about some problems in production environment at the same time when that happens. Let's add WebHook for your own Slack channel to get notifications in some accident cases. Alertmanager is a component for Prometheus that handle alerts and rote sufisticated notifications
- new service
alertmanager
should be defined indocker-compose.yml
andprometheus.yml
that Prometheus should to know about it - edit
config.yml
and define your ownslack_api_url
andreceivers.slack_configs.channel
- build image of
alertmanager
and run container
~compose/alertmanager$ ./docker_build.sh
~compose$ docker-compose up -d alertmanager
- if you edit
alert.rules.yml
you should rebuildprometheus
docker image and run it again
~compose$ docker-compose stop prometheus
~compose$ docker-compose rm prometheus
~compose$ docker-compose up -d prometheus
- open URL [http://<host_ip>:9090] -> set
up
-> submitexecute
and look for this graph - stop container
post
and wait 1 minute, Slack should receive notification[FIRING:1] InstanceDown (post:5000 post page)
~compose$ docker-compose stop post
... wait 1m
~compose$ docker-compose up -d post
At the end remove docker containers and remote instance of docker machine
~compose$ docker-compose kill
~compose$ docker-compose rm
~compose$ docker-machine kill <docker_instance_name>
~compose$ docker-machine rm <docker_instance_name>
1.1) Docker include swarm
mode for natively managing a cluster of Docker Engines called a Docker Swarm.
Docker Swarm available out of the box for cluster management and orchestration features.
- create several instances in
GCE
bydocker-machine
.
~swarm$ docker-machine create --driver google \
--google-project <project_id> \
--google-zone europe-west1-b \
--google-machine-type g1-small \
--google-machine-image $(gcloud compute images list --filter ubuntu-1604-lts --uri) \
--google-open-port 80/tcp \
--google-open-port 3000/tcp \
--google-open-port 8080/tcp \
--google-open-port 9090/tcp \
--google-open-port 9292/tcp \
master-1
~swarm$ docker-machine create --driver google \
--google-project <project_id> \
--google-zone europe-west1-b \
--google-machine-type g1-small \
--google-machine-image $(gcloud compute images list --filter ubuntu-1604-lts --uri) \
worker-1
~swarm$ docker-machine create --driver google \
--google-project <project_id> \
--google-zone europe-west1-b \
--google-machine-type g1-small \
--google-machine-image $(gcloud compute images list --filter ubuntu-1604-lts --uri) \
worker-2
- change the environment variables for the
Docker Client
and connect to the remoteDocker Engine
(nodemaster-1
)
~swarm$ eval $(docker-machine env master-1)
~swarm$ export USERNAME=dashishmakov
- rebuild docker image for
ui
service and push latest version to Docker Hub repository
~swarm/ui$ bash docker_build.sh
~swarm$ docker push $USERNAME/ui
- init Docker Swarm
- node
master-1
switch toswarm
mode - node to become
swarm
manager node - node assigned a hostname
- node listen port 2377
- node has status
active
and could get task from scheduler - init distributed persistent key-value storage for orchestration
- generates self-signed root certificate for
swarm
- generates tokens to bind
Worker
иManager
nodes to the cluster - creates overlay-network
Ingress
to publish ports outside the swarm. All nodes participate in an ingress routing mesh
- node
~swarm$ docker swarm init --advertise-addr <master-1_internal_ip>:2377
Swarm initialized: current node (1scjwp239x4ajg5kabr1kv0wm) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-0wo27ownjz1zgw0zsbit5e7aa69r7td8a91tpvnu5m32cenbtp-bmrtyoioou9zupoc2w9sug6zv <master-1_internal_ip>:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
- add
worker-1
andworker-2
toswarm
cluster
~swarm$ docker-machine ssh worker-1
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.10.0-40-generic x86_64)
...
docker-user@worker-1:~$ sudo docker swarm join \
--token SWMTKN-1-0wo27ownjz1zgw0zsbit5e7aa69r7td8a91tpvnu5m32cenbtp-bmrtyoioou9zupoc2w9sug6zv \
<master-1_internal_ip>:2377
This node joined a swarm as a worker.
~swarm$ docker-machine ssh worker-2
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.10.0-40-generic x86_64)
...
docker-user@worker-2:~$ sudo docker swarm join \
--token SWMTKN-1-0wo27ownjz1zgw0zsbit5e7aa69r7td8a91tpvnu5m32cenbtp-bmrtyoioou9zupoc2w9sug6zv \
<master-1_internal_ip>:2377
This node joined a swarm as a worker.
- print all nodes in the
swarm
~swarm$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
1scjwp239x4ajg5kabr1kv0wm * master-1 Ready Active Leader
h4rf9on49q2v2n623ei0a2zlg worker-1 Ready Active
whc2xwhg3rfvbzxp3icpirnd9 worker-2 Ready Active
1.2) Docker Compose is a good tool for defining and running multi-container of Docker applications.
Docker Compose is the heart of Docker Stack
and define stack of services for swarm
- let's deploy a new Docker Stack
~swarm$ docker stack deploy --compose-file=<(docker-compose -f docker-compose.yml config 2>/dev/null) DEV
~swarm$ docker stack services DEV
ID NAME MODE REPLICAS IMAGE PORTS
kuk8aiu1zb9h DEV_ui replicated 1/1 dashishmakov/ui:latest *:9292->9292/tcp
rhq65y0tlqrf DEV_mongo replicated 1/1 mongo:3.2
td9jgtmq6buy DEV_comment replicated 1/1 dashishmakov/comment:latest
vlykfrg53f15 DEV_post replicated 1/1 dashishmakov/post:latest
- add
label
to the master node
~swarm$ docker node update --label-add reliability=high master-1
master-1
- describe all available
labels
~swarm$ docker node ls -q | xargs docker node inspect -f '{{ .ID }} [{{ .Description.Hostname }}]: {{ .Spec.Labels }}'
- update Docker Stack
~swarm$ docker stack deploy --compose-file=<(docker-compose -f docker-compose.yml config 2>/dev/null) DEV
Updating service DEV_mongo (id: rhq65y0tlqrfb5fwb110xrsts)
Updating service DEV_post (id: vlykfrg53f15t41di63ocpefo)
Updating service DEV_ui (id: kuk8aiu1zb9hxxwhrvbwjy0o0)
Updating service DEV_comment (id: td9jgtmq6buyxj7897s2sa6n4)
- check statuses of containers and scaling by all nodes
- 1x scale on master node (mongo) - no scaling
- 2x scale on worker nodes (ui, pos, comment) - replicated scaling
- 3x scale on master and worker nodes (node-exporter) - global replicated
~swarm$ docker stack ps DEV
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
mit4pxws64uk DEV_node-exporter.h4rf9on49q2v2n623ei0a2zlg prom/node-exporter:v0.15.0 worker-1 Running Running 25 seconds ago
bqn4962cey46 DEV_node-exporter.1scjwp239x4ajg5kabr1kv0wm prom/node-exporter:v0.15.0 master-1 Running Running 25 seconds ago
zhpe6t4zw4qa DEV_node-exporter.whc2xwhg3rfvbzxp3icpirnd9 prom/node-exporter:v0.15.0 worker-2 Running Running 23 seconds ago
oqsmddkcj1iq DEV_comment.1 dashishmakov/comment:latest worker-2 Running Running 12 hours ago
u699vk83e2i7 DEV_ui.1 dashishmakov/ui:latest worker-2 Running Running 12 hours ago
jic5u0vk2mkw DEV_post.1 dashishmakov/post:latest worker-1 Running Running 12 hours ago
quvmp86hzb8c DEV_mongo.1 mongo:3.2 master-1 Running Running 12 hours ago
bivw1mfilrc0 DEV_comment.2 dashishmakov/comment:latest worker-1 Running Running 23 minutes ago
fgucvzrvfbdo DEV_ui.2 dashishmakov/ui:latest worker-1 Running Running 23 minutes ago
iwgd5gjlxfmk DEV_post.2 dashishmakov/post:latest worker-2 Running Running 24 minutes ago
- add new one worker node to
swarm
~swarm$ docker-machine create --driver google \
--google-project <project_id> \
--google-zone europe-west1-b \
--google-machine-type g1-small \
--google-machine-image $(gcloud compute images list --filter ubuntu-1604-lts --uri) \
worker-3
~swarm$ docker swarm join-token worker
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-0wo27ownjz1zgw0zsbit5e7aa69r7td8a91tpvnu5m32cenbtp-bmrtyoioou9zupoc2w9sug6zv <master-1_internal_ip>:2377
~swarm$ docker-machine ssh worker-3
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.10.0-40-generic x86_64)
...
docker-user@worker-3:~$ sudo docker swarm join \
--token SWMTKN-1-0wo27ownjz1zgw0zsbit5e7aa69r7td8a91tpvnu5m32cenbtp-bmrtyoioou9zupoc2w9sug6zv \
<master-1_internal_ip>:2377
This node joined a swarm as a worker.
- check statuses of containers and scaling by all 4 nodes
~swarm$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
1scjwp239x4ajg5kabr1kv0wm * master-1 Ready Active Leader
h4rf9on49q2v2n623ei0a2zlg worker-1 Ready Active
whc2xwhg3rfvbzxp3icpirnd9 worker-2 Ready Active
pt2fhi36b036eo2yvbm8giplk worker-3 Ready Active
~swarm$ docker stack ps DEV
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
vyjf8tvz9f3n DEV_node-exporter.pt2fhi36b036eo2yvbm8giplk prom/node-exporter:v0.15.0 worker-3 Running Running 10 minutes ago
mit4pxws64uk DEV_node-exporter.h4rf9on49q2v2n623ei0a2zlg prom/node-exporter:v0.15.0 worker-1 Running Running 26 minutes ago
bqn4962cey46 DEV_node-exporter.1scjwp239x4ajg5kabr1kv0wm prom/node-exporter:v0.15.0 master-1 Running Running 26 minutes ago
zhpe6t4zw4qa DEV_node-exporter.whc2xwhg3rfvbzxp3icpirnd9 prom/node-exporter:v0.15.0 worker-2 Running Running 26 minutes ago
oqsmddkcj1iq DEV_comment.1 dashishmakov/comment:latest worker-2 Running Running 12 hours ago
u699vk83e2i7 DEV_ui.1 dashishmakov/ui:latest worker-2 Running Running 12 hours ago
jic5u0vk2mkw DEV_post.1 dashishmakov/post:latest worker-1 Running Running 12 hours ago
quvmp86hzb8c DEV_mongo.1 mongo:3.2 master-1 Running Running 12 hours ago
bivw1mfilrc0 DEV_comment.2 dashishmakov/comment:latest worker-1 Running Running about an hour ago
fgucvzrvfbdo DEV_ui.2 dashishmakov/ui:latest worker-1 Running Running about an hour ago
iwgd5gjlxfmk DEV_post.2 dashishmakov/post:latest worker-2 Running Running about an hour ago
- increase min replica values and check again
1.3) Docker Swarm has routing mesh, load balancer and decentralization architecture. It helps to route all requests between worker nodes in cluster. Do several requests on concrete host in example
~swarm$ docker-machine ip $(docker-machine ls -q)
<external_ip_master-1>
<external_ip_worker-1>
<external_ip_worker-2>
<external_ip_worker-3>
~swarm$ curl <external_ip_worker-3>:9292 | grep "Microservices Reddit in"
<a class='navbar-brand' href='/'>Microservices Reddit in DEV 03a8760bef10 container</a>
...
<a class='navbar-brand' href='/'>Microservices Reddit in DEV 3e1e5763d96e container</a>
...
<a class='navbar-brand' href='/'>Microservices Reddit in DEV 5d1bf5267ca2 container</a>
1.4) Keep all services into the file docker-compose.yml
and all infrastructure services to the file docker-compose.infra.yml
~swarm$ docker stack rm DEV
~swarm$ docker stack deploy --compose-file=<(docker-compose -f docker-compose.infra.yml -f docker-compose.yml config 2>/dev/null) DEV
At the end remove docker containers and remote instance of docker machine
~swarm$ docker stack rm DEV
~swarm$ docker-machine kill $(docker-machine ls -q)
~swarm$ docker-machine rm $(docker-machine ls -q)
Kubernetes is a system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery.
1.1) Kubernetes The Hard Way is a guide written by Kelsey Hightower, Developer Advocate from Google, Inc. It is optimized for learning to ensure you understand each task required to bootstrap a Kubernetes cluster.
1.2) Deploy test pods in Kubernetes environment at the end of tutorial before chapter Cleaning Up
~kubernetes_the_hard_way$ kubectl apply -f ./deployments/post-deployment.yml
~kubernetes_the_hard_way$ kubectl apply -f ./deployments/comment-deployment.yml
~kubernetes_the_hard_way$ kubectl apply -f ./deployments/ui-deployment.yml
~kubernetes_the_hard_way$ kubectl apply -f ./deployments/mongo-deployment.yml
~kubernetes_the_hard_way$ kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox-855686df5d-4mkr9 1/1 Running 0 18m
comment-deployment-5cdd78fdff-bpxjh 1/1 Running 0 5m
mongo-deployment-5d95d48bf8-4pbgl 1/1 Running 0 8m
nginx-8586cf59-dwj86 1/1 Running 0 16m
post-deployment-7b946f8bb5-94dsn 1/1 Running 0 6m
ui-deployment-b9d9d4d9-ggfhk 1/1 Running 0 4m
- at the end remove Kubernetes cluster and clear context
~kubernetes_the_hard_way$ kubectl config delete-context kubernetes-the-hard-way
~kubernetes_the_hard_way$ kubectl config delete-cluster kubernetes-the-hard-way
- read chapter Cleaning Up to delete compute resources created during this tutorial
Wide the knowledge about Kubernetes. Minikube is a tool that makes it easy to run Kubernetes locally. It runs a single-node Kubernetes cluster inside a VM on your desktop computer to try out Kubernetes.
1.1) Minikube need one of virtalization: VirtualBox (default), KVM, VMware Fusion, xhyve or other. Will use VirtualBox
- run small Kubernetes cluster of 1 node
~kubernetes$ minikube start <--vm-driver=virtualbox>
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.
~kubernetes$ minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100
kubectl
was configured for this cluster and let's look at nodes and pods are running in the empty cluster
~kubernetes$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready <none> 56m v1.8.0
~kubernetes$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-addon-manager-minikube 1/1 Running 0 56m
kube-system kube-dns-86f6f55dd5-dksv9 3/3 Running 0 56m
kube-system kubernetes-dashboard-lr7v4 1/1 Running 0 56m
kube-system storage-provisioner 1/1 Running 0 56m
kubectl
could manage different clusters with different users bycontext
~kubernetes$ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority: /Users/dima/.minikube/ca.crt
server: https://192.168.99.100:8443
name: minikube
contexts:
- context:
cluster: minikube
user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
user:
as-user-extra: {}
client-certificate: /Users/dima/.minikube/client.crt
client-key: /Users/dima/.minikube/client.key
~kubernetes$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* minikube minikube minikube
1.2) Run application into k8s
- apply deployments for all components
~kubernetes$ kubectl apply -f ./comment/comment-deployment.yml -n dev
deployment "comment" created
~kubernetes$ kubectl apply -f ./post/post-deployment.yml -n dev
deployment "post" created
~kubernetes$ kubectl apply -f ./ui/ui-deployment.yml -n dev
deployment "ui" created
~kubernetes$ kubectl apply -f ./mongo/mongo-deployment.yml -n dev
deployment "mongo" created
~kubernetes$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
comment-6576c99dfc-dgcls 1/1 Running 0 3m 172.17.0.6 minikube
comment-6576c99dfc-fnzpx 1/1 Running 0 3m 172.17.0.4 minikube
comment-6576c99dfc-jvrr2 1/1 Running 0 3m 172.17.0.8 minikube
mongo-95f974ff5-nqz5r 1/1 Running 0 3m 172.17.0.14 minikube
post-78f54477b9-5tt99 1/1 Running 0 3m 172.17.0.10 minikube
post-78f54477b9-rshhf 1/1 Running 0 3m 172.17.0.7 minikube
post-78f54477b9-zz8cd 1/1 Running 0 3m 172.17.0.5 minikube
ui-cd75bf6d5-4ffqx 1/1 Running 0 3m 172.17.0.13 minikube
ui-cd75bf6d5-czk8n 1/1 Running 0 3m 172.17.0.12 minikube
ui-cd75bf6d5-nqthw 1/1 Running 0 3m 172.17.0.11 minikube
~kubernetes$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
comment 3 3 3 3 2m
mongo 1 1 1 1 2m
post 3 3 3 3 2m
ui 3 3 3 3 2m
- let's check out programs health into
pods
by forwarding local port to target port
~kubernetes$ kubectl port-forward <ui-pod> 8080:9292
Forwarding from 127.0.0.1:8080 -> 9292
~kubernetes$ kubectl port-forward <post-pod> 5000:5000
Forwarding from 127.0.0.1:5000 -> 5000
~kubernetes$ kubectl port-forward <comment-pod> 9292:9292
Forwarding from 127.0.0.1:9292 -> 9292
-
open URL http://localhost:8080, http://localhost:5000/healthcheck, http://localhost:9292/healthcheck and test the apps
-
Kubernetes services help to automates port forwarding for deployments
~kubernetes$ kubectl apply -f ./ui/ui-service.yml
service "ui" created
~kubernetes$ kubectl apply -f ./post/post-service.yml
service "post" created
~kubernetes$ kubectl apply -f ./comment/comment-service.yml
service "comment" created
~kubernetes$ kubectl apply -f ./mongo/comment-mongodb-service.yml
service "comment-db" created
~kubernetes$ kubectl apply -f ./mongo/post-mongodb-service.yml
service "post-db" created
~kubernetes$ kubectl get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
comment ClusterIP 10.103.177.123 <none> 9292/TCP 28m app=reddit,component=comment
comment-db ClusterIP 10.99.107.79 <none> 27017/TCP 28m app=reddit,comment-db=true,component=mongo
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1h <none>
post ClusterIP 10.110.243.67 <none> 5000/TCP 28m app=reddit,component=post
post-db ClusterIP 10.104.224.229 <none> 27017/TCP 28m app=reddit,component=mongo,post-db=true
ui NodePort 10.96.169.222 <none> 9292:32092/TCP 28m app=reddit,component=ui
- let's check out that
ui
could save posts and comments
~kubernetes$ minikube service ui
1.3) Kubernetes provides several addons. Dashboard is one of the addons that can be used to manage a cluster from UI, get detailed information about state.
- list of addons available from Minikube
~kubernetes$ minikube addons list
- enable
dashboard
and run it
~kubernetes$ minikube addons enable dashboard
~kubernetes$ minikube dashboard
1.4) Namespaces is a virtual clusters with separate scope for names. It helps to divide cluster resources between multiple users
- create new namespace
dev
and runpods
,services
intodev
~kubernetes$ kubectl apply -f ./dev-namespace.yml
namespace "dev" created
~kubernetes$ kubectl get namespaces
NAME STATUS AGE
default Active 5h
dev Active 3m
kube-public Active 5h
kube-system Active 5h
~kubernetes$ kubectl -n dev apply -f ./deployments
~kubernetes$ kubectl -n dev apply -f ./service
- you need to change
nodePort
inui-service.yml
if you want to runpods
andservices
into seconddev
namespace paralleldefault
1.5) Stop and delete local Kubernetes cluster provided by Minikube
~kubernetes$ minikube stop
Stopping local Kubernetes cluster...
Machine stopped.
~kubernetes$ minikube delete
Deleting local Kubernetes cluster...
Machine deleted.
2.1) Google Kubernetes Engine is a managed environment for deploying containerized application.
This is service based on instances from GCE
- create Kubernetes cluster in
GCE
and connect withkubectl
~kubernetes$ gcloud container clusters create cluster-1 \
--project <project_id> \
--cluster-version 1.8.3-gke.0 \
--disk-size=20 \
--machine-type=g1-small \
--num-nodes=2 \
--no-enable-basic-auth
Creating cluster cluster-1...done.
Created [https://container.googleapis.com/v1/projects/kubernetes-188619/zones/us-west1-c/clusters/cluster-1].
kubeconfig entry generated for cluster-1.
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
cluster-1 us-west1-c 1.8.3-gke.0 35.197.95.173 g1-small 1.8.3-gke.0 2 RUNNING
~kubernetes$ gcloud container clusters get-credentials cluster-1 --zone us-west1-c --project <project_id>
Fetching cluster endpoint and auth data.
kubeconfig entry generated for cluster-1.
~kubernetes$ kubectl config current-context
gke_kubernetes-188619_us-west1-c_cluster-1
- 2 worker nodes are basic
compute engine nodes
~kubernetes$ gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
gke-cluster-1-default-pool-93c565d1-wx7r us-west1-c g1-small 10.138.0.2 35.197.70.210 RUNNING
gke-cluster-1-default-pool-93c565d1-z45m us-west1-c g1-small 10.138.0.3 35.197.6.193 RUNNING
- create custom namespace
dev
~kubernetes$ kubectl apply -f ./dev-namespace.yml
namespace "dev" created
~ kubernetes$ kubectl get namespaces
NAME STATUS AGE
default Active 24m
dev Active 11s
kube-public Active 24m
kube-system Active 24m
- deploy components and run services
~kubernetes$ kubectl apply -f ./comment -n dev
deployment "comment" created
service "comment" created
~kubernetes$ kubectl apply -f ./post -n dev
deployment "post" created
service "post" created
~kubernetes$ kubectl apply -f ./ui -n dev
deployment "ui" created
service "ui" created
~kubernetes$ kubectl apply -f ./mongo -n dev
service "comment-db" created
deployment "mongo" created
service "post-db" created
~kubernetes$ kubectl get pods -n dev -o wide
NAME READY STATUS RESTARTS AGE IP NODE
comment-b986998b4-rcltn 1/1 Running 0 2m 10.16.1.8 gke-cluster-1-default-pool-93c565d1-z45m
comment-b986998b4-vjv88 1/1 Running 0 2m 10.16.0.7 gke-cluster-1-default-pool-93c565d1-wx7r
comment-b986998b4-x9flw 1/1 Running 0 2m 10.16.0.8 gke-cluster-1-default-pool-93c565d1-wx7r
mongo-77dcb74cd5-vs48h 1/1 Running 0 1m 10.16.1.9 gke-cluster-1-default-pool-93c565d1-z45m
post-c994fb486-bq94b 1/1 Running 0 1m 10.16.1.10 gke-cluster-1-default-pool-93c565d1-z45m
post-c994fb486-cb7sh 1/1 Running 0 1m 10.16.0.10 gke-cluster-1-default-pool-93c565d1-wx7r
post-c994fb486-qfc76 1/1 Running 0 1m 10.16.0.9 gke-cluster-1-default-pool-93c565d1-wx7r
ui-759c55c666-5kqbs 1/1 Running 0 1m 10.16.1.11 gke-cluster-1-default-pool-93c565d1-z45m
ui-759c55c666-64s5m 1/1 Running 0 1m 10.16.0.11 gke-cluster-1-default-pool-93c565d1-wx7r
ui-759c55c666-jj6jn 1/1 Running 0 1m 10.16.1.12 gke-cluster-1-default-pool-93c565d1-z45m
~kubernetes$ kubectl get services -n dev -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
comment ClusterIP 10.19.248.233 <none> 9292/TCP 2m app=reddit,component=comment
comment-db ClusterIP 10.19.249.36 <none> 27017/TCP 2m app=reddit,comment-db=true,component=mongo
post ClusterIP 10.19.244.202 <none> 5000/TCP 2m app=reddit,component=post
post-db ClusterIP 10.19.251.2 <none> 27017/TCP 2m app=reddit,component=mongo,post-db=true
ui NodePort 10.19.249.191 <none> 9292:32092/TCP 2m app=reddit,component=ui
- create firewall rule to open range of ports for Kubernetes services
~kubernetes$ gcloud compute firewall-rules create default-k8s-ports\
--project=kubernetes-188619 \
--direction=INGRESS \
--priority=1000 \
--action=ALLOW \
--description="Range port for k8s" \
--rules=tcp:30000-32767 \
--source-ranges=0.0.0.0/0
~kubernetes$ gcloud compute firewall-rules list --filter k8s
NAME NETWORK DIRECTION PRIORITY ALLOW DENY
default-k8s-ports default INGRESS 1000 tcp:30000-32767
- let's check availability
ui
component in Google Kubernetes Engine
~kubernetes$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
gke-cluster-1-default-pool-2b69dddc-257l Ready <none> 1h v1.8.3-gke.0 35.225.188.134 Container-Optimized OS from Google 4.4.64+ docker://1.13.1
gke-cluster-1-default-pool-2b69dddc-zwtm Ready <none> 1h v1.8.3-gke.0 35.184.139.79 Container-Optimized OS from Google 4.4.64+ docker://1.13.1
~kubernetes$ kubectl describe service ui -n dev | grep -i nodeport
Type: NodePort
NodePort: <unset> 32092/TCP
- open URL http://<external-ip>:<node-port>
At the end remove Kubernetes cluster and clear context
~kubernetes$ gcloud container clusters delete cluster-1