Skip to content

0.8. Designing Server Architectures

Jump to bottom
Jim Schwoebel edited this page Dec 2, 2018 · 1 revision

This section documents all the scripts in the Chapter_8_server folder.

Chapter 8 setup instructions

This is the only chapter with custom installation requirements. Here are some step-by-step instructions on how to install and configure everything.

MongoDB

You can instal mongodb with homebrew:

brew install mongodb
mkdir -p /data/db
sudo chmod 777 /data/db

All you need to run a local server is to type in this into the CLI:

mongod

Kafka

You can install kafka with homebrew:

brew install kafka

To run a kafka server all you need to do is:

# run zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties

# run kafka 
bin/kafka-server-start.sh config/server.properties

Robo 3T

You can install Robo 3T directly from website: https://robomongo.org/download

Docker

You can install Docker directly from website: https://docs.docker.com/docker-for-mac/install/

Kubernetes

To install kubernetes, use homebrew:

brew install kubernetes-cli

To run kubernetes:

kubectl run [imagename]

Google cloud SDK

You can install the Google Cloud SDK download and follow instructions on website: https://cloud.google.com/sdk/docs/quickstart-macos

Python web frameworks

flask server

flask_project.py

from flask import Flask
application = Flask(__name__)

@application.route("/")
def hello():
    return "<h1 style='color:blue'>Hello There!</h1>"

if __name__ == "__main__":
    application.run(host='0.0.0.0')

django server

django_server.py

import os

def dirtree(rootDir): 
    list_dirs = os.walk(rootDir) 
    for root, dirs, files in list_dirs: 
        for d in dirs: 
            print(os.path.join(root, d))     
        for f in files: 
            print(os.path.join(root, f))
            
# make a django project in the current directory 
os.system('django-admin startproject django_server')

print('CREATING FILES...')
print(dirtree(os.getcwd()+'/django_server'))

print('lauching server...')
os.chdir('django_server')
os.system('python3 manage.py runserver')

MongoDB databases

Here are some ways to use the pymongo module:

import pymongo
from pymongo import MongoClient
import datetime
from bson.objectid import ObjectId

####################################################
##               MAKING MONGO CLIENT              ##
####################################################
# check server status
client = MongoClient()
client = MongoClient('localhost', 27017)
# or client = MongoClient('mongodb://localhost:27017/')
print(db.command('serverStatus'))

####################################################
##               MAKE TEST DATABASE               ##
####################################################

# make a test database
print('made test database') 
db = client.test_database
db = client['test-database']

####################################################
##               INSERT DATA IN DB               ##
####################################################
post = {"author": "Mike",
       "text": "My first blog post!",
        "tags": ["mongodb", "python", "pymongo"],
        "date": datetime.datetime.utcnow()}


posts = db.posts
post_id = posts.insert_one(post).inserted_id
print(post_id)
print(post)

new_posts = [{"author": "Mike",
              "text": "Another post!",
              "tags": ["bulk", "insert"],
              "date": datetime.datetime(2009, 11, 12, 11, 14)},
             {"author": "Eliot",
              "title": "MongoDB is fun",
              "text": "and pretty easy too!",
              "date": datetime.datetime(2009, 11, 10, 10, 45)}]

result=posts.insert_many(new_posts)
print(result.inserted_ids)
print(new_posts)

####################################################
##           LIST ALL COLLECTIONS IN DB           ##
####################################################

print(db.collection_names(include_system_collections=False))


####################################################
##           INDEXING / FINDING BY IDs            ##
####################################################

# first document
print(posts.find_one())
# a specific request
posts.find_one({"author": "Eliot"})
# note that posts.find_one({“_id”:str(post_id)})) will not work 
posts.find_one({"_id":post_id})

# The web framework gets post_id from the URL and passes it as a string
def get(post_id):
    # Convert from string to ObjectId:
    document = client.db.collection.find_one({'_id': ObjectId(post_id)})

####################################################
##           COUNTING NUMBER OF DOCS IN DB       ##
####################################################
    
posts.count()
posts.find({"author":"Mike"}).count()

####################################################
##           SORT DOCS VIA EXPRESSIONS            ##
####################################################

d = datetime.datetime(2009, 11, 12, 12)
for post in posts.find({"date": {"$lt": d}}).sort("author"):
    print(post)

####################################################
##           CUSTOM SCHEMAS TO HAVE UNIQUENESS    ##
####################################################
    
result = db.profiles.create_index([('user_id', pymongo.ASCENDING)],
                                                    unique=True)
sorted(list(db.profiles.index_information()))
user_profiles = [
    {'user_id': 211, 'name': 'Luke'},
    {'user_id': 212, 'name': 'Ziltoid'}]
result = db.profiles.insert_many(user_profiles)
new_profile = {'user_id': 213, 'name': 'Drew'}
duplicate_profile = {'user_id': 212, 'name': 'Tommy'}
# This is fine.
result = db.profiles.insert_one(new_profile)  
# Results in error 
result = db.profiles.insert_one(duplicate_profile)

Building kafka microservices

making a consumer

kakfa_consumer.py

from kafka import KafkaConsumer

topic='test'
consumer = KafkaConsumer(topic, bootstrap_servers='localhost:9092')

# creates a running loop to listen for messages 
for msg in consumer:
	print(msg)

making a producer

kafka_producer.py

from kafka import KafkaProducer
import time 

producer = KafkaProducer(bootstrap_servers='localhost:9092')
topic='test'

for i in range(30):
	print('sending message...%s'%(str(i)))
	producer.send(topic, b'test2')
	time.sleep(1)

print(producer.metrics())

Minio as a wrapper for GCP/AWS

minio.py

# Import Minio library.
from minio import Minio
from minio.error import (ResponseError, BucketAlreadyOwnedByYou,
                         BucketAlreadyExists)

# Initialize minioClient with an endpoint and access/secret keys.
minioClient = Minio('play.minio.io:9000',
                    access_key='Q3AM3UQ867SPQQA43P2F',
                    secret_key='zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG',
                    secure=True)

# Make a bucket with the make_bucket API call.
try:
       minioClient.make_bucket("maylogs", location="us-east-1")
except BucketAlreadyOwnedByYou as err:
       pass
except BucketAlreadyExists as err:
       pass
except ResponseError as err:
       raise
else:
        # Put an object 'pumaserver_debug.log' with contents from 'pumaserver_debug.log'.
        try:
               minioClient.fput_object('maylogs', 'pumaserver_debug.log', '/tmp/pumaserver_debug.log')
        except ResponseError as err:
               print(err)

Authentication with Auth0

auth0.py

from auth0.v3.authentication import GetToken
from auth0.v3.management import Auth0

# obtain a management token 
domain = 'myaccount.auth0.com'
non_interactive_client_id = 'exampleid'
non_interactive_client_secret = 'examplesecret'

get_token = GetToken(domain)
token = get_token.client_credentials(non_interactive_client_id,
    non_interactive_client_secret, 'https://{}/api/v2/'.format(domain))
mgmt_api_token = token['access_token']

# use your management token 
auth0 = Auth0(domain, mgmt_api_token)

#The Auth0() object is now ready to take orders! Let's see how we can use this to get all available connections. (this action requires the token to have the following scope: read:connections)
print(auth0.connections.all())

Working with docker containers

Check out this docker and python intro talk.

Unit test and integration tests

unit_test.py

import unittest

class SimplisticTest(unittest.TestCase):

    def test(self):
        a = 'a'
        b = 'a'
        self.assertEqual(a, b)
 
if __name__ == '__main__':
    unittest.main()

Resources

If you are interested to read more on any of these topics, check out the documentation below.

Web frameworks

Databases

Kafka microservices

File storage

Containers

Authentication

Unit and integration tests

Deploying software applications

Clone this wiki locally