Tools used in this project

• Poetry: Dependency management - article
• hydra: Manage configuration files - article
• Prefect: Orchestration - article
• pre-commit plugins: Automate code reviewing formatting - article
• pdoc: Automatically create an API documentation for your project

Project structure

.
├── config                      
│   ├── main.yaml                   # Main configuration file
│   ├── model                       # Configurations for training model
│   │   ├── model.yaml             # First variation of parameters to train model
├── data            
│   ├── processed                   # data after processing
│   ├── raw                         # data before pricessing
│   └── features_data               # data before modelling
├── docs                            # documentation
├── .flake8                         # configuration for flake8 - a Python formatter tool
├── .gitignore                      # ignore files that cannot commit to Git
├── Makefile                        # store useful commands to set up the environment
├── models                          # store models
├── notebooks                       # store notebooks
├── .pre-commit-config.yaml         # configurations for pre-commit
├── pyproject.toml                  # dependencies for poetry
├── README.md                       # describe your project
├── Dockerfile                      # Dockerfile
├── src                             # source code
│   ├── __init__.py                 # make src a Python module 
│   ├── process.py                  # process data before training model
│   └── feature_engineer.py         # create features
│   └── feature_main.py             # run process and feature engineer
│   └── train_model.py              # train model
└── tests                           # store tests
    ├── __init__.py                 # make tests a Python module 
    ├── test_process.py             # test functions for process.py
└── app                             # app for docker
    ├── main.py                     # FastAPI module with sklearn pipelines
    ├── serve.py                    

Set up the environment

1. Install Poetry
2. Set up the environment:

make install
make activate

3. To persist the output of Prefect's flow, run

export PREFECT__FLOWS__CHECKPOINTING=true

Run the Project

To run all flows, type:

python src/feature_main.py
python src/train_model.py

Run Tests

To run all flows, type:

make test

Run FastAPI through Docker

1. Build the Docker Image

docker build -t testliodocker ./  

2. Run the Docker Image

docker run -d --name testliodocker -p 80:80 testliodocker

3. Test the API-Go to http://127.0.0.1/docs#/default/predict_predict_post