DIY_Deep_Learning

DIY(Do it youself) Deep Learning is a python package for our own deep learning framework using pure Numpy, highlights are:

• Education focus: Main purpose of this package is obtaining deep understanding of modern deep learning frameworks like Tensorflow and PyTorch, so we will implement most of deep learning functions on our own, like convolution, batch normalization, dropout, etc
• High speed: GPU acceleration is used to speed up our code, GPU Python and CuPy make our package comparablely fast as PyTorch
• High reliability: All important functions come with a unit test, corner cases are considered to ensure high reliability
• High availability: Out of box example code is provided for all important functions for easy usage
• High scalability: Everyone can contribute his/her awesome code to this repository, scaling up the ability of this package. Further questions and communication can also be sent to email: 940417022@qq.com
• High productivity: Life is short, I use python! Get away from time-consuming and tedious C/C++ code for high productivity.
• Complete documentation: Professional document tool pdoc is massively used to help users master every important function

Limitations are:

• Non-distributed: Until now, multi-GPU traning is not supported.

Table of contents:

• DIY_Deep_Learning
• Why should we DIY deep learning ?
• How to get started ?
  • Prerequisites
  • Overview on technology roadmap
• Development plan
• Tutorials and references

Why should we DIY deep learning ?

There are two important reasons:

1. PyTorch/Tensorflow blocks you from mastering deep learning

“What I Cannot Create, I Do Not Understand”
--Richard Feynman, Nobel prize winning Physicist

If you merely use PyTorch/Tensorflow for deep learning, it's unlikely that you will master deep learning quite well. For reasons, let's consider two cases: (1) You only use PyTorch/Tensorflow API and don't like reading API's source code, in this case you may just know something about API's math or principle, but you don't know how it actually works; (2) You use PyTorch/Tensorflow API and often read API's source code, in this case, you will find that the source code is really hard to read and understand, because most source code is written in C/C++, for example:

• Source code of Batch normalization in pytorch is here:

template<typename scalar_t>
void batch_norm_cpu_inference_contiguous(Tensor& output, const Tensor& input,
    const Tensor& weight /* optional */, const Tensor& bias /* optional */,
    const Tensor& mean, const Tensor& variance, double eps) {
  int64_t n_batch = input.size(0);
  int64_t n_channel = input.size(1);
  int64_t image_size = input.numel() / n_batch / n_channel;

  scalar_t* output_data = output.data_ptr<scalar_t>();
  const scalar_t* input_data = input.data_ptr<scalar_t>();
  const scalar_t* weight_data = weight.defined() ? weight.data_ptr<scalar_t>() : nullptr;
  const scalar_t* bias_data = bias.defined() ? bias.data_ptr<scalar_t>() : nullptr;
  const scalar_t* mean_data = mean.data_ptr<scalar_t>();
  const scalar_t* var_data = variance.data_ptr<scalar_t>();

  Tensor alpha = at::empty_like(mean);
  Tensor beta = at::empty_like(mean);
  scalar_t* alpha_data = alpha.data_ptr<scalar_t>();
  scalar_t* beta_data = beta.data_ptr<scalar_t>();
  for (int64_t c = 0; c < n_channel; c++) {
    scalar_t inv_var = 1 / std::sqrt(var_data[c] + static_cast<scalar_t>(eps));
    scalar_t weight_v = weight_data ? weight_data[c] : 1;
    scalar_t bias_v = bias_data ? bias_data[c] : 0;
    alpha_data[c] = inv_var * weight_v;
    beta_data[c] = bias_v - mean_data[c] * inv_var * weight_v;
  }
  ...

it is really frustrating for people not familiar with C++, mkldnn, cudnn, torch c lib etc. The time cost to understand these c++ code is prohibitive for most of us.

2. It's not easy to customize Pytorch/Tensorflow for your specific purpose

Suppose you have to deploy your deep learning models on AIot devices, only a subset and customized version of Pytorch/Tensorflow is needed, it would be time-consuming and risky to achieve your goal.

How to get started ?

Prerequisites

1. Passion for deep learning: The tour to build a customized and reliable deep learning package is time-consuming and challenging, we will not get success without the passion for deep learning.

2. Basic knowledge about deep learning: You need to know the basics of deep learning, like convolution, batch normalization etc. Andrew Ng's deep learning courses are highly recommended: here

Overview on technology roadmap

Building a customized and reliable deep learning package is a huge workload, which is comprehensive, complex and challenging. Therefore, survey was performed on all popular open-source deep learning frameworks, important metrics are:

• Focus: main focus of the framework, like speed or clean code;
• Popularity: we use Github star number for the sake of simplicity, on 2022-5-25;
• Backprop: how the backpropagation of networks was implemented

Framework	Focus	Popularity	Backprop
Tensorflow	Industrial usage	165 K	Jax
PyTorch	Research and industrial usage	56.2 K	C++
MXNet	Portable, efficient and scalable	20 K	C++
Deeplearning4j	JVM support	12.5 K	C++, JAVA
ONNX	Unified format of ML model	12.5 K	-
Chainer(not maintained)	Flexibility	5.7 K	Python and C++
Trax	Clear code and speed	5.7 K	Jax
ML-From-Scratch	Pure Numpy, education focus	21.1 K	Pure numpy
NumpyDL	Pure Numpy, education focus	203	Pure numpy
deepnet	Pure Numpy, education focus	308	Pure numpy
MyDeepLearning	Pure Numpy, education focus	291	Pure numpy
xshinnosuke	Pure Numpy, Dynamic Graph, GPU usage	266	Pure numpy
TensorLayer	Pure Python	7K	backends like Tensorflow

Based on the survey above, we plan to develop our package based on five packages:

• ML-From-Scratch
• NumpyDL
• deepnet
• MyDeepLearning
• xshinnosuke

Detailed technology roadmap is here

Development plan

• 2022-6-2 to 2022-6-8: output the fisrt version V0.0.1 which support network of liner layers, cupy acceleration, pdoc documentation
• 2022-5-26 to 2022-6-1: get familar with existing reference frameworks
• 2022-5-23 to 2022-5-25: conduct a survey on existing open-source deep learning frameworks
• 2022-5-22: initial readme.md including: Why should we DIY deep learning and Features and principles of this package

Tutorials and references

1. Deep learning courses by Andrew Ng: here

2. Machine learning using numpy: here