integralnet-formalize-v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integral Network\n",
    "Given a vector function $F(X)=Y$ and a neural network $\\aleph(X|\\Theta)=\\hat{\\mathbf{I}}$, where $\\theta_{k} \\in \\Theta$ is the parameters of the neural network, $F(X) \\in \\mathbb{R}^{N}\\rightarrow\\mathbb{R}^{M}$, $\\aleph(X|\\Theta) \\in \\mathbb{R}^{N}\\rightarrow\\mathbb{R}^{M\\times N}$, $x_{i} \\in X$, $y_{j} \\in Y$, $\\hat{\\mathbf{i}}_{ji} \\in \\hat{\\mathbf{I}}$, denote the j-th output of $F(X)$ as $F_{j}(X)=y_{j}$, the $ji$-th output of $\\aleph(X|\\Theta)$ as $\\aleph_{ji}(X|\\Theta)=\\hat{\\mathbf{i}}_{ji}$. Define the integral matrix of $F(X)$ as $\\mathbf{I}_{F} \\in \\mathbb{R}^{M\\times N}$, which consist of all possible integrals of $F(X)$:\n",
    "$$\\mathbf{I}_{F} = \\begin{pmatrix}\n",
    "\\int y_{1}dx_{1} & \\int y_{1}dx_{2} & \\dots & \\int y_{1}dx_{N} \\\\\n",
    "\\int y_{2}dx_{1} & \\int y_{2}dx_{2} & \\dots & \\int y_{2}dx_{N} \\\\\n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "\\int y_{M}dx_{1} & \\int y_{M}dx_{2} & \\dots & \\int y_{M}dx_{N} \\\\\n",
    "\\end{pmatrix}$$\n",
    "I introduce a deep learning method to end to end approximate $\\mathbf{I}_{F}$ with individual shifting scalar constant $c_{ji} \\in C, C \\in \\mathbb{R}^{M\\times N}$:\n",
    "$$\\mathbf{I}_{F} \\approx \\aleph(X|\\Theta) + C$$\n",
    "And definite integrals can be computed as:\n",
    "$$\\int_{b}^{a}y_{j}dx_{i} \\approx \\aleph_{ji}(x_{i}=a;X|\\Theta) - \\aleph_{ji}(x_{i}=b;X|\\Theta)$$\n",
    "### Partial derivative network\n",
    "Define the partial derivative network of input $x_{i}$ as ${}_{i}\\aleph^{'}(X|\\Theta)={}_{i}\\hat{Y}$, where ${}_{i}\\aleph^{'}(X|\\Theta) \\in \\mathbb{R}^{N}\\rightarrow\\mathbb{R}^{M}$:\n",
    "$${}_{i}\\aleph^{'}(X|\\Theta) = \\frac{\\partial}{\\partial x_{i}}\\aleph_{*i}(X|\\Theta)$$\n",
    "With mean squared error loss function and cost function of partial derivative networks: $$\\mathcal{L}_{{}_{i}\\aleph^{'}}(X,Y|\\Theta)=\\frac{1}{M}\\sum_{j=1}^{M}({}_{i}\\aleph_{j}^{'}(X|\\Theta)-y_{j})^{2}$$\n",
    "$$\\mathcal{J}_{{}_{i}\\aleph^{'}}(X,Y|\\Theta)=\\mathbf{E}_{sample}[\\mathcal{L}_{{}_{i}\\aleph^{'}}(X,Y|\\Theta)]$$\n",
    "One could perform gradient descent with learning rate $\\alpha$:\n",
    "$$\\theta_{k} \\leftarrow \\theta_{k} - \\alpha\\frac{\\partial}{\\partial\\theta_{k}}\\mathcal{J}_{{}_{i}\\aleph^{'}}(X,Y|\\Theta)$$\n",
    "If the cost do converge to 0, then:\n",
    "$$F(X) \\approx {}_{i}\\aleph^{'}(X|\\Theta)$$\n",
    "Then it can be showed:\n",
    "$$\\int F_{j}(X)dx_{i} \\approx \\int{}_{i}\\aleph_{j}^{'}(X|\\Theta)dx_{i}$$\n",
    "$$\\int F_{j}(X)dx_{i} \\approx \\aleph_{ji}(X|\\Theta) + c_{ji}$$\n",
    "$$\\int y_{j}dx_{i} \\approx \\aleph_{ji}(X|\\Theta) + c_{ji}$$\n",
    "$$\\int_{b}^{a}y_{j}dx_{i} \\approx \\aleph_{ji}(x_{i}=a;X|\\Theta) - \\aleph_{ji}(x_{i}=b;X|\\Theta)$$\n",
    "One can end to end approximate the whole integral matrix $\\mathbf{I}_{F}$ with individual shifting matrix constant $C$, with $N$ partial derivative network ${}_{i}\\aleph^{'}(X|\\Theta)$ for each input $x_{i}$ to approximate the i-th integral matrix column.\n",
    "### Combining all partial derivative network into one\n",
    "Due to there are N different partial derivative network respect for N different column of the integral matrix $\\mathbf{I}_{F}$, which all N different partial derivative network regress to the same truth value Y, it is inflexible to replace Y to attaching another nerual network layer, and prevent practical end to end deep learning architecture combination in practice, here we propose a simple modification of the output, the loss function and the cost function to combining all partial derivative network into one single $M$ dimensional output, with an output divergence penalty factor $\\gamma \\in \\mathbb{R}_{>0}$:\n",
    "$$\\aleph^{'}(X|\\Theta)=\\mathbf{E}_{i}[{}_{i}\\aleph^{'}(X|\\Theta)]$$\n",
    "$$\\mathcal{L}_{\\aleph^{'}}(X,Y|\\Theta)=\\underbrace{\\mathbf{E}_{i}[\\mathcal{L}_{{}_{i}\\aleph^{'}}(X,Y|\\Theta)]}_\\text{Mean of losses} + \\gamma\\underbrace{\\sum_{j=1}^{M}\\mathbf{Var}_{i}[{}_{i}\\aleph_{j}^{'}(X|\\Theta)]}_\\text{Variance of regressions}$$\n",
    "$$\\mathcal{J}_{\\aleph^{'}}(X,Y|\\Theta)=\\mathbf{E}_{sample}[\\mathcal{L}_{\\aleph^{'}}(X,Y|\\Theta)]$$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml",
   "language": "python",
   "name": "ml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}