3.1_reinforcement_trading_offline_simple_derivs.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Derivatives for reinforcement trading notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given a trading system model $F_{t}(\\theta)$, the goal is to adjust the parameters $\\theta$ in order to maximise $U_{t}$ (wealth)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# $$ \\frac{dU_{T}(\\theta)}{d\\theta} = \n",
    "\\sum \\limits _{t=1} ^ {T} \n",
    "\\frac{dU_{T}}{dR_{t}} \n",
    "\\{ \\frac{dR_{t}}{dF_{t}} \\frac{dF_{t}}{d\\theta} + \\frac{dR_{t}}{dF_{t-1}} \\frac{dF_{t-1}}{d\\theta}  \\}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where:\n",
    "* $U$ = wealth function (Sharpe ratio / Sterling ratio)\n",
    "* $R_{t}$ = realised returns \n",
    "* $r_{t}$ = asset returns\n",
    "* $F_{t}$ = position  \n",
    "* $\\theta$ = model weights\n",
    "* $\\delta$ = transaction costs\n",
    "* $\\mu$ = max position (multiplied by $F_{t}$ which goes from -1 to 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### References\n",
    "\n",
    "* J. Moody and M. Saffell, \"Learning to trade via direct reinforcement,\" in IEEE Transactions on Neural Networks, vol. 12, no. 4, pp. 875-889, July 2001, doi: 10.1109/72.935097.\n",
    "\n",
    "* https://teddykoker.com/2019/06/trading-with-reinforcement-learning-in-python-part-ii-application/\n",
    "* http://cs229.stanford.edu/proj2006/Molina-StockTradingWithRecurrentReinforcementLearning.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $$\\frac{dU_{T}}{dR_{t}} $$  \n",
    "  \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *Sterling Ratio* is used as a wealth function:\n",
    "\n",
    "$$ U_{t} = SterlingRatio = \\frac{Annualized Average Return}{Maximum Drawdown}$$  \n",
    "  \n",
    "  \n",
    "  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  \n",
    "This has been approximated to the Downside Deviation Ratio (DDR):\n",
    "\n",
    "$$ DDR_{T} = \\frac{Average(R_{t})}{DD_{T}} $$\n",
    "\n",
    "Where $$DD_{T} = (\\frac{1}{T}\\sum \\limits _{t=1} ^ {T} min \\{R_{t},0\\}^2)^\\frac{1}{2} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jax.numpy as jnp\n",
    "from jax import grad, jit, vmap\n",
    "from jax import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cheat and use autograd\n",
    "\n",
    "def DDR(R, Rt):\n",
    "    _R = jnp.append(R, Rt)\n",
    "    return jnp.mean(_R) / jnp.mean(jnp.minimum(_R, 0)**2)**0.5\n",
    "\n",
    "ddr_auto = jit(grad(DDR, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Approximated for online learning as:\n",
    "The Differential Downside Deviation Ratio (DDR):\n",
    "\n",
    "$$D_{t} = \\frac{DD_{t-1}^2 \\cdot (R_{t} - \\frac{1}{2} A_{t-1}) - \\frac{1}{2} A_{t-1}R_{t}^2}{DD_{t-1}^3} $$\n",
    "\n",
    "Based on the moving average of returns (A) and squared drawdown (DD2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exponential moving average of returns:\n",
    "$$A_{t} = A_{t-1} + n(R_{t} - A_{t-1})$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_A(R, n):\n",
    "    T = len(R)\n",
    "    A = np.zeros(T)\n",
    "    \n",
    "    for t in range(1, T):\n",
    "        A[t] = A[t-1] + (n * (R[t] - A[t-1]))\n",
    "        A[t] = np.nan_to_num(A[t])\n",
    "\n",
    "    return A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exponential moving average of squared drawdown: \n",
    "$$DD_{t}^2 = DD_{t-1}^2 + n(min(R_{t},0)^2 - DD_{t-1}^2)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_DD2(R, n):\n",
    "    T = len(R)\n",
    "    DD2 = np.zeros(T)\n",
    "    for t in range(1, T):\n",
    "        DD2[t] = DD2[t-1] + n * (min(R[t],0)**2 - DD2[t-1])\n",
    "        DD2[t] = np.nan_to_num(DD2[t])\n",
    "        \n",
    "    return DD2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Differential downside deviation ratio "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sympy as sp\n",
    "\n",
    "dd_t_1, rt, A_t_1 = sp.symbols('DD_t_-1, R_t, A_t_-1')\n",
    "Dt = (dd_t_1**2 * (rt - 0.5*A_t_1) - (0.5*A_t_1*rt**2)) / dd_t_1**3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Differential downside deviation ratio (Dt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{- 0.5 A_{t -1} R_{t}^{2} + DD_{t -1}^{2} \\left(- 0.5 A_{t -1} + R_{t}\\right)}{DD_{t -1}^{3}}$"
      ],
      "text/plain": [
       "(-0.5*A_t_-1*R_t**2 + DD_t_-1**2*(-0.5*A_t_-1 + R_t))/DD_t_-1**3"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Dt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Partial derivative of Dt with respect to Rt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{- 1.0 A_{t -1} R_{t} + DD_{t -1}^{2}}{DD_{t -1}^{3}}$"
      ],
      "text/plain": [
       "(-1.0*A_t_-1*R_t + DD_t_-1**2)/DD_t_-1**3"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dDdR = sp.diff(Dt, rt)\n",
    "dDdR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $$ \\frac{dU_{T}}{dR_{t}} = \\frac{-1A_{t-1}R_{t}+DD^2_{t-1}}{DD^3_{t-1}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dUdR(A, t, R, DD2):\n",
    "    return ((-1.0*A[t-1]*R[t]) + DD2[t-1]) / DD2[t-1]**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $$\\frac{dR_{t}}{dF_{t}} = -\\mu\\delta \\cdot sgn(F_{t} - F_{t-1})$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dRdFt(delta, Ft, t):\n",
    "    return -delta * np.sign(Ft[t] - Ft[t-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $$\\frac{dF_{t}}{d\\theta} = (1-tanh(\\theta^Tx_{t})^2) \\cdot \\{ x_{t} + w_{M+2} \\frac{dF_{t-1}}{d\\theta} \\}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dFtdTheta(Ft, t, state, theta, prev):\n",
    "    return (1 - Ft[t] ** 2) * (state + theta[-1] * prev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# $$\\frac{dR_{t}}{dF_{t-1}} = \\mu \\cdot r_{t} + \\mu\\delta \\cdot sgn(F_{t}-F_{t-1})$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dRdFtp(r, delta, Ft, t):\n",
    "    return r[t] + delta * np.sign(Ft[t] - Ft[t-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DEbug"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{A}{\\sqrt{- A^{2} + B}}$"
      ],
      "text/plain": [
       "A/sqrt(-A**2 + B)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A, B = sp.symbols('A, B', real=True)\n",
    "S_T = A / sp.sqrt(B-A**2)\n",
    "S_T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{B}{\\left(- A^{2} + B\\right)^{\\frac{3}{2}}}$"
      ],
      "text/plain": [
       "B/(-A**2 + B)**(3/2)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sp.simplify(S_T.diff(A))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# deriv of sums = sum of derivs\n",
    "# https://math.stackexchange.com/a/3910714"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$A=\\frac{1}{T} \\sum_{t=1} ^{T} R_{t}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\frac{A^{2}}{\\left(- A^{2} + B\\right)^{\\frac{3}{2}}} + \\frac{1}{\\sqrt{- A^{2} + B}}$"
      ],
      "text/plain": [
       "A**2/(-A**2 + B)**(3/2) + 1/sqrt(-A**2 + B)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sp.diff(S_T, A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}