guide-to-pytorch-learning-rate-scheduling.ipynb

{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# 1. LAMBDA LR\n\nSets the learning rate of each parameter group to the initial lr times a given function. When last_epoch=-1, sets initial lr as lr.\n\n$$\nl r_{\\text {epoch}} = l r_{\\text {initial}} * Lambda(epoch)\n$$\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=100)\nlambda1 = lambda epoch: 0.65 ** epoch\nscheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda1)\n\n\nlrs = []\n\nfor i in range(10):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \", round(0.65 ** i,3),\" , Learning Rate = \",round(optimizer.param_groups[0][\"lr\"],3))\n    scheduler.step()\n\nplt.plot(range(10),lrs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. MultiplicativeLR\n\nMultiply the learning rate of each parameter group by the factor given in the specified function. When last_epoch=-1, sets initial lr as lr.\n\n$$\nl r_{\\text {epoch}} = l r_{\\text {epoch - 1}} * Lambda(epoch)\n$$","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=100)\nlmbda = lambda epoch: 0.65 ** epoch\nscheduler = torch.optim.lr_scheduler.MultiplicativeLR(optimizer, lr_lambda=lmbda)\nlrs = []\n\nfor i in range(10):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \",0.95,\" , Learning Rate = \",optimizer.param_groups[0][\"lr\"])\n    scheduler.step()\n\nplt.plot(range(10),lrs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. StepLR\n\nDecays the learning rate of each parameter group by gamma every step_size epochs. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.\n\n$$\nl r_{\\text {epoch}}=\\left\\{\\begin{array}{ll}\nGamma * l r_{\\text {epoch - 1}}, & \\text { if } {\\text {epoch % step_size}}=0 \\\\\nl r_{\\text {epoch - 1}}, & \\text { otherwise }\n\\end{array}\\right.\n$$","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=100)\nscheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=2, gamma=0.1)\nlrs = []\n\nfor i in range(10):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \",0.1 if i!=0 and i%2!=0 else 1,\" , Learning Rate = \",optimizer.param_groups[0][\"lr\"])\n    scheduler.step()\n\nplt.plot(range(10),lrs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. MultiStepLR\n\nDecays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.\n\n$$\nl r_{\\text {epoch}}=\\left\\{\\begin{array}{ll}\nGamma * l r_{\\text {epoch - 1}}, & \\text { if } {\\text{ epoch in [milestones]}} \\\\\nl r_{\\text {epoch - 1}}, & \\text { otherwise }\n\\end{array}\\right.\n$$","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=100)\nscheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[6,8,9], gamma=0.1)\nlrs = []\n\nfor i in range(10):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \",0.1 if i in [6,8,9] else 1,\" , Learning Rate = \",optimizer.param_groups[0][\"lr\"])\n    scheduler.step()\n\nplt.plot(range(10),lrs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. ExponentialLR\n\nDecays the learning rate of each parameter group by gamma every epoch. When last_epoch=-1, sets initial lr as lr.\n\n$$\nl r_{\\text {epoch}}= Gamma * l r_{\\text {epoch - 1}}\n$$\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=100)\nscheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.1)\nlrs = []\n\n\nfor i in range(10):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \",0.1,\" , Learning Rate = \",optimizer.param_groups[0][\"lr\"])\n    scheduler.step()\n\nplt.plot(lrs)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. CosineAnnealingLR\n\nSet the learning rate of each parameter group using a cosine annealing schedule.\nWhen last_epoch=-1, sets initial lr as lr. Notice that because the schedule is defined recursively, the learning rate can be simultaneously modified outside this scheduler by other operators. If the learning rate is set solely by this scheduler, the learning rate at each step becomes:\n\n$$\n\\eta_{t}=\\eta_{\\min }+\\frac{1}{2}\\left(\\eta_{\\max }-\\eta_{\\min }\\right)\\left(1+\\cos \\left(\\frac{T_{c u r}}{T_{\\max }} \\pi\\right)\\right)\n$$\n\nIt has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts. Note that this only implements the cosine annealing part of SGDR, and not the restarts.https://arxiv.org/abs/1608.03983","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=100)\nscheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10, eta_min=0)\nlrs = []\n\n\nfor i in range(100):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \",i,\" , Learning Rate = \",optimizer.param_groups[0][\"lr\"])\n    scheduler.step()\n\nplt.plot(lrs)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. CyclicLR - triangular\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=0.1)\nscheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.001, max_lr=0.1,step_size_up=5,mode=\"triangular\")\nlrs = []\n\n\nfor i in range(100):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \",i,\" , Learning Rate = \",optimizer.param_groups[0][\"lr\"])\n    scheduler.step()\n\nplt.plot(lrs)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. CyclicLR - triangular2\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=0.1)\nscheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.001, max_lr=0.1,step_size_up=5,mode=\"triangular2\")\nlrs = []\n\n\nfor i in range(100):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \",i,\" , Learning Rate = \",optimizer.param_groups[0][\"lr\"])\n    scheduler.step()\n\nplt.plot(lrs)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. CyclicLR - exp_range\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=100)\nscheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.001, max_lr=0.1,step_size_up=5,mode=\"exp_range\",gamma=0.85)\nlrs = []\n\n\nfor i in range(100):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \",i,\" , Learning Rate = \",optimizer.param_groups[0][\"lr\"])\n    scheduler.step()\n\nplt.plot(lrs)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 8.OneCycleLR - cos\n\nSets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates.\n\nThe 1cycle learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training.\n\nThis scheduler is not chainable.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=0.1)\nscheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.1, steps_per_epoch=10, epochs=10)\nlrs = []\n\n\nfor i in range(100):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \",i,\" , Learning Rate = \",optimizer.param_groups[0][\"lr\"])\n    scheduler.step()\n\nplt.plot(lrs)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 8.OneCycleLR - linear","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=0.1)\nscheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.1, steps_per_epoch=10, epochs=10,anneal_strategy='linear')\nlrs = []\n\n\nfor i in range(100):\n    optimizer.step()\n    lrs.append(optimizer.param_groups[0][\"lr\"])\n#     print(\"Factor = \",i,\" , Learning Rate = \",optimizer.param_groups[0][\"lr\"])\n    scheduler.step()\n\nplt.plot(lrs)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 9.CosineAnnealingWarmRestarts\nSet the learning rate of each parameter group using a cosine annealing schedule, and restarts after Ti epochs.\n\n\n$$\n\\eta_{t}=\\eta_{\\min }+\\frac{1}{2}\\left(\\eta_{\\max }-\\eta_{\\min }\\right)\\left(1+\\cos \\left(\\frac{T_{\\operatorname{cur}}}{T_{i}} \\pi\\right)\\right)\n$$\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport matplotlib.pyplot as plt\n\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=0.1)\nlr_sched = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=10, T_mult=1, eta_min=0.001, last_epoch=-1)\n\n\nlrs = []\n\nfor i in range(100):\n    lr_sched.step()\n    lrs.append(\n        optimizer.param_groups[0][\"lr\"]\n    )\n\nplt.plot(lrs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport matplotlib.pyplot as plt\n\nmodel = torch.nn.Linear(2, 1)\noptimizer = torch.optim.SGD(model.parameters(), lr=0.1)\nlr_sched = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=10, T_mult=2, eta_min=0.01, last_epoch=-1)\n\n\nlrs = []\n\nfor i in range(300):\n    lr_sched.step()\n    lrs.append(\n        optimizer.param_groups[0][\"lr\"]\n    )\n\nplt.plot(lrs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}