Lifelong Learning with Dynamically Expandable Networks

• Jaehong Yoon(KAIST, AItrics), Eunho Yang(KAIST, AItrics), Jeongtae Lee(UNIST), and Sung Ju Hwang(KAIST, AItrics)

This project hosts the code for our ICLR 2018 paper.

We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks. DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them. We validate DEN on multiple public datasets in lifelong learning scenarios on multiple public datasets, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch model with substantially fewer number of parameters.

Reference

If you use this code as part of any published research, please refer the following paper.

@inproceedings{yoon2018lifelong,
  title={Lifelong Learning with Dynamically Expandable Networks},
  author={Yoon, Jaehong and Yang, Eunho and Lee, Jeongtae and Hwang, Sung Ju},
  year={2018},
  publisher={ICLR}
}

Running Code

We implemented the model as described in the paper based on Tensorflow library, Tensorflow.

Get our code

git clone --recursive https://github.com/jaehong-yoon93/DEN.git DEN

Run examples

In this code, you can run our model on MNIST dataset with permutation. Then, you don't need to download dataset on your own, just you get the dataset when you run our code.

For convenience, we added the logs that are printed out validation & test accuracy, and several process. If you execute DEN_run.py, you can reproduce our model.

python DEN_run.py

Authors

Jaehong Yoon^13*, Eunho Yang¹³, Jeongtae Lee², and Sung Ju Hwang¹³

¹KAIST @ School of Computing, KAIST, Daejeon, South Korea

²UNIST @ School of Electrical and Computer Engineering, UNIST, Ulsan, South Korea

³AItrics @ Seoul, South Korea

^* Work done while at UNIST