Code for paper Statistical mechanics of continual learning: variational principle and mean-field potential (PhysRevE.108.014309). Here, we focus on the continual learning in single-layered and multi-layered neural networks of binary weights. A variational Bayesian learning setting is thus proposed, where the neural network is trained in a field-space, rather than the gradient-ill-defined discrete-weight space, and furthermore, the weight uncertainty is naturally incorporated, and modulates the synaptic resources among tasks. From a physics perspective, we translate the variational continual learning into the Franz-Parisi thermodynamic potential framework, where the previous task knowledge acts as a prior and a reference as well. Therefore, the learning performance can be analytically studied with mean-field order parameters, whose predictions coincide with the numerical experiments using stochastic gradient descent methods. Our proposed principled frameworks also connect to elastic weight consolidation, and neuroscience inspired metaplasticity, providing a theory-grounded method for the real-world multi-task learning with deep networks.
Python 3.8
-
The original datasets MNIST and Fashion MNIST are not uploaded.
-
The codes are arranged according to the sections of the original paper. In the Folder 'Continual learning in deep neural networks', 'Bayesian-Permuated.py' shows code for the permuated MNIST classification task, while 'Bayesian-Sequential.py' shows code for the sequential learning between MNIST and Fashion MNIST classification task.
THE MNIST DATABASE of handwritten digits.
THE FASHION MNIST DATABASE of handwritten digits
This code is the product of work carried out by the group of PMI lab, Sun Yat-sen University. If the code helps, consider giving us a shout-out in your publications.
If you have any question, please contact me via lich89@mail2.sysu.edu.cn.