On robustness of generative representations against catastrophic forgetting

A schematic overview of our main experiment. Representations learned by the generative model are less susceptible to catastrophic forgetting than discriminative ones. Results of the main experiment and details can be found in Sec. 3.1.

Abstract

Catastrophic forgetting of previously learned knowledge while learning new tasks is a widely observed limitation of contemporary neural networks. Although many continual learning methods are proposed to mitigate this drawback, the main question remains unanswered: what is the root cause of catastrophic forgetting? In this work, we aim at answering this question by posing and validating a set of research hypotheses related to the specificity of representations built internally by neural models. More specifically, we design a set of empirical evaluations that compare the robustness of representations in discriminative and generative models against catastrophic forgetting. We observe that representations learned by discriminative models are more prone to catastrophic forgetting than their generative counterparts, which sheds new light on the advantages of developing generative models for continual learning. Finally, our work opens new research pathways and possibilities to adopt generative models in continual learning beyond mere replay mechanisms.

Publication
In International Conference on Neural Information Processing 2021 Proceedings
Wojciech Masarczyk
Wojciech Masarczyk
PhD Student
Kamil Deja
Kamil Deja
Assistant Professor
Tomasz Trzciński
Tomasz Trzciński
Principal Investigator

Related