Toward AGI: Google Deepmind's New Learning Model

Researchers at Google DeepMind have come up with “Nested Learning”. This could solve the problem of catastrophic forgetting in models, where a model which is trained on one set of tasks forgets all of its learnings by the time it comes to the next set of tasks. If continuous learning works, these models are going to gain general knowledge, and it's an important step towards AGI. Current LLMs have no separation between memory and learning. The internal neural network for a LLM has its weights updated only through the Attention Mechanism, meaning the rate at which we update our current context and long-term memory is the same. In contrast, humans learn at different frequencies, meaning short-term, current problems are stored in short-term memory, and after a good night's sleep, The general ideas in important memory is moved to the next layer. These would be called repeatable skills. In this way we have progressive layers depending on the importance and frequency of the learnings. Google is trying to mimic this behavior using multi-layer gradient flow where the lowest layer is updated continuously and higher layers are updated less frequently. Since these higher layers are abstractions of general concepts, learning new things does not lead to catastrophic forgetting. And the results are promising, with the model having attained higher reasoning scores as compared to Transformers or even Titans. Ironically, the researchers have named this model... HOPE. Thanks for watching. Cheers.