Kronecker-factored Approximate Curvature with adaptive learning rate for optimizing model-agnostic meta-learning

Model-agnostic meta-learning (MAML) highlights the ability to quickly adapt to new tasks with only a small amount of labeled training data among many few-shot learning algorithms. However, the computational complexity is high, because the MAML algorithm generates a large number of second-order parameters in the secondary gradient update. In addition, due to the non-convex nature of the neural network, the loss landscape has many flat areas, leading to slow convergence during training, and excessively long training. In this paper, a second-order optimization method called Kronecker-factored Approximate Curvature (K-FAC) is proposed to approximate Natural Gradient Descent. K-FAC reduces the computational complexity by approximating the large matrix of the Fisher information as the Kronecker product of two much smaller matrices, and the second-order parameter information is fully utilized to accelerate the convergence. Moreover, in order to solve the problem that Natural Gradient Descent is sensitive to the learning rate, this paper proposes Kronecker-factored Approximate Curvature with adaptive learning rate for optimizing model-agnostic meta-learning (AK-MAML), which automatically adjusts the learning rate according to the curvature and improves the efficiency of training. Experimental results show that AK-MAML has the ability of faster convergence, lower computation, and higher accuracy on few-shot datasets.