大数据管理与分析方法研究北京市重点实验室（BDAI）研究生沙龙由a片动画 师生组织定期举行。12月14日研讨会由学院苏冰准聘副教授指导的博士生王杰鑫介绍自己的研究工作。欢迎同学们积极参与研讨！

报告题目：Towards the Gradient Adjustment by Loss Status for Neural Network Optimization

讲者：王杰鑫，博士二年级 导师：苏冰

研究方向：优化算法、动作预测

Abstract:

Gradient descent-based algorithms are crucial in neural network optimization, and most of them only depend on local properties such as the first and second-order momentum of gradients to determine the local optimization directions. As a result, such algorithms often converge slowly in the case of a small gradient and easily fall into the local optimum. Since the goal of optimization is to minimize the loss function, the status of the loss indicates the overall progress of the optimization but has not been fully explored. In this paper, we propose a loss-aware gradient adjusting strategy (LGA) based on the loss status. LGA automatically adjusts the update magnitude of parameters to accelerate convergence and escape local optimums by introducing a loss-incentive correction term monitoring the loss and adapting gradient experience. The proposed strategy can be applied to various gradient descent-based optimization algorithms. We provide theoretical analysis on the convergence rate and empirical evaluations on different datasets to demonstrate the effectiveness of our method.