A Nonmonotone Line Search Method for Stochastic Optimization Problems

Abstract

A non-monotone line search method for solving unconstrained optimization problems with the objective function in the form of mathematical expectation is proposed and analyzed. The method works with approximate values of the objective function obtained with increasing sample sizes and improves accuracy gradually.  Non-monotone rule significantly enlarges the set of admissible search directions and prevents unnecessarily small steps at the beginning of the iterative procedure. The convergence is shown for any search direction that approaches the negative gradient in the limit. The convergence results are obtained in the sense of  zero upper density. Initial numerical results confirm theoretical results and show efficiency of the proposed approach.