Many machine learning methods require non-linear optimization, performed by the backward propagation of model errors, with the process complicated by the presence of multiple minima and saddle points. Numerous gradient descent algorithms are available for optimization, including stochastic gradient descent, conjugate gradient, quasi-Newton and non-linear least squares such as Levenberg-Marquardt. In contrast to deterministic optimization, stochastic optimization methods repeatedly introduce randomness during the search process to avoid getting trapped in a local minimum. Evolutionary algorithms, borrowing concepts from evolution to solve optimization problems, include genetic algorithm and differential evolution.

Although there are many ways of describing nonlinear relationships among variables, this chapter focuses primarily on the polynomial regression, which is related to the multiple linear regression model. We pay particular attention to models using the second-order polynomial. These models are often employed in the field of community ecology to describe unimodal changes of species abundances along environmental gradients. The downsides of using polynomial regression are also addressed. We bring this chapter to a close by touching on the non-linear least-squares regression models and the appropriate context in which they should be applied. The methods described in this chapter are accompanied by a carefully-explained guide to the R code needed for their use, including the nlme package.