Consensus-based optimisation (CBO) is a versatile multi-particle metaheuristic optimisation method suitable for performing non-convex and non-smooth global optimisations in high dimensions. It has proven effective in various applications while at the same time being amenable to a theoretical convergence analysis. In this paper, we explore a variant of CBO, which incorporates truncated noise in order to enhance the well-behavedness of the statistics of the law of the dynamics. By introducing this additional truncation in the noise term of the CBO dynamics, we achieve that, in contrast to the original version, higher moments of the law of the particle system can be effectively bounded. As a result, our proposed variant exhibits enhanced convergence performance, allowing in particular for wider flexibility in choosing the noise parameter of the method as we confirm experimentally. By analysing the time evolution of the Wasserstein-$2$ distance between the empirical measure of the interacting particle system and the global minimiser of the objective function, we rigorously prove convergence in expectation of the proposed CBO variant requiring only minimal assumptions on the objective function and on the initialisation. Numerical evidences demonstrate the benefit of truncating the noise in CBO.

Solving a nonsmooth and nonconvex minimization problem can be approached as finding a zero of a set-valued operator. With this perspective, we propose a novel Majorizer–Minimizer technique to find a local minimizer of a nonsmooth and nonconvex function and establish its convergence. Our approach leverages Bregman distances to generalize the classical quadratic regularization. By doing so, we generate a family of regularized problems that encompasses quadratic regularization as a special case. To further demonstrate the effectiveness of our method, we apply it on a lasso regression model, showcasing its performance.

In this paper, we study consensus-based optimisation (CBO), a versatile, flexible and customisable optimisation method suitable for performing nonconvex and nonsmooth global optimisations in high dimensions. CBO is a multi-particle metaheuristic, which is effective in various applications and at the same time amenable to theoretical analysis thanks to its minimalistic design. The underlying dynamics, however, is flexible enough to incorporate different mechanisms widely used in evolutionary computation and machine learning, as we show by analysing a variant of CBO which makes use of memory effects and gradient information. We rigorously prove that this dynamics converges to a global minimiser of the objective function in mean-field law for a vast class of functions under minimal assumptions on the initialisation of the method. The proof in particular reveals how to leverage further, in some applications advantageous, forces in the dynamics without loosing provable global convergence. To demonstrate the benefit of the herein investigated memory effects and gradient information in certain applications, we present numerical evidence for the superiority of this CBO variant in applications such as machine learning and compressed sensing, which en passant widen the scope of applications of CBO.

Narrow escape and narrow capture problems which describe the average times required to stop the motion of a randomly travelling particle within a domain have applications in various areas of science. While for general domains, it is known how the escape time decreases with the increase of the trap sizes, for some specific 2D and 3D domains, higher-order asymptotic formulas have been established, providing the dependence of the escape time on the sizes and locations of the traps. Such results allow the use of global optimisation to seek trap arrangements that minimise average escape times. In a recent paper (Iyaniwura (2021) SIAM Rev. 63(3), 525–555), an explicit size- and trap location-dependent expansion of the average mean first passage time (MFPT) in a 2D elliptic domain was derived. The goal of this work is to systematically seek global minima of MFPT for $1\leq N\leq 50$ traps in elliptic domains using global optimisation techniques and compare the corresponding putative optimal trap arrangements for different values of the domain eccentricity. Further, an asymptotic formula for the average MFPT in elliptic domains with N circular traps of arbitrary sizes is derived, and sample optimal configurations involving non-equal traps are computed.

Loss functions with a large number of saddle points are one of the major obstacles for training modern machine learning (ML) models efficiently. First-order methods such as gradient descent (GD) are usually the methods of choice for training ML models. However, these methods converge to saddle points for certain choices of initial guesses. In this paper, we propose a modification of the recently proposed Laplacian smoothing gradient descent (LSGD) [Osher et al., arXiv:1806.06317], called modified LSGD (mLSGD), and demonstrate its potential to avoid saddle points without sacrificing the convergence rate. Our analysis is based on the attraction region, formed by all starting points for which the considered numerical scheme converges to a saddle point. We investigate the attraction region’s dimension both analytically and numerically. For a canonical class of quadratic functions, we show that the dimension of the attraction region for mLSGD is $\lfloor (n-1)/2\rfloor$, and hence it is significantly smaller than that of GD whose dimension is $n-1$.

In this paper, we introduce a class of nonsmooth nonconvex optimization problems, and we propose to use a local iterative minimization-majorization (MM) algorithm to find an optimal solution for the optimization problem. The cost functions in our optimization problems are an extension of convex functions with MC separable penalty, which were previously introduced by Ivan Selesnick. These functions are not convex; therefore, convex optimization methods cannot be applied here to prove the existence of optimal minimum point for these functions. For our purpose, we use convex analysis tools to first construct a class of convex majorizers, which approximate the value of non-convex cost function locally, then use the MM algorithm to prove the existence of local minimum. The convergence of the algorithm is guaranteed when the iterative points $x^{(k)}$ are obtained in a ball centred at $x^{(k-1)}$ with small radius. We prove that the algorithm converges to a stationary point (local minimum) of cost function when the surregators are strongly convex.

Metric regularity theory lies at the very heart of variational analysis, a relatively new discipline whose appearance was, to a large extent, determined by the needs of modern optimization theory in which such phenomena as nondifferentiability and set-valued mappings naturally appear. The roots of the theory go back to such fundamental results of the classical analysis as the implicit function theorem, Sard theorem and some others. This paper offers a survey of the state of the art of some principal parts of the theory along with a variety of its applications in analysis and optimization.

A stochastic algorithm for bound-constrained global optimization is described. The method can be applied to objective functions that are nonsmooth or even discontinuous. The algorithm forms a partition on the search region using classification and regression trees (CART), which defines a region where the objective function is relatively low. Further points are drawn directly from the low region before a new partition is formed. Alternating between partition and sampling phases provides an effective method for nonsmooth global optimization. The sequence of iterates generated by the algorithm is shown to converge to an essential global minimizer with probability one under mild conditions. Nonprobabilistic results are also given when random sampling is replaced with points taken from the Halton sequence. Numerical results are presented for both smooth and nonsmooth problems and show that the method is effective and competitive in practice.

In this paper, we consider variational approaches to handle the multiplicative noise removal and deblurring problem. Based on rather reasonable physical blurring-noisy assumptions, we derive a new variational model for this issue. After the study of the basic properties, we propose to approximate it by a convex relaxation model which is a balance between the previous non-convex model and a convex model. The relaxed model is solved by an alternating minimization approach. Numerical examples are presented to illustrate the effectiveness and efficiency of the proposed method.

In this paper, a new approach to a characterization of solvability of a nonlinear nonsmooth multiobjective programming problem with inequality constraints is introduced. A family of η-approximated vector optimization problems is constructed by a modification of the objective and the constraint functions in the original nonsmooth multiobjective programming problem. The connection between (weak) efficient points in the original nonsmooth multiobjective programming problem and its equivalent η-approximated vector optimization problems is established under V-invexity. It turns out that, in most cases, solvability of a nonlinear nonsmooth multiobjective programming problem can be characterized by solvability of differentiable vector optimization problems.

Differentiability properties of optimal value functions associated with perturbed optimization problems require strong assumptions. We consider such a set of assumptions which does not use compactness hypothesis but which involves a kind of coherence property. Moreover, a strict differentiability property is obtained by using techniques of Ekeland and Lebourg and a result of Preiss. Such a strengthening is required in order to obtain genericity results.

Formulas for the Clarke subdifferential are always expressed in the form of inclusion. The equality form in these formulas generally requires the functions to be directionally regular. This paper studies the directional regularity of the general class of extended-real-valued functions that are directionally Lipschitzian. Connections with the concept of subdifferential regularity are also established.