Approximate Versions of the Alternating Direction Method of Multipliers

Download or read book Approximate Versions of the Alternating Direction Method of Multipliers written by Wang Yao and published by . This book was released on 2016 with total page 115 pages. Available in PDF, EPUB and Kindle. Book excerpt: Convex optimization is at the core of many of today's analysis tools for large datasets, and in particular machine learning methods. This thesis will develop approximate versions of the alternating directrion of multipliers (ADMM) for the general setting of minimizing the sum of two convex functions. The alternating direction method of multipliers is a form of augmented Lagrangian algorithm that has experienced a renaissance in recent years due to its applicability to optimization problems arising from ``big data'' and image processing applications, and the relative ease with which it may be implemented in parallel and distributed computational environments. There are two fundamental approaches for proving the convergence of the ADMM, each based on a different form of two-way emph{splitting}, that is, expressing a mapping as the sum of two simpler mappings. The first approach is based on Douglas-Rachford operator splitting theory, and yields considerable insight into the convergence of the ADMM. The second convergence proof approach is at its core based on the Lagrangian splitting analysis. We present three new approximate versions of ADMM based on both convergence analyses, all of which require only knowledge of subgradients of the subproblem objectives, rather bounds on the distance to the exact subproblem solution. One version, which applies only to certain common special cases, is based on combining the operator splitting analysis of the ADMM with a relative-error proximal point algorithm of Solodov and Svaiter. A byproduct of this analysis is a new, relative-error version of the Douglas-Rachford splitting algorithm for monotone operators. The other two approximate versions of the ADMM are more general and based on the Lagrangian splitting analysis of the ADMM: one uses a summable absolute error criterion, and the other uses a relative error criterion and an auxiliary iterate sequence. We experimentally compare our new algorithms to an essentially exact form of the ADMM and to an inexact form that can be easily derived from prior theory (but again applies only to certain common special cases). These experiments show that our methods can significantly reduce total computational effort when iterative methods are used to solve ADMM subproblems.