p-regularized least squares (0 <p <1) and critical path

Masahiro Yukawa, Shun Ichi Amari

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

This paper elucidates the underlying structures of ℓp-regularized least squares problems in the nonconvex case of 0 <p <1. The difference between two formulations is highlighted (which does not occur in the convex case of p = 1): 1) an ℓp -constrained optimization (Pp c) and 2) an ℓp-penalized (unconstrained) optimization (Lλ p). It is shown that the solution path of (Lλ p) is discontinuous and also a part of the solution path of (Pp c). As an alternative to the solution path, a critical path is considered, which is a maximal continuous curve consisting of critical points. Critical paths are piecewise smooth, as can be seen from the viewpoint of the variational method, and generally contain non-optimal points, such as saddle points and local maxima as well as global/local minima. Our study reveals multiplicity (non-monotonicity) in the correspondence between the regularization parameters of (Pp c) and (Lλ p). Two particular paths of critical points connecting the origin and an ordinary least squares (OLS) solution are studied further. One is a main path starting at an OLS solution, and the other is a greedy path starting at the origin. Part of the greedy path can be constructed with a generalized Minkowskian gradient. This paper of greedy path leads to a nontrivial close-link between the optimization problem of ℓp -regularized least squares and the greedy method of orthogonal matching pursuit.

Original languageEnglish
Article number7330004
Pages (from-to)488-502
Number of pages15
JournalIEEE Transactions on Information Theory
Volume62
Issue number1
DOIs
Publication statusPublished - 2016 Jan 1

Fingerprint

Constrained optimization

Keywords

  • Critical points
  • LARS
  • Nonconvex optimization
  • Sparse solution
  • ℓ quasi-norm (0 < p < 1)

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Library and Information Sciences

Cite this

p-regularized least squares (0 <p <1) and critical path. / Yukawa, Masahiro; Amari, Shun Ichi.

In: IEEE Transactions on Information Theory, Vol. 62, No. 1, 7330004, 01.01.2016, p. 488-502.

Research output: Contribution to journalArticle

@article{33237cf56fe2482399f304f932724cda,
title = "ℓp-regularized least squares (0 <p <1) and critical path",
abstract = "This paper elucidates the underlying structures of ℓp-regularized least squares problems in the nonconvex case of 0 p -constrained optimization (Pp c) and 2) an ℓp-penalized (unconstrained) optimization (Lλ p). It is shown that the solution path of (Lλ p) is discontinuous and also a part of the solution path of (Pp c). As an alternative to the solution path, a critical path is considered, which is a maximal continuous curve consisting of critical points. Critical paths are piecewise smooth, as can be seen from the viewpoint of the variational method, and generally contain non-optimal points, such as saddle points and local maxima as well as global/local minima. Our study reveals multiplicity (non-monotonicity) in the correspondence between the regularization parameters of (Pp c) and (Lλ p). Two particular paths of critical points connecting the origin and an ordinary least squares (OLS) solution are studied further. One is a main path starting at an OLS solution, and the other is a greedy path starting at the origin. Part of the greedy path can be constructed with a generalized Minkowskian gradient. This paper of greedy path leads to a nontrivial close-link between the optimization problem of ℓp -regularized least squares and the greedy method of orthogonal matching pursuit.",
keywords = "Critical points, LARS, Nonconvex optimization, Sparse solution, ℓ quasi-norm (0 < p < 1)",
author = "Masahiro Yukawa and Amari, {Shun Ichi}",
year = "2016",
month = "1",
day = "1",
doi = "10.1109/TIT.2015.2501362",
language = "English",
volume = "62",
pages = "488--502",
journal = "IEEE Transactions on Information Theory",
issn = "0018-9448",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "1",

}

TY - JOUR

T1 - ℓp-regularized least squares (0 <p <1) and critical path

AU - Yukawa, Masahiro

AU - Amari, Shun Ichi

PY - 2016/1/1

Y1 - 2016/1/1

N2 - This paper elucidates the underlying structures of ℓp-regularized least squares problems in the nonconvex case of 0 p -constrained optimization (Pp c) and 2) an ℓp-penalized (unconstrained) optimization (Lλ p). It is shown that the solution path of (Lλ p) is discontinuous and also a part of the solution path of (Pp c). As an alternative to the solution path, a critical path is considered, which is a maximal continuous curve consisting of critical points. Critical paths are piecewise smooth, as can be seen from the viewpoint of the variational method, and generally contain non-optimal points, such as saddle points and local maxima as well as global/local minima. Our study reveals multiplicity (non-monotonicity) in the correspondence between the regularization parameters of (Pp c) and (Lλ p). Two particular paths of critical points connecting the origin and an ordinary least squares (OLS) solution are studied further. One is a main path starting at an OLS solution, and the other is a greedy path starting at the origin. Part of the greedy path can be constructed with a generalized Minkowskian gradient. This paper of greedy path leads to a nontrivial close-link between the optimization problem of ℓp -regularized least squares and the greedy method of orthogonal matching pursuit.

AB - This paper elucidates the underlying structures of ℓp-regularized least squares problems in the nonconvex case of 0 p -constrained optimization (Pp c) and 2) an ℓp-penalized (unconstrained) optimization (Lλ p). It is shown that the solution path of (Lλ p) is discontinuous and also a part of the solution path of (Pp c). As an alternative to the solution path, a critical path is considered, which is a maximal continuous curve consisting of critical points. Critical paths are piecewise smooth, as can be seen from the viewpoint of the variational method, and generally contain non-optimal points, such as saddle points and local maxima as well as global/local minima. Our study reveals multiplicity (non-monotonicity) in the correspondence between the regularization parameters of (Pp c) and (Lλ p). Two particular paths of critical points connecting the origin and an ordinary least squares (OLS) solution are studied further. One is a main path starting at an OLS solution, and the other is a greedy path starting at the origin. Part of the greedy path can be constructed with a generalized Minkowskian gradient. This paper of greedy path leads to a nontrivial close-link between the optimization problem of ℓp -regularized least squares and the greedy method of orthogonal matching pursuit.

KW - Critical points

KW - LARS

KW - Nonconvex optimization

KW - Sparse solution

KW - ℓ quasi-norm (0 < p < 1)

UR - http://www.scopus.com/inward/record.url?scp=84959378738&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84959378738&partnerID=8YFLogxK

U2 - 10.1109/TIT.2015.2501362

DO - 10.1109/TIT.2015.2501362

M3 - Article

AN - SCOPUS:84959378738

VL - 62

SP - 488

EP - 502

JO - IEEE Transactions on Information Theory

JF - IEEE Transactions on Information Theory

SN - 0018-9448

IS - 1

M1 - 7330004

ER -