An implementation of LARS, a stage-wise homotopy-based algorithm for l1-regularized linear regression (LASSO) and l1+l2 regularized linear regression (Elastic Net). More...

Public Member Functions
	LARS (const bool useCholesky, const double lambda1=0.0, const double lambda2=0.0, const double tolerance=1e-16)
	Set the parameters to LARS.
	LARS (const bool useCholesky, const arma::mat &gramMatrix, const double lambda1=0.0, const double lambda2=0.0, const double tolerance=1e-16)
	Set the parameters to LARS, and pass in a precalculated Gram matrix.
const std::vector< size_t > &	ActiveSet () const
	Access the set of active dimensions.
const std::vector< arma::vec > &	BetaPath () const
	Access the set of coefficients after each iteration; the solution is the last element.
const std::vector< double > &	LambdaPath () const
	Access the set of values for lambda1 after each iteration; the solution is the last element.
const arma::mat &	MatUtriCholFactor () const
	Access the upper triangular cholesky factor.
void	Regress (const arma::mat &data, const arma::vec &responses, arma::vec &beta, const bool rowMajor=false)
	Run LARS.
Private Member Functions
void	Activate (const size_t varInd)
	Add dimension varInd to active set.
void	CholeskyDelete (const size_t colToKill)
void	CholeskyInsert (const arma::vec &newX, const arma::mat &X)
void	CholeskyInsert (double sqNormNewX, const arma::vec &newGramCol)
void	ComputeYHatDirection (const arma::mat &matX, const arma::vec &betaDirection, arma::vec &yHatDirection)
void	Deactivate (const size_t activeVarInd)
	Remove activeVarInd'th element from active set.
void	GivensRotate (const arma::vec::fixed< 2 > &x, arma::vec::fixed< 2 > &rotatedX, arma::mat &G)
void	InterpolateBeta ()
Private Attributes
std::vector< size_t >	activeSet
	Active set of dimensions.
std::vector< arma::vec >	betaPath
	Solution path.
bool	elasticNet
	True if this is the elastic net problem.
std::vector< bool >	isActive
	Active set membership indicator (for each dimension).
double	lambda1
	Regularization parameter for l1 penalty.
double	lambda2
	Regularization parameter for l2 penalty.
std::vector< double >	lambdaPath
	Value of lambda_1 for each solution in solution path.
bool	lasso
	True if this is the LASSO problem.
const arma::mat &	matGram
	Reference to the Gram matrix we will use.
arma::mat	matGramInternal
	Gram matrix.
arma::mat	matUtriCholFactor
	Upper triangular cholesky factor; initially 0x0 matrix.
double	tolerance
	Tolerance for main loop.
bool	useCholesky
	Whether or not to use Cholesky decomposition when solving linear system.

Detailed Description

An implementation of LARS, a stage-wise homotopy-based algorithm for l1-regularized linear regression (LASSO) and l1+l2 regularized linear regression (Elastic Net).

Let $ X $ be a matrix where each row is a point and each column is a dimension and let $ y $ be a vector of responses.

The Elastic Net problem is to solve

$\min_{\beta} 0.5 || X \beta - y ||_2^2 + \lambda_1 || \beta ||_1 + 0.5 \lambda_2 || \beta ||_2^2$

where $\beta$ is the vector of regression coefficients.

If $\lambda_1 > 0$ and $\lambda_2 = 0$ , the problem is the LASSO. If $\lambda_1 > 0$ and $\lambda_2 > 0$ , the problem is the elastic net. If $\lambda_1 = 0$ and $\lambda_2 > 0$ , the problem is ridge regression. If $\lambda_1 = 0$ and $\lambda_2 = 0$ , the problem is unregularized linear regression.

Note: This algorithm is not recommended for use (in terms of efficiency) when $\lambda_1$ = 0.

For more details, see the following papers:

 @article{efron2004least,
   title={Least angle regression},
   author={Efron, B. and Hastie, T. and Johnstone, I. and Tibshirani, R.},
   journal={The Annals of statistics},
   volume={32},
   number={2},
   pages={407--499},
   year={2004},
   publisher={Institute of Mathematical Statistics}
 }

 @article{zou2005regularization,
   title={Regularization and variable selection via the elastic net},
   author={Zou, H. and Hastie, T.},
   journal={Journal of the Royal Statistical Society Series B},
   volume={67},
   number={2},
   pages={301--320},
   year={2005},
   publisher={Royal Statistical Society}
 }

Definition at line 100 of file lars.hpp.

Constructor & Destructor Documentation

mlpack::regression::LARS::LARS	(	const bool	useCholesky,
		const double	lambda1 = `0.0`,
		const double	lambda2 = `0.0`,
		const double	tolerance = `1e-16`
	)

Set the parameters to LARS.

Both lambda1 and lambda2 default to 0.

Parameters:

useCholesky	Whether or not to use Cholesky decomposition when solving linear system (as opposed to using the full Gram matrix).
lambda1	Regularization parameter for l1-norm penalty.
lambda2	Regularization parameter for l2-norm penalty.
tolerance	Run until the maximum correlation of elements in (X^T y) is less than this.

mlpack::regression::LARS::LARS	(	const bool	useCholesky,
		const arma::mat &	gramMatrix,
		const double	lambda1 = `0.0`,
		const double	lambda2 = `0.0`,
		const double	tolerance = `1e-16`
	)

Set the parameters to LARS, and pass in a precalculated Gram matrix.

Both lambda1 and lambda2 default to 0.

Parameters:

useCholesky	Whether or not to use Cholesky decomposition when solving linear system (as opposed to using the full Gram matrix).
gramMatrix	Gram matrix.
lambda1	Regularization parameter for l1-norm penalty.
lambda2	Regularization parameter for l2-norm penalty.
tolerance	Run until the maximum correlation of elements in (X^T y) is less than this.

Member Function Documentation

void mlpack::regression::LARS::Activate ( const size_t varInd ) [private]

Add dimension varInd to active set.

Parameters:

varInd Dimension to add to active set.

const std::vector<size_t>& mlpack::regression::LARS::ActiveSet ( ) const [inline]

Access the set of active dimensions.

Definition at line 156 of file lars.hpp.

References activeSet.

const std::vector<arma::vec>& mlpack::regression::LARS::BetaPath ( ) const [inline]

Access the set of coefficients after each iteration; the solution is the last element.

Definition at line 160 of file lars.hpp.

References betaPath.

void mlpack::regression::LARS::CholeskyDelete ( const size_t colToKill ) [private]

void mlpack::regression::LARS::CholeskyInsert	(	const arma::vec &	newX,
		const arma::mat &	X
	)		`[private]`

void mlpack::regression::LARS::CholeskyInsert	(	double	sqNormNewX,
		const arma::vec &	newGramCol
	)		`[private]`

void mlpack::regression::LARS::ComputeYHatDirection	(	const arma::mat &	matX,
		const arma::vec &	betaDirection,
		arma::vec &	yHatDirection
	)		`[private]`

void mlpack::regression::LARS::Deactivate ( const size_t activeVarInd ) [private]

Remove activeVarInd'th element from active set.

Parameters:

activeVarInd Index of element to remove from active set.

void mlpack::regression::LARS::GivensRotate	(	const arma::vec::fixed< 2 > &	x,
		arma::vec::fixed< 2 > &	rotatedX,
		arma::mat &	G
	)		`[private]`

void mlpack::regression::LARS::InterpolateBeta ( ) [private]

const std::vector<double>& mlpack::regression::LARS::LambdaPath ( ) const [inline]

Access the set of values for lambda1 after each iteration; the solution is the last element.

Definition at line 164 of file lars.hpp.

References lambdaPath.

const arma::mat& mlpack::regression::LARS::MatUtriCholFactor ( ) const [inline]

Access the upper triangular cholesky factor.

Definition at line 167 of file lars.hpp.

References matUtriCholFactor.

void mlpack::regression::LARS::Regress	(	const arma::mat &	data,
		const arma::vec &	responses,
		arma::vec &	beta,
		const bool	rowMajor = `false`
	)

Run LARS.

The input matrix (like all MLPACK matrices) should be column-major -- each column is an observation and each row is a dimension. However, because LARS is more efficient on a row-major matrix, this method will (internally) transpose the matrix. If this transposition is not necessary (i.e., you want to pass in a row-major matrix), pass 'true' for the rowMajor parameter.

Parameters:

data	Column-major input data (or row-major input data if rowMajor = true).
responses	A vector of targets.
beta	Vector to store the solution (the coefficients) in.
rowMajor	Set to true if matX is row-major.