Limited-memory BFGS (L-BFGS or LM-BFGS). More...
#include <FgLBFGSsolver.h>
Public Member Functions | |
LBFGSSolver (int numberOfVariables) | |
Creates a new instance of the L-BFGS optimization algorithm. More... | |
LBFGSSolver (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function) | |
Creates a new instance of the L-BFGS optimization algorithm. More... | |
LBFGSSolver (NonlinearObjectiveFunction< Scalar > *function) | |
Creates a new instance of the L-BFGS optimization algorithm. More... | |
void | SetNumCorrections (int corrections) |
Sets number of corrections. More... | |
void | SetDelta (Scalar inDelta) |
Sets Delta for convergence test. More... | |
void | SetMaxLinesearch (int maxIter) |
Sets the maximum number of trials for the line search. More... | |
![]() | |
Scalar | GetTolerance () |
Gets the relative difference threshold to be used as stopping criteria between two iterations. Default is 0 (iterate until convergence). More... | |
void | SetTolerance (Scalar tolerance) |
Sets the relative difference threshold to be used as stopping criteria between two iterations. Default is 0 (iterate until convergence). More... | |
int | GetMaxIterations () |
Gets the maximum number of iterations to be performed during optimization. Default is 0 (iterate until convergence). More... | |
void | SetMaxIterations (int iter) |
Sets the maximum number of iterations to be performed during optimization. Default is 0 (iterate until convergence). More... | |
int | GetIterations () |
Gets the number of iterations performed in the last call to IOptimizationMethod.Minimize(). More... | |
![]() | |
virtual int | GetNumberOfVariables () |
Gets the number of variables (free parameters) in the optimization problem. More... | |
virtual af::array | GetSolution () |
Gets the current solution found, the values of the parameters which optimizes the function. More... | |
virtual void | SetSolution (af::array &x) |
Sets the current solution found, the values of the parameters which optimizes the function. More... | |
virtual Scalar | GetValue () |
Gets the output of the function at the current Solution. More... | |
virtual bool | Maximize (af::array &values, int *cycle=nullptr) |
Finds the maximum value of a function. The solution vector will be made available at the Solution property. More... | |
virtual bool | Minimize (af::array &values, int *cycle=nullptr) |
Finds the minimum value of a function. The solution vector will be made available at the Solution property. More... | |
virtual bool | Maximize (int *cycle=nullptr) |
Finds the maximum value of a function. The solution vector will be made available at the Solution property. More... | |
virtual bool | Minimize (int *cycle=nullptr) |
Finds the minimum value of a function. The solution vector will be made available at the Solution property. More... | |
void | Display (bool display) |
Set to display optimization information. More... | |
virtual int | GetNumberOfVariables ()=0 |
Gets the number of variables (free parameters) in the optimization problem. More... | |
virtual af::array | GetSolution ()=0 |
Gets the current solution found, the values of the parameters which optimizes the function. More... | |
virtual void | SetSolution (af::array &x)=0 |
Gets a solution. More... | |
virtual Scalar | GetValue ()=0 |
Gets the output of the function at the current Solution. More... | |
virtual bool | Minimize (int *cycle=nullptr)=0 |
Finds the minimum value of a function. The solution vector will be made available at the Solution property. More... | |
virtual bool | Maximize (int *cycle=nullptr)=0 |
Finds the maximum value of a function. The solution vector will be made available at the Solution property. More... | |
Protected Member Functions | |
virtual bool | Optimize (int *cycle=nullptr) override |
Implements the actual optimization algorithm. This method should try to minimize the objective function. More... | |
![]() | |
BaseGradientOptimizationMethod (int numberOfVariables) | |
Initializes a new instance of the BaseGradientOptimizationMethod class. More... | |
BaseGradientOptimizationMethod (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function) | |
Initializes a new instance of the BaseGradientOptimizationMethod class. More... | |
BaseGradientOptimizationMethod (NonlinearObjectiveFunction< Scalar > *function) | |
Initializes a new instance of the BaseGradientOptimizationMethod class. More... | |
void | InitLinesearch () |
Inits linesearch. More... | |
![]() | |
void | SetValue (Scalar v) |
Sets the output of the function at the current Solution. More... | |
void | SetNumberOfVariables (int n) |
Sets the number of variables (free parameters) in the optimization problem. More... | |
BaseOptimizationMethod (int numberOfVariables) | |
Initializes a new instance of the BaseOptimizationMethod class. More... | |
BaseOptimizationMethod (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function) | |
Initializes a new instance of the BaseOptimizationMethod class. More... | |
BaseOptimizationMethod (NonlinearObjectiveFunction< Scalar > *function) | |
Initializes a new instance of the BaseOptimizationMethod class. More... | |
virtual bool | Optimize (int *cycle=nullptr)=0 |
Implements the actual optimization algorithm. This method should try to minimize the objective function. More... | |
Private Attributes | |
int | m |
int | past |
Scalar | delta |
int | max_linesearch |
Scalar | min_step |
Scalar | max_step |
Scalar | ftol |
Scalar | wolfe |
Additional Inherited Members | |
![]() | |
int | maxIterations |
Scalar | _tolerance |
int | iterations |
ILineSearch< Scalar > * | linesearch |
![]() | |
NonlinearObjectiveFunction< Scalar > * | _function |
af::array | _x |
bool | _display |
af::dtype | m_dtype |
Limited-memory BFGS (L-BFGS or LM-BFGS).
Limited-memory BFGS (L-BFGS or LM-BFGS) is an optimization algorithm in the family of quasi-Newton methods that approximates the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm using a limited amount of computer memory. It is a popular algorithm for parameter estimation in machine learning.[1][2] The algorithm's target problem is to minimize $f ( x )$ $f(\mathbf {x} )$ over unconstrained values of the real-vector $\mathbf{x}$ where $f$ is a differentiable scalar function.
Like the original BFGS, L-BFGS uses an estimation to the inverse Hessian matrix to steer its search through variable space, but where BFGS stores a dense $n\times n$ approximation to the inverse Hessian (n being the number of variables in the problem), L-BFGS stores only a few vectors that represent the approximation implicitly. Due to its resulting linear memory requirement, the L-BFGS method is particularly well suited for optimization problems with a large number of variables. Instead of the inverse Hessian $\mathbf{H}_k$, L-BFGS maintains a history of the past m updates of the position $\mathbf{x}$ and gradient ∇f(x), where generally the history size $m$ can be small (often $m<10$). These updates are used to implicitly do operations requiring the $\mathbf{H}_k$-vector product.
References:
HmetalT, 02.05.2019.
Definition at line 76 of file FgLBFGSsolver.h.
NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::LBFGSSolver | ( | int | numberOfVariables | ) |
Creates a new instance of the L-BFGS optimization algorithm.
Admin, 3/27/2017.
numberOfVariables | The number of free parameters in the optimization problem. |
NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::LBFGSSolver | ( | int | numberOfVariables, |
std::function< Scalar(const af::array &, af::array &)> | function | ||
) |
Creates a new instance of the L-BFGS optimization algorithm.
Admin, 3/27/2017.
numberOfVariables | The number of free parameters in the function to be optimized. |
function | [in,out] The function to be optimized. |
gradient | [in,out] The gradient of the function. |
NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::LBFGSSolver | ( | NonlinearObjectiveFunction< Scalar > * | function | ) |
Creates a new instance of the L-BFGS optimization algorithm.
Admin, 3/27/2017.
function | The objective function and gradients whose optimum values should be found. |
void NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::SetNumCorrections | ( | int | corrections | ) |
Sets number of corrections.
The number of corrections to approximate the inverse hessian matrix. The L-BFGS routine stores the computation results of previous m iterations to approximate the inverse hessian matrix of the current iteration. This parameter controls the size of the limited memories (corrections). The default value is 6
. Values less than 3
are not recommended. Large values will result in excessive computing time.
Hmetal T, 04/06/2019.
corrections | The corrections. |
void NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::SetDelta | ( | Scalar | inDelta | ) |
Sets Delta for convergence test.
The algorithm stops when the following condition is met, \((f_{k-d}(x)-f_k(x))/f_k(x)<\delta\), where \(f_k(x)\) is
the current function value, \(f_{k-d}(x)\) is the function value
\(d\) iterations ago (specified by the past parameter).
The default value is 0
.
Hmetal T, 06/06/2019.
inDelta | The in delta. |
void NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::SetMaxLinesearch | ( | int | maxIter | ) |
Sets the maximum number of trials for the line search.
This parameter controls the number of function and gradients evaluations per iteration for the line search routine. The default value is 20
.
Hmetal T, 06/06/2019.
maxIter | The maximum iterator. |
|
overrideprotectedvirtual |
Implements the actual optimization algorithm. This method should try to minimize the objective function.
Hmetal T, 11.04.2017.
Implements NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >.
|
private |
Definition at line 214 of file FgLBFGSsolver.h.
|
private |
Definition at line 215 of file FgLBFGSsolver.h.
|
private |
Definition at line 216 of file FgLBFGSsolver.h.
|
private |
Definition at line 217 of file FgLBFGSsolver.h.
|
private |
Definition at line 218 of file FgLBFGSsolver.h.
|
private |
Definition at line 219 of file FgLBFGSsolver.h.
|
private |
Definition at line 220 of file FgLBFGSsolver.h.
|
private |
Definition at line 221 of file FgLBFGSsolver.h.