NeuralEngine
A Game Engine with embeded Machine Learning algorithms based on Gaussian Processes.
NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType > Class Template Reference

Limited-memory BFGS (L-BFGS or LM-BFGS). More...

#include <FgLBFGSsolver.h>

Inheritance diagram for NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >:
Collaboration diagram for NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >:

Public Member Functions

 LBFGSSolver (int numberOfVariables)
 Creates a new instance of the L-BFGS optimization algorithm. More...
 
 LBFGSSolver (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function)
 Creates a new instance of the L-BFGS optimization algorithm. More...
 
 LBFGSSolver (NonlinearObjectiveFunction< Scalar > *function)
 Creates a new instance of the L-BFGS optimization algorithm. More...
 
void SetNumCorrections (int corrections)
 Sets number of corrections. More...
 
void SetDelta (Scalar inDelta)
 Sets Delta for convergence test. More...
 
void SetMaxLinesearch (int maxIter)
 Sets the maximum number of trials for the line search. More...
 
- Public Member Functions inherited from NeuralEngine::MachineLearning::BaseGradientOptimizationMethod< Scalar, MoreThuente >
Scalar GetTolerance ()
 Gets the relative difference threshold to be used as stopping criteria between two iterations. Default is 0 (iterate until convergence). More...
 
void SetTolerance (Scalar tolerance)
 Sets the relative difference threshold to be used as stopping criteria between two iterations. Default is 0 (iterate until convergence). More...
 
int GetMaxIterations ()
 Gets the maximum number of iterations to be performed during optimization. Default is 0 (iterate until convergence). More...
 
void SetMaxIterations (int iter)
 Sets the maximum number of iterations to be performed during optimization. Default is 0 (iterate until convergence). More...
 
int GetIterations ()
 Gets the number of iterations performed in the last call to IOptimizationMethod.Minimize(). More...
 
- Public Member Functions inherited from NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >
virtual int GetNumberOfVariables ()
 Gets the number of variables (free parameters) in the optimization problem. More...
 
virtual af::array GetSolution ()
 Gets the current solution found, the values of the parameters which optimizes the function. More...
 
virtual void SetSolution (af::array &x)
 Sets the current solution found, the values of the parameters which optimizes the function. More...
 
virtual Scalar GetValue ()
 Gets the output of the function at the current Solution. More...
 
virtual bool Maximize (af::array &values, int *cycle=nullptr)
 Finds the maximum value of a function. The solution vector will be made available at the Solution property. More...
 
virtual bool Minimize (af::array &values, int *cycle=nullptr)
 Finds the minimum value of a function. The solution vector will be made available at the Solution property. More...
 
virtual bool Maximize (int *cycle=nullptr)
 Finds the maximum value of a function. The solution vector will be made available at the Solution property. More...
 
virtual bool Minimize (int *cycle=nullptr)
 Finds the minimum value of a function. The solution vector will be made available at the Solution property. More...
 
void Display (bool display)
 Set to display optimization information. More...
 
virtual int GetNumberOfVariables ()=0
 Gets the number of variables (free parameters) in the optimization problem. More...
 
virtual af::array GetSolution ()=0
 Gets the current solution found, the values of the parameters which optimizes the function. More...
 
virtual void SetSolution (af::array &x)=0
 Gets a solution. More...
 
virtual Scalar GetValue ()=0
 Gets the output of the function at the current Solution. More...
 
virtual bool Minimize (int *cycle=nullptr)=0
 Finds the minimum value of a function. The solution vector will be made available at the Solution property. More...
 
virtual bool Maximize (int *cycle=nullptr)=0
 Finds the maximum value of a function. The solution vector will be made available at the Solution property. More...
 

Protected Member Functions

virtual bool Optimize (int *cycle=nullptr) override
 Implements the actual optimization algorithm. This method should try to minimize the objective function. More...
 
- Protected Member Functions inherited from NeuralEngine::MachineLearning::BaseGradientOptimizationMethod< Scalar, MoreThuente >
 BaseGradientOptimizationMethod (int numberOfVariables)
 Initializes a new instance of the BaseGradientOptimizationMethod class. More...
 
 BaseGradientOptimizationMethod (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function)
 Initializes a new instance of the BaseGradientOptimizationMethod class. More...
 
 BaseGradientOptimizationMethod (NonlinearObjectiveFunction< Scalar > *function)
 Initializes a new instance of the BaseGradientOptimizationMethod class. More...
 
void InitLinesearch ()
 Inits linesearch. More...
 
- Protected Member Functions inherited from NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >
void SetValue (Scalar v)
 Sets the output of the function at the current Solution. More...
 
void SetNumberOfVariables (int n)
 Sets the number of variables (free parameters) in the optimization problem. More...
 
 BaseOptimizationMethod (int numberOfVariables)
 Initializes a new instance of the BaseOptimizationMethod class. More...
 
 BaseOptimizationMethod (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function)
 Initializes a new instance of the BaseOptimizationMethod class. More...
 
 BaseOptimizationMethod (NonlinearObjectiveFunction< Scalar > *function)
 Initializes a new instance of the BaseOptimizationMethod class. More...
 
virtual bool Optimize (int *cycle=nullptr)=0
 Implements the actual optimization algorithm. This method should try to minimize the objective function. More...
 

Private Attributes

int m
 
int past
 
Scalar delta
 
int max_linesearch
 
Scalar min_step
 
Scalar max_step
 
Scalar ftol
 
Scalar wolfe
 

Additional Inherited Members

- Protected Attributes inherited from NeuralEngine::MachineLearning::BaseGradientOptimizationMethod< Scalar, MoreThuente >
int maxIterations
 
Scalar _tolerance
 
int iterations
 
ILineSearch< Scalar > * linesearch
 
- Protected Attributes inherited from NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >
NonlinearObjectiveFunction< Scalar > * _function
 
af::array _x
 
bool _display
 
af::dtype m_dtype
 

Detailed Description

template<typename Scalar, LineSearchType LSType = MoreThuente>
class NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >

Limited-memory BFGS (L-BFGS or LM-BFGS).


Limited-memory BFGS (L-BFGS or LM-BFGS) is an optimization algorithm in the family of quasi-Newton methods that approximates the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm using a limited amount of computer memory. It is a popular algorithm for parameter estimation in machine learning.[1][2] The algorithm's target problem is to minimize $f ( x )$ $f(\mathbf {x} )$ over unconstrained values of the real-vector $\mathbf{x}$ where $f$ is a differentiable scalar function.

Like the original BFGS, L-BFGS uses an estimation to the inverse Hessian matrix to steer its search through variable space, but where BFGS stores a dense $n\times n$ approximation to the inverse Hessian (n being the number of variables in the problem), L-BFGS stores only a few vectors that represent the approximation implicitly. Due to its resulting linear memory requirement, the L-BFGS method is particularly well suited for optimization problems with a large number of variables. Instead of the inverse Hessian $\mathbf{H}_k$, L-BFGS maintains a history of the past m updates of the position $\mathbf{x}$ and gradient ∇f(x), where generally the history size $m$ can be small (often $m<10$). These updates are used to implicitly do operations requiring the $\mathbf{H}_k$-vector product.

References:

HmetalT, 02.05.2019.

Definition at line 76 of file FgLBFGSsolver.h.

Constructor & Destructor Documentation

◆ LBFGSSolver() [1/3]

template<typename Scalar , LineSearchType LSType = MoreThuente>
NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::LBFGSSolver ( int  numberOfVariables)

Creates a new instance of the L-BFGS optimization algorithm.

Admin, 3/27/2017.

Parameters
numberOfVariablesThe number of free parameters in the optimization problem.

◆ LBFGSSolver() [2/3]

template<typename Scalar , LineSearchType LSType = MoreThuente>
NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::LBFGSSolver ( int  numberOfVariables,
std::function< Scalar(const af::array &, af::array &)>  function 
)

Creates a new instance of the L-BFGS optimization algorithm.

Admin, 3/27/2017.

Parameters
numberOfVariablesThe number of free parameters in the function to be optimized.
function[in,out] The function to be optimized.
gradient[in,out] The gradient of the function.

◆ LBFGSSolver() [3/3]

template<typename Scalar , LineSearchType LSType = MoreThuente>
NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::LBFGSSolver ( NonlinearObjectiveFunction< Scalar > *  function)

Creates a new instance of the L-BFGS optimization algorithm.

Admin, 3/27/2017.

Parameters
functionThe objective function and gradients whose optimum values should be found.

Member Function Documentation

◆ SetNumCorrections()

template<typename Scalar , LineSearchType LSType = MoreThuente>
void NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::SetNumCorrections ( int  corrections)

Sets number of corrections.


The number of corrections to approximate the inverse hessian matrix. The L-BFGS routine stores the computation results of previous m iterations to approximate the inverse hessian matrix of the current iteration. This parameter controls the size of the limited memories (corrections). The default value is 6. Values less than 3 are not recommended. Large values will result in excessive computing time.

Hmetal T, 04/06/2019.

Parameters
correctionsThe corrections.

◆ SetDelta()

template<typename Scalar , LineSearchType LSType = MoreThuente>
void NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::SetDelta ( Scalar  inDelta)

Sets Delta for convergence test.


The algorithm stops when the following condition is met, \((f_{k-d}(x)-f_k(x))/f_k(x)<\delta\), where \(f_k(x)\) is
the current function value, \(f_{k-d}(x)\) is the function value
\(d\) iterations ago (specified by the past parameter).
The default value is 0.

Hmetal T, 06/06/2019.

Parameters
inDeltaThe in delta.

◆ SetMaxLinesearch()

template<typename Scalar , LineSearchType LSType = MoreThuente>
void NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::SetMaxLinesearch ( int  maxIter)

Sets the maximum number of trials for the line search.


This parameter controls the number of function and gradients evaluations per iteration for the line search routine. The default value is 20.

Hmetal T, 06/06/2019.

Parameters
maxIterThe maximum iterator.

◆ Optimize()

template<typename Scalar , LineSearchType LSType = MoreThuente>
virtual bool NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::Optimize ( int cycle = nullptr)
overrideprotectedvirtual

Implements the actual optimization algorithm. This method should try to minimize the objective function.

Hmetal T, 11.04.2017.

Returns
true if it succeeds, false if it fails.

Implements NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >.

Member Data Documentation

◆ m

template<typename Scalar , LineSearchType LSType = MoreThuente>
int NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::m
private

Definition at line 214 of file FgLBFGSsolver.h.

◆ past

template<typename Scalar , LineSearchType LSType = MoreThuente>
int NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::past
private

Definition at line 215 of file FgLBFGSsolver.h.

◆ delta

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::delta
private

Definition at line 216 of file FgLBFGSsolver.h.

◆ max_linesearch

template<typename Scalar , LineSearchType LSType = MoreThuente>
int NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::max_linesearch
private

Definition at line 217 of file FgLBFGSsolver.h.

◆ min_step

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::min_step
private

Definition at line 218 of file FgLBFGSsolver.h.

◆ max_step

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::max_step
private

Definition at line 219 of file FgLBFGSsolver.h.

◆ ftol

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::ftol
private

Definition at line 220 of file FgLBFGSsolver.h.

◆ wolfe

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::LBFGSSolver< Scalar, LSType >::wolfe
private

Definition at line 221 of file FgLBFGSsolver.h.


The documentation for this class was generated from the following file: