NeuralEngine
A Game Engine with embeded Machine Learning algorithms based on Gaussian Processes.
NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType > Class Template Reference

AdaMax optimizer. More...

#include <FgNadamSolver.h>

Inheritance diagram for NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >:
Collaboration diagram for NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >:

Public Member Functions

 NadamSolver (int numberOfVariables)
 Creates a new instance of the L-BFGS optimization algorithm. More...
 
 NadamSolver (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function)
 Creates a new instance of the L-BFGS optimization algorithm. More...
 
 NadamSolver (NonlinearObjectiveFunction< Scalar > *function)
 Creates a new instance of the L-BFGS optimization algorithm. More...
 
 ~NadamSolver ()
 Destructor. More...
 
void SetBeta1 (Scalar beta1)
 Sets decay rate for the first moment estimates. More...
 
void SetBeta2 (Scalar beta2)
 Sets decay rate for the second-moment estimates. More...
 
void SetAlpha (Scalar alpha)
 Sets the learning rate. More...
 
void SetEpsilon (Scalar epsilon)
 Sets an epsilon to avoid division by zero. More...
 
void SetDecay (Scalar decay)
 Sets initial decay rate. More...
 
Scalar GetBeta1 ()
 Gets decay rate for the first moment estimates. More...
 
Scalar GetBeta2 ()
 Gets decay rate for the second-moment estimates. More...
 
Scalar GetAlpha ()
 Gets the learning rate. More...
 
Scalar GetEpsilon ()
 Gets the epsilon. More...
 
Scalar GetDecay ()
 Gets the initial decay. More...
 
- Public Member Functions inherited from NeuralEngine::MachineLearning::BaseGradientOptimizationMethod< Scalar, MoreThuente >
Scalar GetTolerance ()
 Gets the relative difference threshold to be used as stopping criteria between two iterations. Default is 0 (iterate until convergence). More...
 
void SetTolerance (Scalar tolerance)
 Sets the relative difference threshold to be used as stopping criteria between two iterations. Default is 0 (iterate until convergence). More...
 
int GetMaxIterations ()
 Gets the maximum number of iterations to be performed during optimization. Default is 0 (iterate until convergence). More...
 
void SetMaxIterations (int iter)
 Sets the maximum number of iterations to be performed during optimization. Default is 0 (iterate until convergence). More...
 
int GetIterations ()
 Gets the number of iterations performed in the last call to IOptimizationMethod.Minimize(). More...
 
- Public Member Functions inherited from NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >
virtual int GetNumberOfVariables ()
 Gets the number of variables (free parameters) in the optimization problem. More...
 
virtual af::array GetSolution ()
 Gets the current solution found, the values of the parameters which optimizes the function. More...
 
virtual void SetSolution (af::array &x)
 Sets the current solution found, the values of the parameters which optimizes the function. More...
 
virtual Scalar GetValue ()
 Gets the output of the function at the current Solution. More...
 
virtual bool Maximize (af::array &values, int *cycle=nullptr)
 Finds the maximum value of a function. The solution vector will be made available at the Solution property. More...
 
virtual bool Minimize (af::array &values, int *cycle=nullptr)
 Finds the minimum value of a function. The solution vector will be made available at the Solution property. More...
 
virtual bool Maximize (int *cycle=nullptr)
 Finds the maximum value of a function. The solution vector will be made available at the Solution property. More...
 
virtual bool Minimize (int *cycle=nullptr)
 Finds the minimum value of a function. The solution vector will be made available at the Solution property. More...
 
void Display (bool display)
 Set to display optimization information. More...
 
virtual int GetNumberOfVariables ()=0
 Gets the number of variables (free parameters) in the optimization problem. More...
 
virtual af::array GetSolution ()=0
 Gets the current solution found, the values of the parameters which optimizes the function. More...
 
virtual void SetSolution (af::array &x)=0
 Gets a solution. More...
 
virtual Scalar GetValue ()=0
 Gets the output of the function at the current Solution. More...
 
virtual bool Minimize (int *cycle=nullptr)=0
 Finds the minimum value of a function. The solution vector will be made available at the Solution property. More...
 
virtual bool Maximize (int *cycle=nullptr)=0
 Finds the maximum value of a function. The solution vector will be made available at the Solution property. More...
 

Protected Member Functions

virtual bool Optimize (int *cycle=nullptr) override
 Implements the actual optimization algorithm. This method should try to minimize the objective function. More...
 
- Protected Member Functions inherited from NeuralEngine::MachineLearning::BaseGradientOptimizationMethod< Scalar, MoreThuente >
 BaseGradientOptimizationMethod (int numberOfVariables)
 Initializes a new instance of the BaseGradientOptimizationMethod class. More...
 
 BaseGradientOptimizationMethod (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function)
 Initializes a new instance of the BaseGradientOptimizationMethod class. More...
 
 BaseGradientOptimizationMethod (NonlinearObjectiveFunction< Scalar > *function)
 Initializes a new instance of the BaseGradientOptimizationMethod class. More...
 
void InitLinesearch ()
 Inits linesearch. More...
 
- Protected Member Functions inherited from NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >
void SetValue (Scalar v)
 Sets the output of the function at the current Solution. More...
 
void SetNumberOfVariables (int n)
 Sets the number of variables (free parameters) in the optimization problem. More...
 
 BaseOptimizationMethod (int numberOfVariables)
 Initializes a new instance of the BaseOptimizationMethod class. More...
 
 BaseOptimizationMethod (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function)
 Initializes a new instance of the BaseOptimizationMethod class. More...
 
 BaseOptimizationMethod (NonlinearObjectiveFunction< Scalar > *function)
 Initializes a new instance of the BaseOptimizationMethod class. More...
 
virtual bool Optimize (int *cycle=nullptr)=0
 Implements the actual optimization algorithm. This method should try to minimize the objective function. More...
 

Private Attributes

Scalar min_step
 
Scalar max_step
 
Scalar sAlpha
 
Scalar sBeta1
 
Scalar sBeta2
 
Scalar sEpsilon
 
Scalar sDecay
 
Scalar delta
 
Scalar sCumBeta1
 

Additional Inherited Members

- Protected Attributes inherited from NeuralEngine::MachineLearning::BaseGradientOptimizationMethod< Scalar, MoreThuente >
int maxIterations
 
Scalar _tolerance
 
int iterations
 
ILineSearch< Scalar > * linesearch
 
- Protected Attributes inherited from NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >
NonlinearObjectiveFunction< Scalar > * _function
 
af::array _x
 
bool _display
 
af::dtype m_dtype
 

Detailed Description

template<typename Scalar, LineSearchType LSType = MoreThuente>
class NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >

AdaMax optimizer.


Nadam update rule. Nadam is an optimizer that combines the effect of Adam and NAG to the gradient descent to improve its Performance. As name suggests the idea is to use Nesterov momentum term for the first moving averages. Let’s take a look at update rule of the SGD with momentum:

$$m_t=\beta m_{t-1}+\nu g_t$$ $$w_t=w_{t-1}-m_t=w_{t-1}\beta m_{t-1}+\nu g_t$$

As shown above, the update rule is equivalent to taking a step in the direction of momentum vector and then taking a step in the direction of gradient. However, the momentum step doesn’t depend on the current gradient , so we can get a higher-quality gradient step direction by updating the parameters with the momentum step before computing the gradient. To achieve that, we modify the update as follows:

$$g_t=\nabla f(w_{t-1}-\beta m_{t-1})$$ $$m_t=\beta m_{t-1}+\nu g_t$$ $$w_t=w_{t-1}-m_t$$

So, with Nesterov accelerated momentum we first make make a big jump in the direction of the previous accumulated gradient and then measure the gradient where we ended up to make a correction. The same method can be incorporated into Adam, by changing the first moving average to a Nesterov accelerated momentum. One computation trick can be applied here: instead of updating the parameters to make momentum step and changing back again, we can achieve the same effect by applying the momentum step of time step t + 1 only once, during the update of the previous time step t instead of t + 1. Using this trick, the implementation of Nadam may look like this:

for t in range(num_iterations):
g = compute_gradient(x, y)
m = beta_1 * m + (1 - beta_1) * g
v = beta_2 * v + (1 - beta_2) * np.power(g, 2)
m_hat = m / (1 - np.power(beta_1, t)) + (1 - beta_1) * g / (1 - np.power(beta_1, t))
v_hat = v / (1 - np.power(beta_2, t))
w = w - step_size * m_hat / (np.sqrt(v_hat) + epsilon)

References:

HmetalT, 02.05.2019.

Definition at line 80 of file FgNadamSolver.h.

Constructor & Destructor Documentation

◆ NadamSolver() [1/3]

template<typename Scalar , LineSearchType LSType = MoreThuente>
NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::NadamSolver ( int  numberOfVariables)

Creates a new instance of the L-BFGS optimization algorithm.

Admin, 3/27/2017.

Parameters
numberOfVariablesThe number of free parameters in the optimization problem.

◆ NadamSolver() [2/3]

template<typename Scalar , LineSearchType LSType = MoreThuente>
NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::NadamSolver ( int  numberOfVariables,
std::function< Scalar(const af::array &, af::array &)>  function 
)

Creates a new instance of the L-BFGS optimization algorithm.

Admin, 3/27/2017.

Parameters
numberOfVariablesThe number of free parameters in the function to be optimized.
function[in,out] The function to be optimized.
gradient[in,out] The gradient of the function.

◆ NadamSolver() [3/3]

template<typename Scalar , LineSearchType LSType = MoreThuente>
NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::NadamSolver ( NonlinearObjectiveFunction< Scalar > *  function)

Creates a new instance of the L-BFGS optimization algorithm.

Admin, 3/27/2017.

Parameters
functionThe objective function and gradients whose optimum values should be found.

◆ ~NadamSolver()

template<typename Scalar , LineSearchType LSType = MoreThuente>
NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::~NadamSolver ( )

Destructor.

, 15.08.2019.

Member Function Documentation

◆ SetBeta1()

template<typename Scalar , LineSearchType LSType = MoreThuente>
void NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::SetBeta1 ( Scalar  beta1)

Sets decay rate for the first moment estimates.

, 15.08.2019.

Parameters
beta1The first beta.

◆ SetBeta2()

template<typename Scalar , LineSearchType LSType = MoreThuente>
void NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::SetBeta2 ( Scalar  beta2)

Sets decay rate for the second-moment estimates.

, 15.08.2019.

Parameters
beta2The second beta.

◆ SetAlpha()

template<typename Scalar , LineSearchType LSType = MoreThuente>
void NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::SetAlpha ( Scalar  alpha)

Sets the learning rate.

, 15.08.2019.

Parameters
alphaThe alpha.

◆ SetEpsilon()

template<typename Scalar , LineSearchType LSType = MoreThuente>
void NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::SetEpsilon ( Scalar  epsilon)

Sets an epsilon to avoid division by zero.

, 15.08.2019.

Parameters
epsilonThe epsilon.

◆ SetDecay()

template<typename Scalar , LineSearchType LSType = MoreThuente>
void NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::SetDecay ( Scalar  decay)

Sets initial decay rate.

, 15.08.2019.

Parameters
decayThe decay.

◆ GetBeta1()

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::GetBeta1 ( )

Gets decay rate for the first moment estimates.

, 15.08.2019.

Returns
The beta 1.

◆ GetBeta2()

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::GetBeta2 ( )

Gets decay rate for the second-moment estimates.

, 15.08.2019.

Returns
The beta 2.

◆ GetAlpha()

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::GetAlpha ( )

Gets the learning rate.

, 15.08.2019.

Returns
The alpha.

◆ GetEpsilon()

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::GetEpsilon ( )

Gets the epsilon.

, 15.08.2019.

Returns
The epsilon.

◆ GetDecay()

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::GetDecay ( )

Gets the initial decay.

, 15.08.2019.

Returns
The decay.

◆ Optimize()

template<typename Scalar , LineSearchType LSType = MoreThuente>
virtual bool NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::Optimize ( int cycle = nullptr)
overrideprotectedvirtual

Implements the actual optimization algorithm. This method should try to minimize the objective function.

Hmetal T, 11.04.2017.

Returns
true if it succeeds, false if it fails.

Implements NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >.

Member Data Documentation

◆ min_step

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::min_step
private

Definition at line 230 of file FgNadamSolver.h.

◆ max_step

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::max_step
private

Definition at line 231 of file FgNadamSolver.h.

◆ sAlpha

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::sAlpha
private

Definition at line 233 of file FgNadamSolver.h.

◆ sBeta1

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::sBeta1
private

Definition at line 234 of file FgNadamSolver.h.

◆ sBeta2

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::sBeta2
private

Definition at line 235 of file FgNadamSolver.h.

◆ sEpsilon

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::sEpsilon
private

Definition at line 236 of file FgNadamSolver.h.

◆ sDecay

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::sDecay
private

Definition at line 237 of file FgNadamSolver.h.

◆ delta

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::delta
private

Definition at line 238 of file FgNadamSolver.h.

◆ sCumBeta1

template<typename Scalar , LineSearchType LSType = MoreThuente>
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::sCumBeta1
private

Definition at line 239 of file FgNadamSolver.h.


The documentation for this class was generated from the following file: