AdaMax optimizer. More...
#include <FgNadamSolver.h>
Public Member Functions | |
NadamSolver (int numberOfVariables) | |
Creates a new instance of the L-BFGS optimization algorithm. More... | |
NadamSolver (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function) | |
Creates a new instance of the L-BFGS optimization algorithm. More... | |
NadamSolver (NonlinearObjectiveFunction< Scalar > *function) | |
Creates a new instance of the L-BFGS optimization algorithm. More... | |
~NadamSolver () | |
Destructor. More... | |
void | SetBeta1 (Scalar beta1) |
Sets decay rate for the first moment estimates. More... | |
void | SetBeta2 (Scalar beta2) |
Sets decay rate for the second-moment estimates. More... | |
void | SetAlpha (Scalar alpha) |
Sets the learning rate. More... | |
void | SetEpsilon (Scalar epsilon) |
Sets an epsilon to avoid division by zero. More... | |
void | SetDecay (Scalar decay) |
Sets initial decay rate. More... | |
Scalar | GetBeta1 () |
Gets decay rate for the first moment estimates. More... | |
Scalar | GetBeta2 () |
Gets decay rate for the second-moment estimates. More... | |
Scalar | GetAlpha () |
Gets the learning rate. More... | |
Scalar | GetEpsilon () |
Gets the epsilon. More... | |
Scalar | GetDecay () |
Gets the initial decay. More... | |
![]() | |
Scalar | GetTolerance () |
Gets the relative difference threshold to be used as stopping criteria between two iterations. Default is 0 (iterate until convergence). More... | |
void | SetTolerance (Scalar tolerance) |
Sets the relative difference threshold to be used as stopping criteria between two iterations. Default is 0 (iterate until convergence). More... | |
int | GetMaxIterations () |
Gets the maximum number of iterations to be performed during optimization. Default is 0 (iterate until convergence). More... | |
void | SetMaxIterations (int iter) |
Sets the maximum number of iterations to be performed during optimization. Default is 0 (iterate until convergence). More... | |
int | GetIterations () |
Gets the number of iterations performed in the last call to IOptimizationMethod.Minimize(). More... | |
![]() | |
virtual int | GetNumberOfVariables () |
Gets the number of variables (free parameters) in the optimization problem. More... | |
virtual af::array | GetSolution () |
Gets the current solution found, the values of the parameters which optimizes the function. More... | |
virtual void | SetSolution (af::array &x) |
Sets the current solution found, the values of the parameters which optimizes the function. More... | |
virtual Scalar | GetValue () |
Gets the output of the function at the current Solution. More... | |
virtual bool | Maximize (af::array &values, int *cycle=nullptr) |
Finds the maximum value of a function. The solution vector will be made available at the Solution property. More... | |
virtual bool | Minimize (af::array &values, int *cycle=nullptr) |
Finds the minimum value of a function. The solution vector will be made available at the Solution property. More... | |
virtual bool | Maximize (int *cycle=nullptr) |
Finds the maximum value of a function. The solution vector will be made available at the Solution property. More... | |
virtual bool | Minimize (int *cycle=nullptr) |
Finds the minimum value of a function. The solution vector will be made available at the Solution property. More... | |
void | Display (bool display) |
Set to display optimization information. More... | |
virtual int | GetNumberOfVariables ()=0 |
Gets the number of variables (free parameters) in the optimization problem. More... | |
virtual af::array | GetSolution ()=0 |
Gets the current solution found, the values of the parameters which optimizes the function. More... | |
virtual void | SetSolution (af::array &x)=0 |
Gets a solution. More... | |
virtual Scalar | GetValue ()=0 |
Gets the output of the function at the current Solution. More... | |
virtual bool | Minimize (int *cycle=nullptr)=0 |
Finds the minimum value of a function. The solution vector will be made available at the Solution property. More... | |
virtual bool | Maximize (int *cycle=nullptr)=0 |
Finds the maximum value of a function. The solution vector will be made available at the Solution property. More... | |
Protected Member Functions | |
virtual bool | Optimize (int *cycle=nullptr) override |
Implements the actual optimization algorithm. This method should try to minimize the objective function. More... | |
![]() | |
BaseGradientOptimizationMethod (int numberOfVariables) | |
Initializes a new instance of the BaseGradientOptimizationMethod class. More... | |
BaseGradientOptimizationMethod (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function) | |
Initializes a new instance of the BaseGradientOptimizationMethod class. More... | |
BaseGradientOptimizationMethod (NonlinearObjectiveFunction< Scalar > *function) | |
Initializes a new instance of the BaseGradientOptimizationMethod class. More... | |
void | InitLinesearch () |
Inits linesearch. More... | |
![]() | |
void | SetValue (Scalar v) |
Sets the output of the function at the current Solution. More... | |
void | SetNumberOfVariables (int n) |
Sets the number of variables (free parameters) in the optimization problem. More... | |
BaseOptimizationMethod (int numberOfVariables) | |
Initializes a new instance of the BaseOptimizationMethod class. More... | |
BaseOptimizationMethod (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function) | |
Initializes a new instance of the BaseOptimizationMethod class. More... | |
BaseOptimizationMethod (NonlinearObjectiveFunction< Scalar > *function) | |
Initializes a new instance of the BaseOptimizationMethod class. More... | |
virtual bool | Optimize (int *cycle=nullptr)=0 |
Implements the actual optimization algorithm. This method should try to minimize the objective function. More... | |
Private Attributes | |
Scalar | min_step |
Scalar | max_step |
Scalar | sAlpha |
Scalar | sBeta1 |
Scalar | sBeta2 |
Scalar | sEpsilon |
Scalar | sDecay |
Scalar | delta |
Scalar | sCumBeta1 |
Additional Inherited Members | |
![]() | |
int | maxIterations |
Scalar | _tolerance |
int | iterations |
ILineSearch< Scalar > * | linesearch |
![]() | |
NonlinearObjectiveFunction< Scalar > * | _function |
af::array | _x |
bool | _display |
af::dtype | m_dtype |
AdaMax optimizer.
Nadam update rule. Nadam is an optimizer that combines the effect of Adam and NAG to the gradient descent to improve its Performance. As name suggests the idea is to use Nesterov momentum term for the first moving averages. Let’s take a look at update rule of the SGD with momentum:
$$m_t=\beta m_{t-1}+\nu g_t$$ $$w_t=w_{t-1}-m_t=w_{t-1}\beta m_{t-1}+\nu g_t$$
As shown above, the update rule is equivalent to taking a step in the direction of momentum vector and then taking a step in the direction of gradient. However, the momentum step doesn’t depend on the current gradient , so we can get a higher-quality gradient step direction by updating the parameters with the momentum step before computing the gradient. To achieve that, we modify the update as follows:
$$g_t=\nabla f(w_{t-1}-\beta m_{t-1})$$ $$m_t=\beta m_{t-1}+\nu g_t$$ $$w_t=w_{t-1}-m_t$$
So, with Nesterov accelerated momentum we first make make a big jump in the direction of the previous accumulated gradient and then measure the gradient where we ended up to make a correction. The same method can be incorporated into Adam, by changing the first moving average to a Nesterov accelerated momentum. One computation trick can be applied here: instead of updating the parameters to make momentum step and changing back again, we can achieve the same effect by applying the momentum step of time step t + 1 only once, during the update of the previous time step t instead of t + 1. Using this trick, the implementation of Nadam may look like this:
References:
HmetalT, 02.05.2019.
Definition at line 80 of file FgNadamSolver.h.
NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::NadamSolver | ( | int | numberOfVariables | ) |
Creates a new instance of the L-BFGS optimization algorithm.
Admin, 3/27/2017.
numberOfVariables | The number of free parameters in the optimization problem. |
NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::NadamSolver | ( | int | numberOfVariables, |
std::function< Scalar(const af::array &, af::array &)> | function | ||
) |
Creates a new instance of the L-BFGS optimization algorithm.
Admin, 3/27/2017.
numberOfVariables | The number of free parameters in the function to be optimized. |
function | [in,out] The function to be optimized. |
gradient | [in,out] The gradient of the function. |
NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::NadamSolver | ( | NonlinearObjectiveFunction< Scalar > * | function | ) |
Creates a new instance of the L-BFGS optimization algorithm.
Admin, 3/27/2017.
function | The objective function and gradients whose optimum values should be found. |
NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::~NadamSolver | ( | ) |
Destructor.
, 15.08.2019.
void NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::SetBeta1 | ( | Scalar | beta1 | ) |
Sets decay rate for the first moment estimates.
, 15.08.2019.
beta1 | The first beta. |
void NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::SetBeta2 | ( | Scalar | beta2 | ) |
Sets decay rate for the second-moment estimates.
, 15.08.2019.
beta2 | The second beta. |
void NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::SetAlpha | ( | Scalar | alpha | ) |
Sets the learning rate.
, 15.08.2019.
alpha | The alpha. |
void NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::SetEpsilon | ( | Scalar | epsilon | ) |
Sets an epsilon to avoid division by zero.
, 15.08.2019.
epsilon | The epsilon. |
void NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::SetDecay | ( | Scalar | decay | ) |
Sets initial decay rate.
, 15.08.2019.
decay | The decay. |
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::GetBeta1 | ( | ) |
Gets decay rate for the first moment estimates.
, 15.08.2019.
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::GetBeta2 | ( | ) |
Gets decay rate for the second-moment estimates.
, 15.08.2019.
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::GetAlpha | ( | ) |
Gets the learning rate.
, 15.08.2019.
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::GetEpsilon | ( | ) |
Gets the epsilon.
, 15.08.2019.
Scalar NeuralEngine::MachineLearning::NadamSolver< Scalar, LSType >::GetDecay | ( | ) |
Gets the initial decay.
, 15.08.2019.
|
overrideprotectedvirtual |
Implements the actual optimization algorithm. This method should try to minimize the objective function.
Hmetal T, 11.04.2017.
Implements NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >.
|
private |
Definition at line 230 of file FgNadamSolver.h.
|
private |
Definition at line 231 of file FgNadamSolver.h.
|
private |
Definition at line 233 of file FgNadamSolver.h.
|
private |
Definition at line 234 of file FgNadamSolver.h.
|
private |
Definition at line 235 of file FgNadamSolver.h.
|
private |
Definition at line 236 of file FgNadamSolver.h.
|
private |
Definition at line 237 of file FgNadamSolver.h.
|
private |
Definition at line 238 of file FgNadamSolver.h.
|
private |
Definition at line 239 of file FgNadamSolver.h.