NeuralEngine A Game Engine with embeded Machine Learning algorithms based on Gaussian Processes. |

Public Member Functions | Protected Member Functions | Private Attributes | List of all members
NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType > Class Template Reference

AdaMax optimizer. More...

`#include <FgAdaMaxSolver.h>`

Inheritance diagram for NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >:
[legend]
Collaboration diagram for NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >:
[legend]

## Public Member Functions
| AdaMaxSolver (int numberOfVariables)
| Creates a new instance of the L-BFGS optimization algorithm. More...
| AdaMaxSolver (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function)
| Creates a new instance of the L-BFGS optimization algorithm. More...
| AdaMaxSolver (NonlinearObjectiveFunction< Scalar > *function)
| Creates a new instance of the L-BFGS optimization algorithm. More...
| ~AdaMaxSolver ()
| Destructor. More...
void | SetBeta1 (Scalar beta1)
| Sets decay rate for the first moment estimates. More...
void | SetBeta2 (Scalar beta2)
| Sets decay rate for the second-moment estimates. More...
void | SetAlpha (Scalar alpha)
| Sets the learning rate. More...
void | SetEpsilon (Scalar epsilon)
| Sets an epsilon to avoid division by zero. More...
void | SetDecay (Scalar decay)
| Sets initial decay rate. More...
Scalar | GetBeta1 ()
| Gets decay rate for the first moment estimates. More...
Scalar | GetBeta2 ()
| Gets decay rate for the second-moment estimates. More...
Scalar | GetAlpha ()
| Gets the learning rate. More...
Scalar | GetEpsilon ()
| Gets the epsilon. More...
Scalar | GetDecay ()
| Gets the initial decay. More...
Public Member Functions inherited from NeuralEngine::MachineLearning::BaseGradientOptimizationMethod< Scalar, MoreThuente >
Scalar | GetTolerance ()
| Gets the relative difference threshold to be used as stopping criteria between two iterations. Default is 0 (iterate until convergence). More...
void | SetTolerance (Scalar tolerance)
| Sets the relative difference threshold to be used as stopping criteria between two iterations. Default is 0 (iterate until convergence). More...
int | GetMaxIterations ()
| Gets the maximum number of iterations to be performed during optimization. Default is 0 (iterate until convergence). More...
void | SetMaxIterations (int iter)
| Sets the maximum number of iterations to be performed during optimization. Default is 0 (iterate until convergence). More...
int | GetIterations ()
| Gets the number of iterations performed in the last call to IOptimizationMethod.Minimize(). More...
Public Member Functions inherited from NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >
virtual int | GetNumberOfVariables ()
| Gets the number of variables (free parameters) in the optimization problem. More...
virtual af::array | GetSolution ()
| Gets the current solution found, the values of the parameters which optimizes the function. More...
virtual void | SetSolution (af::array &x)
| Sets the current solution found, the values of the parameters which optimizes the function. More...
virtual Scalar | GetValue ()
| Gets the output of the function at the current Solution. More...
virtual bool | Maximize (af::array &values, int *cycle=nullptr)
| Finds the maximum value of a function. The solution vector will be made available at the Solution property. More...
virtual bool | Minimize (af::array &values, int *cycle=nullptr)
| Finds the minimum value of a function. The solution vector will be made available at the Solution property. More...
virtual bool | Maximize (int *cycle=nullptr)
| Finds the maximum value of a function. The solution vector will be made available at the Solution property. More...
virtual bool | Minimize (int *cycle=nullptr)
| Finds the minimum value of a function. The solution vector will be made available at the Solution property. More...
void | Display (bool display)
| Set to display optimization information. More...
virtual int | GetNumberOfVariables ()=0
| Gets the number of variables (free parameters) in the optimization problem. More...
virtual af::array | GetSolution ()=0
| Gets the current solution found, the values of the parameters which optimizes the function. More...
virtual void | SetSolution (af::array &x)=0
| Gets a solution. More...
virtual Scalar | GetValue ()=0
| Gets the output of the function at the current Solution. More...
virtual bool | Minimize (int *cycle=nullptr)=0
| Finds the minimum value of a function. The solution vector will be made available at the Solution property. More...
virtual bool | Maximize (int *cycle=nullptr)=0
| Finds the maximum value of a function. The solution vector will be made available at the Solution property. More...

## Protected Member Functions
virtual bool | Optimize (int *cycle=nullptr) override
| Implements the actual optimization algorithm. This method should try to minimize the objective function. More...
Protected Member Functions inherited from NeuralEngine::MachineLearning::BaseGradientOptimizationMethod< Scalar, MoreThuente >
| BaseGradientOptimizationMethod (int numberOfVariables)
| Initializes a new instance of the BaseGradientOptimizationMethod class. More...
| BaseGradientOptimizationMethod (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function)
| Initializes a new instance of the BaseGradientOptimizationMethod class. More...
| BaseGradientOptimizationMethod (NonlinearObjectiveFunction< Scalar > *function)
| Initializes a new instance of the BaseGradientOptimizationMethod class. More...
void | InitLinesearch ()
| Inits linesearch. More...
Protected Member Functions inherited from NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >
void | SetValue (Scalar v)
| Sets the output of the function at the current Solution. More...
void | SetNumberOfVariables (int n)
| Sets the number of variables (free parameters) in the optimization problem. More...
| BaseOptimizationMethod (int numberOfVariables)
| Initializes a new instance of the BaseOptimizationMethod class. More...
| BaseOptimizationMethod (int numberOfVariables, std::function< Scalar(const af::array &, af::array &)> function)
| Initializes a new instance of the BaseOptimizationMethod class. More...
| BaseOptimizationMethod (NonlinearObjectiveFunction< Scalar > *function)
| Initializes a new instance of the BaseOptimizationMethod class. More...
virtual bool | Optimize (int *cycle=nullptr)=0
| Implements the actual optimization algorithm. This method should try to minimize the objective function. More...

## Private Attributes
Scalar | min_step
Scalar | max_step
Scalar | sAlpha
Scalar | sBeta1
Scalar | sBeta2
Scalar | sEpsilon
Scalar | sDecay
Scalar | delta

## Additional Inherited Members
Protected Attributes inherited from NeuralEngine::MachineLearning::BaseGradientOptimizationMethod< Scalar, MoreThuente >
int | maxIterations
Scalar | _tolerance
int | iterations
ILineSearch< Scalar > * | linesearch
Protected Attributes inherited from NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >
NonlinearObjectiveFunction< Scalar > * | _function
af::array | _x
bool | _display
af::dtype | m_dtype

## Detailed Description

template<typename Scalar, LineSearchType LSType = MoreThuente>
class NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >

AdaMax optimizer.

AdaMax update rule. Adam is an an algorithm for first-order gradient- -based optimization of stochastic objective functions, based on adaptive estimates of lower - order moments.AdaMax is simply a variant of Adam based n the infinity norm. The idea with Adamax is to look at the value v as the L2 norm of the individual current and past gradients. We can generalize it to Lp update rule, but it gets pretty unstable for large values of p. But if we use the special case of L-infinity norm, it results in a surprisingly stable and well-performing algorithm. Here’s how to implement Adamax with python:

for t in range(num_iterations):
g = compute_gradient(x, y)
m = beta_1 * m + (1 - beta_1) * g
m_hat = m / (1 - np.power(beta_1, t))
v = np.maximum(beta_2 * v, np.abs(g))
w = w - step_size * m_hat / v

References:

- Diederik P. Kingma and Jimmy Ba (2014). "Adam: A Method for Stochastic Optimization". CoRR.

HmetalT, 02.05.2019.

Definition at line 59 of file FgAdaMaxSolver.h.

## Constructor & Destructor Documentation

## ◆ AdaMaxSolver() [1/3]

template<typename Scalar , LineSearchType LSType = MoreThuente>

NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::AdaMaxSolver | ( | int | numberOfVariables | ) |

Creates a new instance of the L-BFGS optimization algorithm.

Admin, 3/27/2017.

Parameters

  numberOfVariables | The number of free parameters in the optimization problem.

## ◆ AdaMaxSolver() [2/3]

template<typename Scalar , LineSearchType LSType = MoreThuente>

NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::AdaMaxSolver | ( | int | numberOfVariables,
|  | std::function< Scalar(const af::array &, af::array &)> | function
| ) |  |

Creates a new instance of the L-BFGS optimization algorithm.

Admin, 3/27/2017.

Parameters

  numberOfVariables | The number of free parameters in the function to be optimized.
  function | [in,out] The function to be optimized.
  gradient | [in,out] The gradient of the function.

## ◆ AdaMaxSolver() [3/3]

template<typename Scalar , LineSearchType LSType = MoreThuente>

NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::AdaMaxSolver | ( | NonlinearObjectiveFunction< Scalar > * | function | ) |

Creates a new instance of the L-BFGS optimization algorithm.

Admin, 3/27/2017.

Parameters

  function | The objective function and gradients whose optimum values should be found.

## ◆ ~AdaMaxSolver()

template<typename Scalar , LineSearchType LSType = MoreThuente>

NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::~AdaMaxSolver | ( |  | ) |

Destructor.

, 15.08.2019.

## Member Function Documentation

## ◆ SetBeta1()

template<typename Scalar , LineSearchType LSType = MoreThuente>

void NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::SetBeta1 | ( | Scalar | beta1 | ) |

Sets decay rate for the first moment estimates.

, 15.08.2019.

Parameters

  beta1 | The first beta.

## ◆ SetBeta2()

template<typename Scalar , LineSearchType LSType = MoreThuente>

void NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::SetBeta2 | ( | Scalar | beta2 | ) |

Sets decay rate for the second-moment estimates.

, 15.08.2019.

Parameters

  beta2 | The second beta.

## ◆ SetAlpha()

template<typename Scalar , LineSearchType LSType = MoreThuente>

void NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::SetAlpha | ( | Scalar | alpha | ) |

Sets the learning rate.

, 15.08.2019.

Parameters

  alpha | The alpha.

## ◆ SetEpsilon()

template<typename Scalar , LineSearchType LSType = MoreThuente>

void NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::SetEpsilon | ( | Scalar | epsilon | ) |

Sets an epsilon to avoid division by zero.

, 15.08.2019.

Parameters

  epsilon | The epsilon.

## ◆ SetDecay()

template<typename Scalar , LineSearchType LSType = MoreThuente>

void NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::SetDecay | ( | Scalar | decay | ) |

Sets initial decay rate.

, 15.08.2019.

Parameters

  decay | The decay.

## ◆ GetBeta1()

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::GetBeta1 | ( |  | ) |

Gets decay rate for the first moment estimates.

, 15.08.2019.

Returns
  The beta 1.

## ◆ GetBeta2()

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::GetBeta2 | ( |  | ) |

Gets decay rate for the second-moment estimates.

, 15.08.2019.

Returns
  The beta 2.

## ◆ GetAlpha()

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::GetAlpha | ( |  | ) |

Gets the learning rate.

, 15.08.2019.

Returns
  The alpha.

## ◆ GetEpsilon()

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::GetEpsilon | ( |  | ) |

Gets the epsilon.

, 15.08.2019.

Returns
  The epsilon.

## ◆ GetDecay()

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::GetDecay | ( |  | ) |

Gets the initial decay.

, 15.08.2019.

Returns
  The decay.

## ◆ Optimize()

template<typename Scalar , LineSearchType LSType = MoreThuente>

virtual bool NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::Optimize | ( | int * | cycle = `nullptr` | ) |
| override protected virtual

Implements the actual optimization algorithm. This method should try to minimize the objective function.

Hmetal T, 11.04.2017.

Returns
  true if it succeeds, false if it fails.

Implements NeuralEngine::MachineLearning::BaseOptimizationMethod< Scalar >.

## Member Data Documentation

## ◆ min_step

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::min_step
| private

Definition at line 209 of file FgAdaMaxSolver.h.

## ◆ max_step

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::max_step
| private

Definition at line 210 of file FgAdaMaxSolver.h.

## ◆ sAlpha

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::sAlpha
| private

Definition at line 212 of file FgAdaMaxSolver.h.

## ◆ sBeta1

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::sBeta1
| private

Definition at line 213 of file FgAdaMaxSolver.h.

## ◆ sBeta2

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::sBeta2
| private

Definition at line 214 of file FgAdaMaxSolver.h.

## ◆ sEpsilon

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::sEpsilon
| private

Definition at line 215 of file FgAdaMaxSolver.h.

## ◆ sDecay

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::sDecay
| private

Definition at line 216 of file FgAdaMaxSolver.h.

## ◆ delta

template<typename Scalar , LineSearchType LSType = MoreThuente>

Scalar NeuralEngine::MachineLearning::AdaMaxSolver< Scalar, LSType >::delta
| private

Definition at line 217 of file FgAdaMaxSolver.h.

The documentation for this class was generated from the following file:

- FgAdaMaxSolver.h