NeuralEngine
A Game Engine with embeded Machine Learning algorithms based on Gaussian Processes.
NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar > Class Template Reference

Sparse deep GPSSM via Approximated Expectation Propagation (AEP). More...

#include <FgAEPSparseDGPSSM.h>

Inheritance diagram for NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >:
Collaboration diagram for NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >:

Public Member Functions

 SDGPSSM (const af::array &Y, int latentDimension, HiddenLayerDescription description, Scalar alpha=1.0, Scalar priorMean=0.0, Scalar priorVariance=1.0, af::array xControl=af::array(), PropagationMode probMode=PropagationMode::MomentMatching, LogLikType lType=LogLikType::Gaussian, XInit emethod=XInit::pca)
 Constructor. More...
 
 SDGPSSM (const af::array &Y, int latentDimension, std::vector< HiddenLayerDescription > descriptions, Scalar alpha=1.0, Scalar priorMean=0.0, Scalar priorVariance=1.0, af::array xControl=af::array(), PropagationMode probMode=PropagationMode::MomentMatching, LogLikType lType=LogLikType::Gaussian, XInit emethod=XInit::pca)
 Constructor. More...
 
 SDGPSSM ()
 Default Constructor. More...
 
virtual ~SDGPSSM ()
 Destructor. More...
 
virtual Scalar Function (const af::array &x, af::array &outGradient) override
 Cost function the given parameter inputs. More...
 
- Public Member Functions inherited from NeuralEngine::MachineLearning::GPModels::SparseDeepGPSSMBaseModel< Scalar >
 SparseDeepGPSSMBaseModel (const af::array &Y, int latentDimension, HiddenLayerDescription description, Scalar priorMean=0.0, Scalar priorVariance=1.0, af::array &xControl=af::array(), PropagationMode probMode=PropagationMode::MomentMatching, LogLikType lType=LogLikType::Gaussian, XInit emethod=XInit::pca)
 Constructor. More...
 
 SparseDeepGPSSMBaseModel (const af::array &Y, int latentDimension, std::vector< HiddenLayerDescription > descriptions, Scalar priorMean=0.0, Scalar priorVariance=1.0, af::array &xControl=af::array(), PropagationMode probMode=PropagationMode::MomentMatching, LogLikType lType=LogLikType::Gaussian, XInit emethod=XInit::pca)
 Constructor. More...
 
 SparseDeepGPSSMBaseModel ()
 Default Constructor. More...
 
virtual ~SparseDeepGPSSMBaseModel ()
 Destructor. More...
 
virtual bool Init () override
 Initializes the model. More...
 
virtual void PredictF (const af::array &testInputs, af::array &mf, af::array &vf) override
 Predict noise free functions values \(\mathbf{F}_*\). More...
 
virtual void PredictForward (int numTimeSamples, af::array &my, af::array &vy, int numSamples=200, af::array *mx=nullptr, af::array *vx=nullptr)
 Noise-free Forward prediction. More...
 
virtual int GetNumParameters () override
 Generate function samples from posterior. More...
 
virtual int GetNumGPLayerParameters ()
 Gets number gp layer parameters. More...
 
virtual void SetParameters (const af::array &param) override
 Sets the parameters for each optimization iteration. More...
 
virtual af::array GetParameters () override
 Gets the parameters for each optimization iteration. More...
 
virtual void UpdateParameters () override
 Updates the parameters. More...
 
virtual int GetNumLayers ()
 Gets number of GP layers. More...
 
virtual std::vector< SparseGPBaseLayer< Scalar > * > GetGPLayers ()
 Gets vector of GP layers. More...
 
virtual void FixKernelParameters (bool isfixed)
 Sets fixation for hyperparameters. More...
 
- Public Member Functions inherited from NeuralEngine::MachineLearning::GPModels::GPSSBaseModel< Scalar >
 GPSSBaseModel (const af::array &Y, int latentDimension, Scalar priorMean=0.0, Scalar priorVariance=1.0, af::array &xControl=af::array(), PropagationMode probMode=PropagationMode::MomentMatching, LogLikType lType=LogLikType::Gaussian, bool GPemission=true, bool controlToEmiss=true, XInit emethod=XInit::pca)
 Constructor. More...
 
 GPSSBaseModel ()
 Default constructor. More...
 
virtual ~GPSSBaseModel ()
 Destructor. More...
 
virtual void Optimise (OptimizerType method=L_BFGS, Scalar tol=0.0, bool reinit_hypers=true, int maxiter=1000, int mb_size=0, LineSearchType lsType=MoreThuente, bool disp=true, int *cycle=nullptr) override
 Optimizes the model parameters for best fit. More...
 
virtual bool Init (af::array &mx)
 Initializes the model. More...
 
virtual void PosteriorLatents (af::array &mx, af::array &vx)
 Get posterior distribution of latent variables /f$\mathbf{X}/f$. More...
 
virtual int GetNumParameters () override
 Gets number of parameters. More...
 
virtual void SetParameters (const af::array &param) override
 Sets the parameters for each optimization iteration. More...
 
virtual af::array GetParameters () override
 Gets the parameters for each optimization iteration. More...
 
void UpdateDynamicIndexes ()
 Updates the dynamic indexes. More...
 
void GetLatents (af::array &mx, af::array &vx)
 
virtual void AddWindowData (af::array data)
 
- Public Member Functions inherited from NeuralEngine::MachineLearning::GPModels::GPBaseModel< Scalar >
 GPBaseModel (const af::array &Y, LogLikType lType=LogLikType::Gaussian, ModelType mtype=ModelType::GPR)
 Constructor. More...
 
 GPBaseModel ()
 Default Constructor. More...
 
virtual ~GPBaseModel ()
 Destructor. More...
 
virtual void Optimise (OptimizerType method=L_BFGS, Scalar tol=0.0, bool reinit_hypers=true, int maxiter=1000, int mb_size=0, LineSearchType lsType=MoreThuente, bool disp=true, int *cycle=nullptr)
 Optimizes the model parameters for best fit. More...
 
virtual bool Init ()
 Initializes the model. More...
 
virtual void PredictF (const af::array &testInputs, af::array &mf, af::array &vf)
 Predict noise free functions values \(\mathbf{F}_*\). More...
 
virtual void PredictY (const af::array &testInputs, af::array &my, af::array &vy)
 Prediction of test outputs \(\mathbf{Y}_*\). More...
 
virtual void SampleY (const af::array inputs, int numSamples, af::array &outFunctions)
 Generate function samples from posterior. More...
 
virtual void AddData (const af::array Ytrain)
 Adds training data to the model. More...
 
af::array GetTrainingData ()
 Gets the training data set Y. More...
 
void SetTrainingData (af::array &data)
 Sets training data Y. More...
 
virtual int GetNumParameters ()
 Gets number of parameters. More...
 
virtual void SetParameters (const af::array &param)
 Sets the parameters for each optimization iteration. More...
 
virtual af::array GetParameters ()
 Gets the parameters for each optimization iteration. More...
 
virtual void UpdateParameters ()
 Updates the parameters. More...
 
virtual void FixLikelihoodParameters (bool isfixed)
 Sets the likelihood parameters to be fixed or not for optimization. More...
 
void SetSegments (af::array segments)
 Sets fixation for hyperparameters. More...
 
af::array GetSegments ()
 Gets the start index array for the sequences. More...
 
- Public Member Functions inherited from NeuralEngine::MachineLearning::IModel< Scalar >
virtual Scalar Function (const af::array &x, af::array &outGradient)
 Cost function the given x inputs. More...
 
virtual int GetNumParameters ()=0
 Gets number of parameters to be optimized. More...
 
virtual void SetParameters (const af::array &param)=0
 Sets the parameters for each optimization iteration. More...
 
virtual af::array GetParameters ()=0
 Gets the parameters for each optimization iteration. More...
 
virtual void UpdateParameters ()=0
 Updates the parameters. More...
 
int GetDataLenght ()
 Gets data lenght. More...
 
int GetDataDimensionality ()
 Gets data dimensionality. More...
 
ModelType GetModelType ()
 Gets model type. More...
 
virtual void SetBatchSize (int size)
 Sets batch size. More...
 
int GetBatchSize ()
 Gets batch size. More...
 
void SetIndexes (af::array &indexes)
 Sets the batch indexes. More...
 
- Public Member Functions inherited from NeuralEngine::MachineLearning::GPNode< Scalar >
 GPNode ()
 Default constructor. More...
 
virtual ~GPNode ()
 Destructor. More...
 
int GetNumChildren () const
 Gets the number of children of this item. More...
 
int AttachChild (std::shared_ptr< GPNode< Scalar > > const &child)
 Attaches a child. More...
 
int DetachChild (std::shared_ptr< GPNode< Scalar > > const &child)
 Detaches a child. More...
 
std::shared_ptr< GPNode< Scalar > > DetachChildAt (int i)
 Detaches a child at index. More...
 
void DetachAllChildren ()
 Detach all children from this node. More...
 
std::shared_ptr< GPNode< Scalar > > SetChild (int i, std::shared_ptr< GPNode< Scalar > > const &child)
 Sets a child. More...
 
std::shared_ptr< GPNode< Scalar > > GetChild (int i)
 Gets a child at index. More...
 
GPNode< Scalar > * GetParent ()
 Access to the parent object, which is null for the root of the hierarchy. More...
 
void SetParent (GPNode< Scalar > *parent)
 
Access to the parent object. Node calls this during attach/detach of children. More...
 

Protected Member Functions

virtual void CavityLatents (af::array &mcav, af::array &vcav, af::array &cav1, af::array &cav2)
 Computes the cavity distribution. More...
 
Scalar ComputeTiltedTransition (const af::array &mprob, const af::array &vprob, const af::array &mcav_t1, const af::array &vcav_t1, Scalar scaleLogZDyn, af::array &dlogZ_dmProb, af::array &dlogZ_dvProb, af::array &dlogZ_dmt, af::array &dlogZ_dvt, Scalar &dlogZ_sn)
 Calculates the tilted transition. More...
 
virtual af::array PosteriorGradientLatents ()
 Posterior gradient w.r.t \(\mathbf{X}\). More...
 
virtual af::array CavityGradientLatents (const af::array &cav1, const af::array &cav2)
 Cavity gradient w.r.t \(\mathbf{X}\). More...
 
virtual af::array LogZGradientLatents (const af::array &cav1, const af::array &cav2, const af::array &dmcav_up, const af::array &dvcav_up, const af::array &dmcav_prev, const af::array &dvcav_prev, const af::array &dmcav_next, const af::array &dvcav_next)
 LogZ gradient w.r.t \(\mathbf{X}\). More...
 
virtual Scalar ComputePhiPriorLatents ()
 Calculates the phi prior. More...
 
virtual Scalar ComputePhiCavityLatents ()
 Calculates the phi cavity. More...
 
virtual Scalar ComputePhiPosteriorLatents ()
 Calculates the phi posterior. More...
 
- Protected Member Functions inherited from NeuralEngine::MachineLearning::GPModels::SparseDeepGPSSMBaseModel< Scalar >
virtual void PredictForwardMM (int numTimeSamples, af::array &my, af::array &vy, af::array *mx=nullptr, af::array *vx=nullptr)
 
virtual void PredictForwardMC (int numTimeSamples, af::array &my, af::array &vy, int numSamples=200, af::array *mx=nullptr)
 
- Protected Member Functions inherited from NeuralEngine::MachineLearning::GPModels::GPSSBaseModel< Scalar >
virtual af::array PosteriorGradientLatents (const af::array &dmx, const af::array &dvx)
 Posterior gradient of latent inputs /f$\mathbf{X}/f$. More...
 
virtual void UpdateParametersInternal ()
 Updates the parameters. More...
 
- Protected Member Functions inherited from NeuralEngine::MachineLearning::IModel< Scalar >
 IModel (int numData, int numDimension, ModelType type)
 Constructor. More...
 

Private Member Functions

template<class Archive >
void serialize (Archive &ar, unsigned int version)
 

Private Attributes

Scalar dAlpha
 fraction parameter More...
 

Friends

class boost::serialization::access
 

Additional Inherited Members

- Protected Attributes inherited from NeuralEngine::MachineLearning::GPModels::SparseDeepGPSSMBaseModel< Scalar >
int iNumLayer
 
std::vector< intvNumPseudosPerLayer
 
std::vector< intvSize
 
std::vector< HiddenLayerDescriptionvDescription
 
SparseGPBaseLayer< Scalar > * dynLayer
 sparse Gaussian Process dynamic layer
More...
 
std::vector< SparseGPBaseLayer< Scalar > * > gpEmissLayer
 sparse Gaussian Process emission layer More...
 
- Protected Attributes inherited from NeuralEngine::MachineLearning::GPModels::GPSSBaseModel< Scalar >
int iq
 latent dimension More...
 
int iDControlEmiss
 
int iDControlDyn
 
Scalar dSn
 
bool bGPemission
 
bool bControlToEmiss
 
Scalar dPriorMean
 prior mean More...
 
Scalar dPriorVariance
 prior variance More...
 
Scalar dPriorX1
 prior /f$x_1/f$ More...
 
Scalar dPriorX2
 prior /f$x_2/f$ More...
 
af::array afFactorX1
 natural parameter factor 1 for latent variable More...
 
af::array afFactorX2
 natural parameter factor 2 for latent variable More...
 
af::array afPosteriorX1
 posterior natural parameter 1 for latent variable More...
 
af::array afPosteriorX2
 posterior natural parameter 2 for latent variable More...
 
af::array afXControl
 
af::array afDynIndexes
 
af::array afPriorX1
 prior /f$x_1/f$ More...
 
af::array afPriorX2
 prior /f$x_2/f$ More...
 
af::array afPriorX1Cav
 prior /f$x_1/f$ More...
 
af::array afPriorX2Cav
 prior /f$x_2/f$ More...
 
af::array afPriorMean
 prior mean for hierarchy mode More...
 
af::array afPriorVariance
 prior variance for hierarchy mode More...
 
af::array afPriorMeanCav
 prior mean for hierarchy mode More...
 
af::array afPriorVarianceCav
 prior variance for hierarchy mode More...
 
af::array afGradMean
 prior mean gradient for hiersrchy mode More...
 
af::array afGradVariance
 prior variance gradient for hiersrchy mode More...
 
af::array afGradMeanCav
 prior mean gradient for hiersrchy mode More...
 
af::array afGradVarianceCav
 prior variance gradient for hiersrchy mode More...
 
af::array afLatentGradientX
 top down gradient More...
 
XInit eEmMethod
 
PropagationMode pMode
 
- Protected Attributes inherited from NeuralEngine::MachineLearning::GPModels::GPBaseModel< Scalar >
bool bInit
 check if model is initialized More...
 
af::array afY
 training dataset, mean substracted More...
 
af::array afBias
 the bias More...
 
af::array afSegments
 Index of starting positions for all trials. More...
 
LikelihoodBaseLayer< Scalar > * likLayer
 liklihood layer More...
 
- Protected Attributes inherited from NeuralEngine::MachineLearning::IModel< Scalar >
ModelType mType
 
int iN
 dataset length More...
 
int iD
 dataset dimension More...
 
int iBatchSize
 size of the batch More...
 
af::array afIndexes
 indexes of /f$\mathbf{X}/f$ for batch learning More...
 
af::dtype m_dType
 floating point precision flag for af::array More...
 
- Protected Attributes inherited from NeuralEngine::MachineLearning::GPNode< Scalar >
std::vector< std::shared_ptr< GPNode< Scalar > > > mChild
 
GPNode< Scalar > * mParent
 

Detailed Description

template<typename Scalar>
class NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >

Sparse deep GPSSM via Approximated Expectation Propagation (AEP).

Instead of taking one Gaussian portion out to form the cavity, we take out a fraction defined by the parameter \(\alpha\), which can also be seen as a ratio parameter between VFE and PowerEp with FITC approximation. This enables deep structures for GPLVM.


GPLVM are the nonlinear dual version of probabilistic PCA, where a low dimensional latent variable \(\mathbf{X}=[\mathbf{x}_1,...,\mathbf{x}_N]^T$\f is mapped onto a high dimensional data variable \)\mathbf{Y}=[\mathbf{y}_1,...,\mathbf{y}_N]^T$\f via prior mapping function \(f(\mathbf{x})$\f. The difference to normal GPs is the uncertainty of \)\mathbf{X}$\f, which will be initialized via PCA and optimized during learning. To avoid memory issues for larger data sets, the algorithm takes use of sparse approximation techniques.

Sparse approximations are used for larger data sets to reduce memory size and computational complexity. This is done by introducing a subset of inducing points or pseudo inputs to approximate the full set. The inversion of the kernel matrix depends only on those points and reduces the computational complexity from \(O(N^3)$\) to $$O(k^2N)$\f, where \(k\) is the number of inducing points and \(N\) the length of the data set.

References:

, 24.11.2019.

Definition at line 66 of file FgAEPSparseDGPSSM.h.

Constructor & Destructor Documentation

◆ SDGPSSM() [1/3]

template<typename Scalar >
NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::SDGPSSM ( const af::array &  Y,
int  latentDimension,
HiddenLayerDescription  description,
Scalar  alpha = 1.0,
Scalar  priorMean = 0.0,
Scalar  priorVariance = 1.0,
af::array  xControl = af::array(),
PropagationMode  probMode = PropagationMode::MomentMatching,
LogLikType  lType = LogLikType::Gaussian,
XInit  emethod = XInit::pca 
)

Constructor.

, 12.06.2018.

Parameters
YThe training data.
XThe training inputs.
hiddenLayerdescriptionThe description for one hidden layer.
alpha(Optional) The alpha.
lType(Optional) likelihood type.

◆ SDGPSSM() [2/3]

template<typename Scalar >
NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::SDGPSSM ( const af::array &  Y,
int  latentDimension,
std::vector< HiddenLayerDescription descriptions,
Scalar  alpha = 1.0,
Scalar  priorMean = 0.0,
Scalar  priorVariance = 1.0,
af::array  xControl = af::array(),
PropagationMode  probMode = PropagationMode::MomentMatching,
LogLikType  lType = LogLikType::Gaussian,
XInit  emethod = XInit::pca 
)

Constructor.

, 26.03.2018.

Parameters
YThe data af::array to process.
XThe training inputs.
hiddenLayerdescriptionsThe hidden layer descriptions.
lType(Optional) the loglik type.

◆ SDGPSSM() [3/3]

Default Constructor.

Hmetal T, 29.11.2017.

◆ ~SDGPSSM()

template<typename Scalar >
virtual NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::~SDGPSSM ( )
virtual

Destructor.

, 23.04.2018.

Member Function Documentation

◆ Function()

template<typename Scalar >
virtual Scalar NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::Function ( const af::array &  x,
af::array &  outGradient 
)
overridevirtual

Cost function the given parameter inputs.

Hmetal T, 29.11.2017.

Parameters
xThe parameters to be optimized.
outGradient[in,out] The out gradient.
Returns
A Scalar.

Reimplemented from NeuralEngine::MachineLearning::IModel< Scalar >.

◆ CavityLatents()

template<typename Scalar >
virtual void NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::CavityLatents ( af::array &  mcav,
af::array &  vcav,
af::array &  cav1,
af::array &  cav2 
)
protectedvirtual

Computes the cavity distribution.

Hmetal T, 11/05/2020.

Parameters
mcav[in,out] The mcav.
vcav[in,out] The vcav.
cav1[in,out] The first cav.
cav2[in,out] The second cav.

◆ ComputeTiltedTransition()

template<typename Scalar >
Scalar NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::ComputeTiltedTransition ( const af::array &  mprob,
const af::array &  vprob,
const af::array &  mcav_t1,
const af::array &  vcav_t1,
Scalar  scaleLogZDyn,
af::array &  dlogZ_dmProb,
af::array &  dlogZ_dvProb,
af::array &  dlogZ_dmt,
af::array &  dlogZ_dvt,
Scalar dlogZ_sn 
)
protected

Calculates the tilted transition.

Hmetal T, 11/05/2020.

Parameters
mprobThe mprob.
vprobThe vprob.
mcav_t1The first mcav t.
vcav_t1The first vcav t.
scaleLogZDynThe scale log z coordinate dynamic.
dmProb[in,out] The dm prob.
dvProb[in,out] The dv prob.
dmt[in,out] The dmt.
dvt[in,out] The dvt.
Returns
The calculated tilted transition.

◆ PosteriorGradientLatents()

template<typename Scalar >
virtual af::array NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::PosteriorGradientLatents ( )
protectedvirtual

Posterior gradient w.r.t \(\mathbf{X}\).

Hmetal T, 11/05/2020.

Returns
An af::array.

◆ CavityGradientLatents()

template<typename Scalar >
virtual af::array NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::CavityGradientLatents ( const af::array &  cav1,
const af::array &  cav2 
)
protectedvirtual

Cavity gradient w.r.t \(\mathbf{X}\).

Hmetal T, 11/05/2020.

Parameters
cav1The first cav.
cav2The second cav.
Returns
An af::array.

◆ LogZGradientLatents()

template<typename Scalar >
virtual af::array NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::LogZGradientLatents ( const af::array &  cav1,
const af::array &  cav2,
const af::array &  dmcav_up,
const af::array &  dvcav_up,
const af::array &  dmcav_prev,
const af::array &  dvcav_prev,
const af::array &  dmcav_next,
const af::array &  dvcav_next 
)
protectedvirtual

LogZ gradient w.r.t \(\mathbf{X}\).

Hmetal T, 11/05/2020.

Parameters
cav1The first cav.
cav2The second cav.
dmcav_upThe dmcav up.
dvcav_upThe dvcav up.
dmcav_prevThe dmcav previous.
dvcav_prevThe dvcav previous.
dmcav_nextThe dmcav next.
dvcav_nextThe dvcav next.
Returns
An af::array.

◆ ComputePhiPriorLatents()

template<typename Scalar >
virtual Scalar NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::ComputePhiPriorLatents ( )
protectedvirtual

Calculates the phi prior.

Hmetal T, 11/05/2020.

Returns
The calculated phi prior latents.

◆ ComputePhiCavityLatents()

template<typename Scalar >
virtual Scalar NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::ComputePhiCavityLatents ( )
protectedvirtual

Calculates the phi cavity.

Hmetal T, 11/05/2020.

Returns
The calculated phi cavity latents.

◆ ComputePhiPosteriorLatents()

template<typename Scalar >
virtual Scalar NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::ComputePhiPosteriorLatents ( )
protectedvirtual

Calculates the phi posterior.

Hmetal T, 11/05/2020.

Returns
The calculated phi posterior latents.

◆ serialize()

template<typename Scalar >
template<class Archive >
void NeuralEngine::MachineLearning::GPModels::AEP::SDGPSSM< Scalar >::serialize ( Archive &  ar,
unsigned int  version 
)
inlineprivate

Definition at line 232 of file FgAEPSparseDGPSSM.h.

Friends And Related Function Documentation

◆ boost::serialization::access

template<typename Scalar >
friend class boost::serialization::access
friend

Definition at line 229 of file FgAEPSparseDGPSSM.h.

Member Data Documentation

◆ dAlpha

fraction parameter

Definition at line 227 of file FgAEPSparseDGPSSM.h.


The documentation for this class was generated from the following file: