From 4a72160aade7d3c286643dfe270b75810c775ab8 Mon Sep 17 00:00:00 2001 From: GaetanMarceauCaron Date: Fri, 15 Apr 2016 16:18:46 +0200 Subject: Adding a default value for gamma depending on the minibatch size --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index dffeb17..c613822 100644 --- a/README.md +++ b/README.md @@ -233,7 +233,7 @@ The algorithm is defined in Riemannian metrics for neural networks I: feedforwar To use this module, simply replace `nn.Linear(ninput,noutput)` with `nnx.QDRiemaNNLinear(ninput,noutput)`. As always, the step-size must be chosen accordingly. Two additional arguments are also possible: -* gamma (default=0.01): determine the update rate of the metric for a minibatch setting, i.e., (1-gamma) * oldMetric + gamma newMetric. Smaller minibatches require a smaller gamma. +* gamma (default=0.01): determine the update rate of the metric for a minibatch setting, i.e., (1-gamma) * oldMetric + gamma newMetric. Smaller minibatches require a smaller gamma. A default value depending on the size of the minibatches is `gamma = 1. - torch.pow(1.-1./nTraining,miniBatchSize)` where `nTraining` is the number of training examples of the dataset and `miniBatchSize` is the number of training examples per minibatch. * qdFlag (default=true): Whether to use the quasi-diagonal reduction (true) or only the diagonal (false). The former should be better. To implement a natural gradient descent, one should also use a module for generating the pseudo-labels. -- cgit v1.2.3