npdl.optimizers
¶
Functions to generate Theano update dictionaries for training.
The update functions implement different methods to control the learning rate for use with stochastic gradient descent.
Update functions take a loss expression or a list of gradient expressions and a list of parameters as input and return an ordered dictionary of updates:
Examples¶
Using SGD
to define an update dictionary for a toy
example network:
>>> import npdl
>>> from npdl.activation import ReLU
>>> from npdl.activation import Softmax
>>> from npdl.objectives import SCCE
>>> model = npdl.model.Model()
>>> model.add(npdl.layers.Dense(n_out=100, n_in=50, activation=ReLU()))
>>> model.add(npdl.layers.Dense(n_out=200, activation=ReLU()))
>>> model.add(npdl.layers.Dense(n_out=100, activation=ReLU()))
>>> model.add(npdl.layers.Dense(n_out=10, activation=Softmax()))
>>> model.compile(loss=SCCE(), optimizer=npdl.optimizers.SGD(lr=0.005))
Optimizers¶
SGD |
Stochastic Gradient Descent (SGD) updates |
Momentum |
Stochastic Gradient Descent (SGD) updates with momentum |
NesterovMomentum |
Stochastic Gradient Descent (SGD) updates with Nesterov momentum |
Adagrad |
Adagrad updates |
RMSprop |
RMSProp updates |
Adadelta |
Adadelta updates |
Adam |
Adam updates |
Adamax |
Adamax updates |
Detailed Description¶
-
class
npdl.optimizers.
SGD
(lr=0.001, clip=-1)[source]¶ Stochastic Gradient Descent (SGD) updates
Generates update expressions of the form:
param := param - learning_rate * gradient
Parameters: loss_or_grads : symbolic expression or list of expressions
A scalar loss expression, or a list of gradient expressions
params : list of shared variables
The variables to generate update expressions for
learning_rate : float or symbolic scalar
The learning rate controlling the size of update steps
Returns: OrderedDict
A dictionary mapping each parameter to its update expression
-
class
npdl.optimizers.
Momentum
(lr=0.01, momentum=0.9)[source]¶ Stochastic Gradient Descent (SGD) updates with momentum
Generates update expressions of the form:
velocity := momentum * velocity - learning_rate * gradient
param := param + velocity
Parameters: loss_or_grads : symbolic expression or list of expressions
A scalar loss expression, or a list of gradient expressions
params : list of shared variables
The variables to generate update expressions for
learning_rate : float or symbolic scalar
The learning rate controlling the size of update steps
momentum : float or symbolic scalar, optional
The amount of momentum to apply. Higher momentum results in smoothing over more update steps. Defaults to 0.9.
Returns: OrderedDict
A dictionary mapping each parameter to its update expression
See also
apply_momentum
- Generic function applying momentum to updates
nesterov_momentum
- Nesterov’s variant of SGD with momentum
Notes
Higher momentum also results in larger update steps. To counter that, you can optionally scale your learning rate by 1 - momentum.
-
class
npdl.optimizers.
NesterovMomentum
[source]¶ Stochastic Gradient Descent (SGD) updates with Nesterov momentum
Generates update expressions of the form:
velocity := momentum * velocity - learning_rate * gradient
param := param + momentum * velocity - learning_rate * gradient
Parameters: loss_or_grads : symbolic expression or list of expressions
A scalar loss expression, or a list of gradient expressions
params : list of shared variables
The variables to generate update expressions for
learning_rate : float or symbolic scalar
The learning rate controlling the size of update steps
momentum : float or symbolic scalar, optional
The amount of momentum to apply. Higher momentum results in smoothing over more update steps. Defaults to 0.9.
Returns: OrderedDict
A dictionary mapping each parameter to its update expression
See also
apply_nesterov_momentum
- Function applying momentum to updates
Notes
Higher momentum also results in larger update steps. To counter that, you can optionally scale your learning rate by 1 - momentum.
The classic formulation of Nesterov momentum (or Nesterov accelerated gradient) requires the gradient to be evaluated at the predicted next position in parameter space. Here, we use the formulation described at https://github.com/lisa-lab/pylearn2/pull/136#issuecomment-10381617, which allows the gradient to be evaluated at the current parameters.
-
class
npdl.optimizers.
Adagrad
[source]¶ Adagrad updates
Scale learning rates by dividing with the square root of accumulated squared gradients. See [R15] for further description.
Parameters: loss_or_grads : symbolic expression or list of expressions
A scalar loss expression, or a list of gradient expressions
params : list of shared variables
The variables to generate update expressions for
learning_rate : float or symbolic scalar
The learning rate controlling the size of update steps
epsilon : float or symbolic scalar
Small value added for numerical stability
Returns: OrderedDict
A dictionary mapping each parameter to its update expression
Notes
Using step size eta Adagrad calculates the learning rate for feature i at time step t as:
\[\eta_{t,i} = \frac{\eta} {\sqrt{\sum^t_{t^\prime} g^2_{t^\prime,i}+\epsilon}} g_{t,i}\]as such the learning rate is monotonically decreasing.
Epsilon is not included in the typical formula, see [R16].
References
[R15] (1, 2) Duchi, J., Hazan, E., & Singer, Y. (2011): Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121-2159. [R16] (1, 2) Chris Dyer: Notes on AdaGrad. http://www.ark.cs.cmu.edu/cdyer/adagrad.pdf
-
class
npdl.optimizers.
RMSprop
[source]¶ RMSProp updates
Scale learning rates by dividing with the moving average of the root mean squared (RMS) gradients. See [R17] for further description.
Parameters: loss_or_grads : symbolic expression or list of expressions
A scalar loss expression, or a list of gradient expressions
params : list of shared variables
The variables to generate update expressions for
learning_rate : float or symbolic scalar
The learning rate controlling the size of update steps
rho : float or symbolic scalar
Gradient moving average decay factor
epsilon : float or symbolic scalar
Small value added for numerical stability
Returns: OrderedDict
A dictionary mapping each parameter to its update expression
Notes
rho should be between 0 and 1. A value of rho close to 1 will decay the moving average slowly and a value close to 0 will decay the moving average fast.
Using the step size \(\eta\) and a decay factor \(\rho\) the learning rate \(\eta_t\) is calculated as:
\[\begin{split}r_t &= \rho r_{t-1} + (1-\rho)*g^2\\ \eta_t &= \frac{\eta}{\sqrt{r_t + \epsilon}}\end{split}\]References
[R17] (1, 2) Tieleman, T. and Hinton, G. (2012): Neural Networks for Machine Learning, Lecture 6.5 - rmsprop. Coursera. http://www.youtube.com/watch?v=O3sxAc4hxZU (formula @5:20)
-
class
npdl.optimizers.
Adadelta
[source]¶ Adadelta updates
Scale learning rates by the ratio of accumulated gradients to accumulated updates, see [R18] and notes for further description.
Parameters: loss_or_grads : symbolic expression or list of expressions
A scalar loss expression, or a list of gradient expressions
params : list of shared variables
The variables to generate update expressions for
learning_rate : float or symbolic scalar
The learning rate controlling the size of update steps
rho : float or symbolic scalar
Squared gradient moving average decay factor
epsilon : float or symbolic scalar
Small value added for numerical stability
Returns: OrderedDict
A dictionary mapping each parameter to its update expression
Notes
rho should be between 0 and 1. A value of rho close to 1 will decay the moving average slowly and a value close to 0 will decay the moving average fast.
rho = 0.95 and epsilon=1e-6 are suggested in the paper and reported to work for multiple datasets (MNIST, speech).
In the paper, no learning rate is considered (so learning_rate=1.0). Probably best to keep it at this value. epsilon is important for the very first update (so the numerator does not become 0).
Using the step size eta and a decay factor rho the learning rate is calculated as:
\[\begin{split}r_t &= \rho r_{t-1} + (1-\rho)*g^2\\ \eta_t &= \eta \frac{\sqrt{s_{t-1} + \epsilon}} {\sqrt{r_t + \epsilon}}\\ s_t &= \rho s_{t-1} + (1-\rho)*(\eta_t*g)^2\end{split}\]References
[R18] (1, 2) Zeiler, M. D. (2012): ADADELTA: An Adaptive Learning Rate Method. arXiv Preprint arXiv:1212.5701.
-
class
npdl.optimizers.
Adam
[source]¶ Adam updates
Adam updates implemented as in [R19].
Parameters: loss_or_grads : symbolic expression or list of expressions
A scalar loss expression, or a list of gradient expressions
params : list of shared variables
The variables to generate update expressions for
learning_rate : float or symbolic scalar
Learning rate
beta1 : float or symbolic scalar
Exponential decay rate for the first moment estimates.
beta2 : float or symbolic scalar
Exponential decay rate for the second moment estimates.
epsilon : float or symbolic scalar
Constant for numerical stability.
Returns: OrderedDict
A dictionary mapping each parameter to its update expression
Notes
The paper [R19] includes an additional hyperparameter lambda. This is only needed to prove convergence of the algorithm and has no practical use (personal communication with the authors), it is therefore omitted here.
References
[R19] (1, 2, 3) Kingma, Diederik, and Jimmy Ba (2014): Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.
-
class
npdl.optimizers.
Adamax
[source]¶ Adamax updates
Adamax updates implemented as in [R20]. This is a variant of of the Adam algorithm based on the infinity norm.
Parameters: loss_or_grads : symbolic expression or list of expressions
A scalar loss expression, or a list of gradient expressions
params : list of shared variables
The variables to generate update expressions for
learning_rate : float or symbolic scalar
Learning rate
beta1 : float or symbolic scalar
Exponential decay rate for the first moment estimates.
beta2 : float or symbolic scalar
Exponential decay rate for the weighted infinity norm estimates.
epsilon : float or symbolic scalar
Constant for numerical stability.
Returns: OrderedDict
A dictionary mapping each parameter to its update expression
References
[R20] (1, 2) Kingma, Diederik, and Jimmy Ba (2014): Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.