ml.optimize#

class HistorySA[source]#

Bases: TypedDict

iteration: list#
loss: list#
point: Optional[list]#
type_ball: tuple#
class NueSGD[source]#

Bases: object

__init__(model, lr=0.0001)[source]#

Implementation of classic SGD (stochastic gradient descent) optimization algorithm.

Parameters:

  • model (Module) – pytorch model that can be called and have a “.loss” method

  • lr (float) – learning rate. Multiplier of gradient step: x = x - lr * grad(x)

optimize(x, y, epochs=1, batch_size=-1, num_verbose=0, lamb=0.3, print_function=<built-in function print>)[source]#

Function apply MySGD optimizer, and train model.

Parameters:

  • x (Tensor) – training set

  • y (Tensor) – target value

  • epochs (int) – max number of sgd implements

  • batch_size (int) – size of batch for each epoch. default is -1 - all data

  • num_verbose (int) – number of iterations to be printed

  • lamb (float) – rate of history loss evaluation

  • print_function (Callable) – e.g. print or streamlit.write or something else

Returns:

trained model and history

Return type:

[<class ‘torch.nn.modules.module.Module’>, <class ‘dict’>]

step()[source]#

Update parameters data

W = W - lr * Grad(W)

Returns:

None

Return type:

None

zero_grad()[source]#

Make the gradients equal to zero

Returns:

None

Return type:

None

class SimulatedAnnealing[source]#

Bases: object

__init__(model, type_center='neighborhood', init_temp=1000000, radius=1, temp_multiplier=0.95)[source]#

Initialization of SimulatedAnnealing algorithm. Minimize real number models (non-discrete)

Parameters:

  • model (Module) – some pytorch model

  • type_center (Literal['zero', 'neighborhood']) – if type_center is zero, new point (x_k+1) would be chosen from Uniform[-radius, radius] for each parameter, elif neighborhood, new point would be chosen from Uniform[x_k - radius, x_k + radius).

  • init_temp (float) – initial temperature. Default is 10_000

  • radius (float) – ball’s radius

  • temp_multiplier (float) –

optimize(x, y)[source]#
Parameters:

  • x (Tensor) –

  • y (Tensor) –

Return type:

[<class ‘torch.nn.modules.module.Module’>, <class ‘nueramic_mathml.ml.optimize.HistorySA’>]

optimize_generator(x, y)[source]#

Generator of Simulated Annealing steps. [1]

\(\rule{125mm}{0.7pt} \\\) \(c = x_{pre} \text{ if type area is `neighborhood' else } c = \theta - \text{zero} \\\) \(x_{cur} \sim \mathcal{U}(c, r) \qquad p \sim \mathcal{U}[0, 1]\\\)

\(\text{if } f(x_{cur}) < f(x_{best}): \\\) \(\qquad x_{pre} = x_{best} = x_{cur}\\\)

\(\text{elif } \displaystyle \exp\left(\frac{f(x_{pre}) - f(x_{cur})}{T}\right) > p:\\\) \(\qquad x_{pre} = x_{cur}\\\)

\(T = T \cdot \delta\) \(\rule{125mm}{0.7pt} \\\)

Parameters:

  • x (Tensor) – training set

  • y (Tensor) – target value

Returns:

verbose strign with iteration and loss

Return type:

str

>>> torch.random.manual_seed(7)

>>> xr = torch.rand(100, 3)
>>> w = torch.tensor([[1., 2., 3.]]).T
>>> yr = xr @ w + 2

>>> model = torch.nn.Sequential(torch.nn.Linear(3, 1))
>>> model.loss = lambda _x, _y: torch.nn.MSELoss()(model(_x), _y)

>>> optimizer = SimulatedAnnealing(model, temp_multiplier=0.01)

>>> for verbose in optimizer.optimize(xr, yr):
>>>     print(verbose)
iteration:    1 | loss: 97.5745
iteration:    2 | loss: 231.5806
iteration:    3 | loss: 3.4633
iteration:    4 | loss: 3.7009
iteration:    5 | loss: 26.9238
iteration:    6 | loss: 6.5509
iteration:    7 | loss: 21.4261

>>> model.loss(xr, yr)
tensor(3.4633, grad_fn=<MseLossBackward0>)

References