Gradient descent with constant step#

$\rule{125mm}{0.7pt}$

Algorithm Flowchart

$\rule{125mm}{0.7pt}$

$gradient descent with fractional step algorithm$ $gradient descent with fractional step algorithm$

$\rule{125mm}{0.7pt}$

gd_constant(function, x0, epsilon=1e-05, gamma=0.1, max_iter=500, verbose=False, keep_history=False)[source]#

Returns a tensor n x 1 with optimal point and history using Algorithm with constant step. The gradient of the function shows us the direction of increasing the function. The idea is to move in the opposite direction to $\displaystyle x_{k + 1} \text{ where } f(x_{k + 1}) < f(x_{k}) \text{ .}$

But, if we add a gradient to $\displaystyle x_{k}$ without changes, our method will often diverge. So we need to add a gradient with some weight $\displaystyle \gamma \text{ .}\\$

Parameters:

function (Callable[[Tensor], Tensor]) – callable that depends on the first positional argument
x0 (Tensor) – Torch tensor which is initial approximation
epsilon (float) – optimization accuracy
gamma (float) – gradient step
max_iter (int) – maximum number of iterations
verbose (bool) – flag of printing iteration logs
keep_history (bool) – flag of return history

Returns:

tuple with point and history.

Return type:

Tuple[Tensor, HistoryGD]

Examples

>>> def func(x): return x[0] ** 2 + x[1] ** 2
>>> x_0 = torch.tensor([1, 2])
>>> solution = gd_constant(func, x_0)
>>> print(solution[0])
tensor([1.9156e-06, 3.8312e-06], dtype=torch.float64)