Neural network optimizers

VTL optimizers update autograd.Variable[T] parameters after a loss has been backpropagated. Construct an optimizer, register the model layers once with build_params, then call update() after each backprop().

mut optimizer := optimizers.radam_optimizer[f32](learning_rate: 0.001)
optimizer.build_params(model.info.layers)
loss.backprop()!
optimizer.update()!

See the optimizer tutorial for end-to-end training examples and neural network tutorial for Sequential models.

Optimizers

SgdOptimizerConfig follows the SGD options momentum, dampening, weight_decay, and nesterov. The first momentum buffer is initialized from the regularized gradient, as in PyTorch; dampening applies on later updates. Nesterov requires positive momentum and zero dampening.

NAdam and RAdam configuration structs accept weight_decay (default 0). Set decoupled_weight_decay: true to apply decay directly to parameter values, without adding it to the gradient or moment estimates. NAdam additionally accepts momentum_decay (default 0.004).

Learning-rate schedulers

StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, CosineAnnealingWarmRestarts, LinearLR, PolynomialLR, ConstantLR, CyclicLR, OneCycleLR, and ReduceLROnPlateau update learning rates by epoch or minibatch step. MultiStepLR applies one decay for each milestone, including duplicate milestones. LinearLR interpolates between two factors, while PolynomialLR decays to zero over a fixed number of steps. ConstantLR scales the base rate for a fixed number of steps, then restores it. CosineAnnealingWarmRestarts resets to the base rate after each cosine cycle, with optional geometric growth of cycle lengths. OneCycleLR uses minibatch steps to ramp from max_lr / div_factor to max_lr, then anneal to max_lr / (div_factor * final_div_factor). It supports cosine or linear annealing, an optional three-phase curve, and inverse momentum cycling with configurable bounds. next_momentum(step) returns the value for the caller to apply; set cycle_momentum: false to disable it. The defaults (0.85 and 0.95) match PyTorch. Optimizer mutation and parameter groups are not implemented. Its constructor returns an error for negative max_lr, nonpositive total_steps or division factors, pct_start outside (0, 1), or invalid momentum bounds. ReduceLROnPlateau.step(current_lr, metric) consumes the monitored value and tracks the best metric, with min or max mode and absolute or relative thresholds. It supports patience, cooldown, min_lr, and epsilon and returns the rate for the caller to apply. try_reduce_lr_on_plateau reports invalid configuration; the original next_lr(current_lr, step, metric_delta) remains available for callers that already calculate metric deltas.

Versioned model training checkpoints support all seven built-in optimizers: SGD, AdaGrad, RMSProp, Adam, AdamW, NAdam, and RAdam. The NAdam checkpoint preserves its scheduled momentum product and step; RAdam preserves the step used by variance rectification. See the model checkpoint API for the save/load methods and their limits.

CyclicLR provides triangular, triangular2, and exp_range amplitude policies with independently configurable rise and fall durations. It also computes the inverse momentum curve through next_momentum(step) by default; set cycle_momentum: false to disable it. The returned momentum is a value for the caller to apply. Supply scale_fn to define a custom cycle amplitude and set scale_mode to .cycle (the default) or .iterations to choose whether the callback receives a one-based cycle number or the zero-based minibatch step, respectively. Optimizer mutation and parameter groups are not implemented.

import vtl.nn.optimizers

fn decay_by_cycle(cycle int) f64 {
    return 1.0 / f64(cycle + 1)
}

scheduler := optimizers.cyclic_lr[f64](optimizers.CyclicLRConfig{
    base_lr:      0.001
    max_lr:       0.01
    step_size_up: 100
    scale_fn:     decay_by_cycle
})!
lr := scheduler.next_lr(0.001, 150)

NAdam

NAdam incorporates Nesterov momentum into Adam. It uses the scheduled momentum coefficient μ_t = β₁(1 - 0.5 × 0.96^(t × momentum_decay)) and the corresponding bias-corrected look-ahead estimate. Its defaults follow PyTorch's NAdam API.

mut optimizer := optimizers.nadam_optimizer[f64](
    learning_rate: 0.002
    momentum_decay: 0.004
)

RAdam

RAdam rectifies the adaptive variance once its running estimate is reliable. It uses the unrectified bias-corrected momentum during early steps and the rectification factor after rho_t > 5, matching PyTorch's threshold and epsilon placement.

mut optimizer := optimizers.radam_optimizer[f64](
    learning_rate: 0.001
    weight_decay: 0.01
    decoupled_weight_decay: true
)

Validation

Run commands from the V module workspace, outside this repository:

cd ~/.vmodules
systemd-run --user --scope --quiet --property=MemoryMax=768M \
  --property=MemorySwapMax=0 --setenv=VJOBS=2 -- \
  v test ./vtl/nn/optimizers
systemd-run --user --scope --quiet --property=MemoryMax=768M \
  --property=MemorySwapMax=0 --setenv=VJOBS=2 -- \
  v -prod test ./vtl/nn/optimizers

In constrained environments, run each command under a systemd scope with an appropriate MemoryMax.

fn adagrad[T](config AdaGradOptimizerConfig) &AdaGradOptimizer[T]

adagrad creates a new AdaGradOptimizer.

fn adam_optimizer[T](config AdamOptimizerConfig) &AdamOptimizer[T]

adam_optimizer creates a new AdamOptimizer with the given configuration.

import vtl.nn.optimizers
opt := optimizers.adam_optimizer[f64](learning_rate: 0.001)
opt.build_params(model.layers())
// inside training loop:
opt.update()!

fn adam_step_f32_cpu(grad []f32, mut theta []f32, mut m []f32, mut v []f32, p AdamStepParams)

adam_step_f32_cpu performs one Adam update on flat f32 buffers (same math as napply path).

fn adam_step_f64(grad []f64, mut theta []f64, mut m []f64, mut v []f64, p AdamStepParams,
	session voidptr, slot int)

adam_step_f64 without CUDA build delegates to CPU (opaque session avoids autograd_cuda import).

fn adam_step_f64_cpu(grad []f64, mut theta []f64, mut m []f64, mut v []f64, p AdamStepParams)

adam_step_f64_cpu performs one Adam update on flat CPU buffers.

fn adam_use_cuda_optimizer() bool

adam_use_cuda_optimizer exposes this operation as part of the public API.

fn adam_use_vulkan_optimizer() bool

adam_use_vulkan_optimizer exposes this operation as part of the public API.

fn adamw[T](config AdamWOptimizerConfig) &AdamWOptimizer[T]

adamw creates a new AdamWOptimizer.

fn clip_grad_norm[T](mut parameters []&autograd.Variable[T], max_norm f64) !f64

clip_grad_norm scales trainable gradients in parameters in place when their global L2 norm exceeds max_norm. It returns the norm before clipping.

fn clip_grad_norm_with_type[T](mut parameters []&autograd.Variable[T], max_norm f64, norm_type f64) !f64

clip_grad_norm_with_type clips by a global p-norm; positive infinity selects the infinity norm. It returns the pre-clipping norm.

fn clip_grad_value[T](mut parameters []&autograd.Variable[T], max_value f64) !

clip_grad_value clamps every trainable gradient element to [-max_value, max_value].

fn clip_grads_with_norm[T](mut parameters []&autograd.Variable[T], max_norm f64, total_norm f64) !

clip_grads_with_norm scales trainable gradients in place using a previously computed total norm.

fn constant_lr[T](config ConstantLRConfig) !&ConstantLR[T]

constant_lr creates a temporary constant-rate scaling scheduler.

fn cosine_annealing_lr[T](t_max int, lrd f64) &CosineAnnealingLR[T]

cosine_annealing_lr creates a CosineAnnealingLR scheduler: LR decays from current_lr to lrd following a cosine schedule over t_max steps.

fn cosine_annealing_warm_restarts[T](config CosineAnnealingWarmRestartsConfig) !&CosineAnnealingWarmRestarts[T]

cosine_annealing_warm_restarts creates the scheduler. eta_min should not exceed the base learning rate passed to next_lr.

fn cyclic_lr[T](config CyclicLRConfig) !&CyclicLR[T]

cyclic_lr creates a CyclicLR scheduler. A zero step_size_down uses the same duration as step_size_up; all other durations must be positive.

fn exponential_lr[T](gamma f64) &ExponentialLR[T]

exponential_lr creates an ExponentialLR scheduler: LR decays by gamma at every step. new_lr = current_lr * gamma^step

fn get_total_grad_norm[T](parameters []&autograd.Variable[T]) !f64

get_total_grad_norm computes the L2 norm of all trainable gradients without changing them.

fn get_total_grad_norm_with_type[T](parameters []&autograd.Variable[T], norm_type f64) !f64

get_total_grad_norm_with_type computes a global p-norm without changing gradients; positive infinity selects the infinity norm.

fn linear_lr[T](start_factor f64, end_factor f64, total_iters int) &LinearLR[T]

linear_lr creates a linear learning-rate schedule. The returned value is based on base_lr and interpolates from start_factor to end_factor.

fn multi_step_lr[T](config MultiStepLRConfig) !&MultiStepLR[T]

multi_step_lr creates a scheduler whose milestones must be in increasing order. Repeated milestones apply repeated decay at the same step, like PyTorch.

fn nadam_optimizer[T](config NAdamOptimizerConfig) &NAdamOptimizer[T]

nadam_optimizer creates a NAdam optimizer.

fn nadam_step_f32_cpu(grad []f32, mut theta []f32, mut m []f32, mut v []f32, p NAdamStepParams)

nadam_step_f32_cpu applies one Dozat NAdam update in f32.

fn nadam_step_f64_cpu(grad []f64, mut theta []f64, mut m []f64, mut v []f64, p NAdamStepParams)

nadam_step_f64_cpu applies one Dozat NAdam update in f64.

fn one_cycle_lr[T](config OneCycleLRConfig) !&OneCycleLR[T]

one_cycle_lr creates a two-phase OneCycleLR by default. anneal_strategy selects cosine or linear interpolation. Invalid cycle parameters return an error.

fn polynomial_lr[T](config PolynomialLRConfig) !&PolynomialLR[T]

polynomial_lr creates a polynomial decay scheduler.

fn radam_optimizer[T](config RAdamOptimizerConfig) &RAdamOptimizer[T]

radam_optimizer creates a RAdam optimizer.

fn radam_step_f32_cpu(grad []f32, mut theta []f32, mut m []f32, mut v []f32, p RAdamStepParams)

radam_step_f32_cpu applies one Rectified Adam update in f32.

fn radam_step_f64_cpu(grad []f64, mut theta []f64, mut m []f64, mut v []f64, p RAdamStepParams)

radam_step_f64_cpu applies one Rectified Adam update in f64.

fn reduce_lr_on_plateau[T](config ReduceLROnPlateauConfig) &ReduceLROnPlateau[T]

reduce_lr_on_plateau creates a ReduceLROnPlateau scheduler. Use step with the monitored metric for PyTorch-style behavior. next_lr remains available for callers that already provide a per-step metric delta.

fn rmsprop[T](config RMSPropOptimizerConfig) &RMSPropOptimizer[T]

rmsprop creates a new RMSPropOptimizer.

fn sgd[T](config SgdOptimizerConfig) &SgdOptimizer[T]

sgd creates a new SgdOptimizer.

fn step_lr[T](step_size int, gamma f64) &StepLR[T]

step_lr creates a StepLR scheduler: LR decays by gamma every step_size steps. e.g. step_size=30, gamma=0.1 → LR is 10× smaller after each 30 steps.

fn try_adam_update_f32_vulkan(mut v autograd.Variable[f32], mut m_tensor vtl.Tensor[f32],
	mut v_tensor vtl.Tensor[f32], step AdamStepParams) bool

try_adam_update_f32_vulkan exposes this operation as part of the public API.

fn try_reduce_lr_on_plateau[T](config ReduceLROnPlateauConfig) !&ReduceLROnPlateau[T]

try_reduce_lr_on_plateau validates the configuration and creates a scheduler.

interface Scheduler[T] {
	next_lr(current_lr f64, step int) f64
}

Scheduler is the interface for learning rate schedulers. Scheduler is the interface for learning rate schedulers. Implementations update the learning rate based on the current training step.

fn (o &AdaGradOptimizer[T]) checkpoint_config() map[string]f64

checkpoint_config returns the scalar AdaGrad state needed to resume training.

fn (mut o AdaGradOptimizer[T]) restore_checkpoint_state(config map[string]f64,
	accumulated_sq_grads []&vtl.Tensor[T]) !

restore_checkpoint_state restores AdaGrad configuration and accumulated gradients.

fn (o &AdaGradOptimizer[T]) validate_checkpoint_state(config map[string]f64,
	accumulated_sq_grads []&vtl.Tensor[T]) !

validate_checkpoint_state checks AdaGrad scalars and accumulators without mutation.

fn (mut o AdaGradOptimizer[T]) add_param_group(params []&autograd.Variable[T], config AdaGradOptimizerConfig) !int

add_param_group registers parameters under their own AdaGrad configuration and returns the group index. Register each variable either flat (via build_params) or in exactly one group.

fn (o &AdaGradOptimizer[T]) group_count() int

group_count returns how many parameter groups are registered.

fn (o &AdaGradOptimizer[T]) group_learning_rate(group_index int) !f64

group_learning_rate returns the current learning rate of one group.

fn (mut o AdaGradOptimizer[T]) set_group_learning_rate(group_index int, learning_rate f64) !

set_group_learning_rate applies a new learning rate to one group only; this is the integration point for learning-rate schedulers.

fn (mut o AdaGradOptimizer[T]) build_params(layers []types.Layer[T])

build_params registers all trainable variables from layers. Call once before training.

fn (mut o AdaGradOptimizer[T]) update() !

update performs one AdaGrad parameter update and zeros all gradients.

fn (o &AdamOptimizer[T]) checkpoint_config() map[string]f64

checkpoint_config returns the scalar parameters needed to resume Adam.

fn (mut o AdamOptimizer[T]) restore_checkpoint_state(config map[string]f64,
	first_moments []&vtl.Tensor[T], second_moments []&vtl.Tensor[T]) !

restore_checkpoint_state restores Adam's scalar parameters and moment buffers. Parameters must already be registered with build_params in the same order.

fn (o &AdamOptimizer[T]) validate_checkpoint_state(config map[string]f64,
	first_moments []&vtl.Tensor[T], second_moments []&vtl.Tensor[T]) !

validate_checkpoint_state checks Adam state without changing the optimizer.

fn (mut o AdamOptimizer[T]) add_param_group(params []&autograd.Variable[T], config AdamOptimizerConfig) !int

add_param_group registers parameters under their own Adam configuration and returns the group index. Register each variable either flat (via build_params) or in exactly one group.

fn (o &AdamOptimizer[T]) group_count() int

group_count returns how many parameter groups are registered.

fn (o &AdamOptimizer[T]) group_learning_rate(group_index int) !f64

group_learning_rate returns the current learning rate of one group.

fn (mut o AdamOptimizer[T]) set_group_learning_rate(group_index int, learning_rate f64) !

set_group_learning_rate applies a new learning rate to one group only; this is the integration point for learning-rate schedulers.

fn (mut o AdamOptimizer[T]) build_params(layers []types.Layer[T])

build_params registers all trainable variables from layers into the optimizer. Call once after constructing the model, before the first update().

fn (mut o AdamOptimizer[T]) update() !

update performs one Adam parameter update step and zeros all gradients. Must be called after loss.backward().

fn (o &AdamWOptimizer[T]) checkpoint_config() map[string]f64

checkpoint_config returns the scalar AdamW state needed to resume training.

fn (mut o AdamWOptimizer[T]) restore_checkpoint_state(config map[string]f64,
	first_moments []&vtl.Tensor[T], second_moments []&vtl.Tensor[T]) !

restore_checkpoint_state restores AdamW configuration and moment buffers. Parameters must already be registered with build_params in the same order.

fn (o &AdamWOptimizer[T]) validate_checkpoint_state(config map[string]f64,
	first_moments []&vtl.Tensor[T], second_moments []&vtl.Tensor[T]) !

validate_checkpoint_state checks AdamW scalars and moments without mutation.

fn (mut o AdamWOptimizer[T]) add_param_group(params []&autograd.Variable[T], config AdamWOptimizerConfig) !int

add_param_group registers parameters under their own AdamW configuration and returns the group index. Register each variable either flat (via build_params) or in exactly one group.

fn (o &AdamWOptimizer[T]) group_count() int

group_count returns how many parameter groups are registered.

fn (o &AdamWOptimizer[T]) group_learning_rate(group_index int) !f64

group_learning_rate returns the current learning rate of one group.

fn (mut o AdamWOptimizer[T]) set_group_learning_rate(group_index int, learning_rate f64) !

set_group_learning_rate applies a new learning rate to one group only; this is the integration point for learning-rate schedulers.

fn (mut o AdamWOptimizer[T]) build_params(layers []types.Layer[T])

build_params registers all trainable variables from layers. Call once before training.

fn (mut o AdamWOptimizer[T]) update() !

update performs one AdamW parameter update and zeros all gradients.

fn (s &ConstantLR[T]) next_lr(base_lr f64, step int) f64

next_lr returns the scaled rate before total_iters and the base rate afterward.

fn (s &CosineAnnealingLR[T]) next_lr(current_lr f64, step int) f64

next_lr exposes this operation as part of the public API.

fn (s &CosineAnnealingWarmRestarts[T]) next_lr(base_lr f64, step int) f64

next_lr returns the rate for a zero-based step, with each cycle beginning at the base rate and annealing toward eta_min.

fn (s &CyclicLR[T]) next_momentum(step int) ?f64

next_momentum returns the inverse momentum curve for a zero-based minibatch step. It returns none when momentum cycling is disabled. Apply the returned value to an optimizer explicitly; this scheduler does not mutate optimizers.

fn (s &CyclicLR[T]) next_lr(_ f64, step int) f64

next_lr returns the rate for a zero-based minibatch step. The input rate is ignored because the schedule is defined by its base and maximum rates.

fn (s &ExponentialLR[T]) next_lr(current_lr f64, step int) f64

next_lr exposes this operation as part of the public API.

fn (s &LinearLR[T]) next_lr(base_lr f64, step int) f64

next_lr exposes this operation as part of the public API.

fn (s &MultiStepLR[T]) next_lr(base_lr f64, step int) f64

next_lr returns the rate for a zero-based epoch, based on the base rate.

fn (o &NAdamOptimizer[T]) checkpoint_config() map[string]f64

checkpoint_config returns the scalar NAdam state needed to resume training.

fn (mut o NAdamOptimizer[T]) restore_checkpoint_state(config map[string]f64,
	first_moments []&vtl.Tensor[T], second_moments []&vtl.Tensor[T]) !

restore_checkpoint_state restores NAdam scalars and moment buffers.

fn (o &NAdamOptimizer[T]) validate_checkpoint_state(config map[string]f64,
	first_moments []&vtl.Tensor[T], second_moments []&vtl.Tensor[T]) !

validate_checkpoint_state checks NAdam scalars and moments without mutation.

fn (mut o NAdamOptimizer[T]) add_param_group(params []&autograd.Variable[T], config NAdamOptimizerConfig) !int

add_param_group registers parameters under their own NAdam configuration and returns the group index. Register each variable either flat (via build_params) or in exactly one group.

fn (o &NAdamOptimizer[T]) group_count() int

group_count returns how many parameter groups are registered.

fn (o &NAdamOptimizer[T]) group_learning_rate(group_index int) !f64

group_learning_rate returns the current learning rate of one group.

fn (mut o NAdamOptimizer[T]) set_group_learning_rate(group_index int, learning_rate f64) !

set_group_learning_rate applies a new learning rate to one group only; this is the integration point for learning-rate schedulers.

fn (mut o NAdamOptimizer[T]) build_params(layers []types.Layer[T])

build_params registers trainable variables and initializes moment estimates.

fn (mut o NAdamOptimizer[T]) update() !

update performs one NAdam parameter update and zeros gradients.

fn (s &OneCycleLR[T]) next_lr(_ f64, step int) f64

next_lr returns the rate for a zero-based minibatch step. Steps outside the configured cycle are clamped to its first or final rate.

fn (s &OneCycleLR[T]) next_momentum(step int) ?f64

next_momentum returns the inverse momentum curve for a zero-based minibatch step. It returns none when cycling is disabled. Apply the returned value to an optimizer explicitly; this scheduler does not mutate optimizer state.

fn (s &PolynomialLR[T]) next_lr(base_lr f64, step int) f64

next_lr returns the polynomially decayed rate for a zero-based step.

fn (o &RAdamOptimizer[T]) checkpoint_config() map[string]f64

checkpoint_config returns the scalar RAdam state needed to resume training.

fn (mut o RAdamOptimizer[T]) restore_checkpoint_state(config map[string]f64,
	first_moments []&vtl.Tensor[T], second_moments []&vtl.Tensor[T]) !

restore_checkpoint_state restores RAdam scalars and moment buffers.

fn (o &RAdamOptimizer[T]) validate_checkpoint_state(config map[string]f64,
	first_moments []&vtl.Tensor[T], second_moments []&vtl.Tensor[T]) !

validate_checkpoint_state checks RAdam scalars and moments without mutation.

fn (mut o RAdamOptimizer[T]) build_params(layers []types.Layer[T])

build_params registers trainable variables and initializes moment estimates.

fn (mut o RAdamOptimizer[T]) add_param_group(params []&autograd.Variable[T], config RAdamOptimizerConfig) !int

add_param_group registers parameters under their own RAdam configuration and returns the group index. Register each variable either flat (via build_params) or in exactly one group.

fn (o &RAdamOptimizer[T]) group_count() int

group_count returns how many parameter groups are registered.

fn (o &RAdamOptimizer[T]) group_learning_rate(group_index int) !f64

group_learning_rate returns the current learning rate of one group.

fn (mut o RAdamOptimizer[T]) set_group_learning_rate(group_index int, learning_rate f64) !

set_group_learning_rate applies a new learning rate to one group only; this is the integration point for learning-rate schedulers.

fn (mut o RAdamOptimizer[T]) update() !

update performs one RAdam parameter update and zeros gradients.

fn (o &RMSPropOptimizer[T]) checkpoint_config() map[string]f64

checkpoint_config returns the scalar RMSProp state needed to resume training.

fn (mut o RMSPropOptimizer[T]) restore_checkpoint_state(config map[string]f64,
	sq_avg []&vtl.Tensor[T]) !

restore_checkpoint_state restores RMSProp configuration and squared-gradient buffers.

fn (o &RMSPropOptimizer[T]) validate_checkpoint_state(config map[string]f64,
	sq_avg []&vtl.Tensor[T]) !

validate_checkpoint_state checks RMSProp scalars and buffers without mutation.

fn (mut o RMSPropOptimizer[T]) add_param_group(params []&autograd.Variable[T], config RMSPropOptimizerConfig) !int

add_param_group registers parameters under their own RMSProp configuration and returns the group index. Register each variable either flat (via build_params) or in exactly one group.

fn (o &RMSPropOptimizer[T]) group_count() int

group_count returns how many parameter groups are registered.

fn (o &RMSPropOptimizer[T]) group_learning_rate(group_index int) !f64

group_learning_rate returns the current learning rate of one group.

fn (mut o RMSPropOptimizer[T]) set_group_learning_rate(group_index int, learning_rate f64) !

set_group_learning_rate applies a new learning rate to one group only; this is the integration point for learning-rate schedulers.

fn (mut o RMSPropOptimizer[T]) build_params(layers []types.Layer[T])

build_params registers all trainable variables from layers. Call once before training.

fn (mut o RMSPropOptimizer[T]) update() !

update performs one RMSProp parameter update and zeros all gradients.

fn (mut s ReduceLROnPlateau[T]) step(current_lr f64, metric f64) f64

step consumes a monitored metric and returns the PyTorch-style updated rate. A metric must beat the absolute or relative threshold to reset patience.

fn (mut s ReduceLROnPlateau[T]) next_lr(current_lr f64, _ int, metric_delta f64) f64

next_lr preserves the original delta-based API. A negative delta beyond -threshold means improvement; other finite deltas count toward patience.

fn (o &SgdOptimizer[T]) checkpoint_config() map[string]f64

checkpoint_config returns the scalar configuration and per-parameter momentum initialization state needed to resume SGD exactly.

fn (mut o SgdOptimizer[T]) restore_checkpoint_state(config map[string]f64,
	momentum_buffers []&vtl.Tensor[T]) !

restore_checkpoint_state restores SGD configuration, momentum buffers, and their initialization state. Parameters must already be registered in order.

fn (o &SgdOptimizer[T]) validate_checkpoint_state(config map[string]f64,
	momentum_buffers []&vtl.Tensor[T]) !

validate_checkpoint_state checks SGD configuration and momentum state without changing the optimizer.

fn (mut o SgdOptimizer[T]) build_params(layers []types.Layer[T])

build_params registers all trainable variables from layers. Call once before training.

fn (mut o SgdOptimizer[T]) add_param_group(params []&autograd.Variable[T], config SgdOptimizerConfig) !int

add_param_group registers parameters under their own SGD configuration and returns the group index. Register each variable either flat (via params or build_params) or in exactly one group. Group momentum buffers are created here, matching build_params behavior for the flat list.

fn (o &SgdOptimizer[T]) group_count() int

group_count returns how many parameter groups are registered.

fn (o &SgdOptimizer[T]) group_learning_rate(group_index int) !f64

group_learning_rate returns the current learning rate of one group.

fn (mut o SgdOptimizer[T]) set_group_learning_rate(group_index int, learning_rate f64) !

set_group_learning_rate applies a new learning rate to one group only. This is the integration point for learning-rate schedulers, which compute the next rate and hand it back for the caller to apply.

fn (mut o SgdOptimizer[T]) update() !

update performs one SGD parameter update and zeros all gradients.

fn (s &StepLR[T]) next_lr(current_lr f64, step int) f64

next_lr exposes this operation as part of the public API.

enum CyclicLRMode {
	triangular
	triangular2
	exp_range
}

CyclicLRMode selects the built-in cycle-amplitude policy.

enum CyclicLRScaleMode {
	cycle
	iterations
}

CyclicLRScaleMode selects whether a custom scale function receives the one-based cycle number or the zero-based minibatch step.

enum OneCycleAnnealStrategy {
	cosine
	linear
}

OneCycleLRConfig defines the learning-rate and momentum bounds of the one-cycle policy.

enum ReduceLROnPlateauMode {
	min
	max
}

ReduceLROnPlateau reduces LR when a metric has stopped improving.

enum ReduceLROnPlateauThresholdMode {
	rel
	abs
}

struct AdaGradOptimizer[T] {
	learning_rate f64
	epsilon       f64
pub mut:
	weight_decay         f64
	params               []&autograd.Variable[T]
	accumulated_sq_grads []&vtl.Tensor[T]
	groups               []&AdaGradParamGroup[T]
}

AdaGradOptimizer implements the AdaGrad (Adaptive Gradient) algorithm. Accumulates squared gradients and adapts the learning rate per parameter.

struct AdaGradOptimizerConfig {
pub:
	learning_rate f64 = 0.01
	epsilon       f64 = 1e-8
	weight_decay  f64 = 0.0
}

AdaGradOptimizerConfig defines a public data structure for this module.

struct AdaGradParamGroup[T] {
pub mut:
	params               []&autograd.Variable[T]
	learning_rate        f64
	epsilon              f64
	weight_decay         f64
	accumulated_sq_grads []&vtl.Tensor[T]
}

AdaGradParamGroup is a set of parameters sharing one AdaGrad configuration, mirroring PyTorch parameter groups, with its own accumulated squared-gradient buffers.

struct AdamOptimizer[T] {
	learning_rate f64
	epsilon       f64
pub mut:
	beta1          f64
	beta2          f64
	beta1_t        f64
	beta2_t        f64
	params         []&autograd.Variable[T]
	first_moments  []&vtl.Tensor[T]
	second_moments []&vtl.Tensor[T]
	groups         []&AdamParamGroup[T]
}

AdamOptimizer implements the Adam optimiser (Adaptive Moment Estimation).

Maintains per-parameter first-moment (mean) and second-moment (uncentred variance) moving averages of the gradients, with bias correction applied at each step.

Update rule: m = β₁·m + (1-β₁)·g v = β₂·v + (1-β₂)·g² θ = θ - lr · √(1-β₂ᵗ) / (1-β₁ᵗ) · m / (√v + ε)

Reference: Kingma & Ba, "Adam: A Method for Stochastic Optimization" (2014).

struct AdamOptimizerConfig {
pub:
	learning_rate f64 = 0.001
	beta1         f64 = 0.9
	beta2         f64 = 0.999
	epsilon       f64 = 1e-8
}

AdamOptimizerConfig defines a public data structure for this module.

struct AdamParamGroup[T] {
pub mut:
	params        []&autograd.Variable[T]
	learning_rate f64
	beta1         f64
	beta2         f64
	epsilon       f64
mut:
	beta1_t        f64
	beta2_t        f64
	first_moments  []&vtl.Tensor[T]
	second_moments []&vtl.Tensor[T]
}

AdamParamGroup is a set of parameters sharing one Adam configuration, mirroring PyTorch parameter groups. Each group keeps its own learning rate, betas, epsilon, bias-correction counters and moment buffers. Grouped parameters always take the CPU step; the CUDA and Vulkan hooks remain available through the flat params list.

struct AdamStepParams {
pub:
	beta1   f64
	beta2   f64
	lr_t    f64
	epsilon f64
}

AdamStepParams holds scalar Adam state for one flat parameter vector.

struct AdamWOptimizer[T] {
	learning_rate f64
	epsilon       f64
pub mut:
	beta1          f64
	beta2          f64
	beta1_t        f64
	beta2_t        f64
	weight_decay   f64
	params         []&autograd.Variable[T]
	first_moments  []&vtl.Tensor[T]
	second_moments []&vtl.Tensor[T]
	groups         []&AdamWParamGroup[T]
}

AdamWOptimizer implements AdamW (Adam with Decoupled Weight Decay).

Identical to Adam but weight decay is applied directly to the parameters (not through the gradient), which typically gives better generalisation.

Update rule (after bias correction): θ = θ - lr · (m̂ / (√v̂ + ε) + weight_decay · θ)

Reference: Loshchilov & Hutter, "Decoupled Weight Decay Regularization" (2017).

struct AdamWOptimizerConfig {
pub:
	learning_rate f64 = 0.001
	beta1         f64 = 0.9
	beta2         f64 = 0.999
	epsilon       f64 = 1e-8
	weight_decay  f64 = 0.01
}

AdamWOptimizerConfig defines a public data structure for this module.

struct AdamWParamGroup[T] {
pub mut:
	params        []&autograd.Variable[T]
	learning_rate f64
	beta1         f64
	beta2         f64
	epsilon       f64
	weight_decay  f64
mut:
	beta1_t        f64
	beta2_t        f64
	first_moments  []&vtl.Tensor[T]
	second_moments []&vtl.Tensor[T]
}

AdamWParamGroup is a set of parameters sharing one AdamW configuration, mirroring PyTorch parameter groups, with its own bias-correction counters and moment buffers.

struct ConstantLR[T] {
	factor      f64
	total_iters int
}

ConstantLR scales the base rate by factor for a fixed number of steps.

struct ConstantLRConfig {
pub:
	factor      f64 = 1.0 / 3.0
	total_iters int = 5
}

ConstantLRConfig defines a temporary constant scaling of the base rate.

struct CosineAnnealingLR[T] {
pub:
	t_max int // maximum number of iterations
	lrd   f64 // lower bound lr (default: 0)
}

CosineAnnealingLR decays using a cosine schedule from lrd to 0.

struct CosineAnnealingWarmRestarts[T] {
	t_0     int
	t_mult  int
	eta_min f64
}

CosineAnnealingWarmRestarts restarts the learning rate at the start of each cycle and can increase cycle lengths geometrically with t_mult.

struct CosineAnnealingWarmRestartsConfig {
pub:
	t_0     int
	t_mult  int = 1
	eta_min f64
}

CosineAnnealingWarmRestartsConfig defines a cosine schedule with periodic restarts.

struct CyclicLR[T] {
	base_lr        f64
	max_lr         f64
	step_size_up   int
	step_size_down int
	mode           CyclicLRMode
	gamma          f64
	cycle_momentum bool
	base_momentum  f64
	max_momentum   f64
	scale_fn       fn (int) f64 = unsafe { nil }
	scale_mode     CyclicLRScaleMode
}

CyclicLR cycles the learning rate between base_lr and max_lr. Momentum can be queried as an inverse cycle with next_momentum. The scheduler does not update optimizer state or support parameter groups.

struct CyclicLRConfig {
pub:
	base_lr        f64
	max_lr         f64
	step_size_up   int = 2000
	step_size_down int
	mode           CyclicLRMode = .triangular
	gamma          f64          = 1.0
	cycle_momentum bool         = true
	base_momentum  f64          = 0.8
	max_momentum   f64          = 0.9
	scale_fn       fn (int) f64 = unsafe { nil }
	scale_mode     CyclicLRScaleMode
}

CyclicLRConfig defines a minibatch-based triangular learning-rate schedule.

struct ExponentialLR[T] {
	gamma f64
}

ExponentialLR decays the learning rate by gamma at every step.

struct LinearLR[T] {
	start_factor f64
	end_factor   f64
	total_iters  int
}

LinearLR scales the base learning rate linearly between two factors.

struct MultiStepLR[T] {
	milestones []int
	gamma      f64
}

MultiStepLR decays the base learning rate at each configured milestone.

struct MultiStepLRConfig {
pub:
	milestones []int
	gamma      f64 = 0.1
}

MultiStepLRConfig defines the epochs at which the rate is decayed.

struct NAdamOptimizer[T] {
	learning_rate          f64
	epsilon                f64
	weight_decay           f64
	momentum_decay         f64
	decoupled_weight_decay bool
pub mut:
	beta1          f64
	beta2          f64
	beta1_t        f64
	beta2_t        f64
	mu_product_t   f64
	step           int
	params         []&autograd.Variable[T]
	first_moments  []&vtl.Tensor[T]
	second_moments []&vtl.Tensor[T]
	groups         []&NAdamParamGroup[T]
}

NAdamOptimizer implements Nesterov-accelerated Adam for f32 and f64 parameters. Reference: Dozat (2016), Incorporating Nesterov Momentum into Adam; PyTorch NAdam.

struct NAdamOptimizerConfig {
pub:
	learning_rate          f64 = 0.002
	beta1                  f64 = 0.9
	beta2                  f64 = 0.999
	epsilon                f64 = 1e-8
	weight_decay           f64
	momentum_decay         f64 = 0.004
	decoupled_weight_decay bool
}

NAdamOptimizerConfig configures NAdamOptimizer.

struct NAdamParamGroup[T] {
pub mut:
	params                 []&autograd.Variable[T]
	learning_rate          f64
	beta1                  f64
	beta2                  f64
	epsilon                f64
	weight_decay           f64
	momentum_decay         f64
	decoupled_weight_decay bool
mut:
	beta1_t        f64
	beta2_t        f64
	mu_product_t   f64
	step           int
	first_moments  []&vtl.Tensor[T]
	second_moments []&vtl.Tensor[T]
}

NAdamParamGroup is a set of parameters sharing one NAdam configuration, mirroring PyTorch parameter groups, with its own step counter, momentum product and moment buffers.

struct NAdamStepParams {
pub:
	beta1           f64
	beta2           f64
	beta1_t         f64
	beta2_t         f64
	mu_product_t    f64
	mu_product_t1   f64
	mu_t            f64
	mu_next         f64
	momentum_decay  f64
	step            int
	lr              f64
	epsilon         f64
	weight_decay    f64
	decoupled_decay bool
}

NAdamStepParams holds one CPU NAdam update's scalar state.

struct OneCycleLR[T] {
	max_lr           f64
	total_steps      int
	pct_start        f64
	div_factor       f64
	final_div_factor f64
	three_phase      bool
	anneal_strategy  OneCycleAnnealStrategy
	cycle_momentum   bool
	base_momentum    f64
	max_momentum     f64
}

OneCycleLR raises the rate from max_lr/div_factor to max_lr, then anneals it to max_lr/(div_factor*final_div_factor). Steps are minibatch steps, as in PyTorch's OneCycleLR. The optional three-phase policy returns to the initial rate before the final annealing phase.

struct OneCycleLRConfig {
pub:
	max_lr           f64
	total_steps      int
	pct_start        f64 = 0.3
	div_factor       f64 = 25.0
	final_div_factor f64 = 10000.0
	three_phase      bool
	anneal_strategy  OneCycleAnnealStrategy = .cosine
	cycle_momentum   bool                   = true
	base_momentum    f64                    = 0.85
	max_momentum     f64                    = 0.95
}

struct PolynomialLR[T] {
	total_iters int
	power       f64
}

PolynomialLR decays the base learning rate to zero over total_iters steps.

struct PolynomialLRConfig {
pub:
	total_iters int = 5
	power       f64 = 1.0
}

PolynomialLRConfig defines the duration and power of a polynomial decay.

struct RAdamOptimizer[T] {
	learning_rate          f64
	epsilon                f64
	weight_decay           f64
	decoupled_weight_decay bool
pub mut:
	beta1          f64
	beta2          f64
	beta1_t        f64
	beta2_t        f64
	step           int
	params         []&autograd.Variable[T]
	first_moments  []&vtl.Tensor[T]
	second_moments []&vtl.Tensor[T]
	groups         []&RAdamParamGroup[T]
}

RAdamOptimizer implements Rectified Adam with adaptive variance rectification. Rectification threshold and epsilon placement follow the PyTorch RAdam algorithm.

struct RAdamOptimizerConfig {
pub:
	learning_rate          f64 = 0.001
	beta1                  f64 = 0.9
	beta2                  f64 = 0.999
	epsilon                f64 = 1e-8
	weight_decay           f64
	decoupled_weight_decay bool
}

RAdamOptimizerConfig configures RAdamOptimizer.

struct RAdamParamGroup[T] {
pub mut:
	params                 []&autograd.Variable[T]
	learning_rate          f64
	beta1                  f64
	beta2                  f64
	epsilon                f64
	weight_decay           f64
	decoupled_weight_decay bool
mut:
	beta1_t        f64
	beta2_t        f64
	step           int
	first_moments  []&vtl.Tensor[T]
	second_moments []&vtl.Tensor[T]
}

RAdamParamGroup is a set of parameters sharing one RAdam configuration, mirroring PyTorch parameter groups, with its own step counter and moment buffers.

struct RAdamStepParams {
pub:
	beta1           f64
	beta2           f64
	beta1_t         f64
	beta2_t         f64
	step            int
	lr              f64
	epsilon         f64
	weight_decay    f64
	decoupled_decay bool
}

RAdamStepParams holds one CPU RAdam update's scalar state.

struct RMSPropOptimizer[T] {
	learning_rate f64
	epsilon       f64
pub mut:
	alpha        f64 // smoothing constant
	weight_decay f64
	params       []&autograd.Variable[T]
	sq_avg       []&vtl.Tensor[T]
	groups       []&RMSPropParamGroup[T]
}

RMSPropOptimizer implements the RMSProp optimiser.

Maintains a running average of the squared gradient per parameter and normalises the update by it, allowing different effective learning rates per parameter.

Update rule: sq_avg = α·sq_avg + (1-α)·g² θ = θ - lr · (g / (√sq_avg + ε) + wd·θ)

Reference: Hinton, "Neural Networks for Machine Learning", Lecture 6e.

struct RMSPropOptimizerConfig {
pub:
	learning_rate f64 = 0.001
	alpha         f64 = 0.99
	epsilon       f64 = 1e-8
	weight_decay  f64 = 0.0
}

RMSPropOptimizerConfig defines a public data structure for this module.

struct RMSPropParamGroup[T] {
pub mut:
	params        []&autograd.Variable[T]
	learning_rate f64
	alpha         f64
	epsilon       f64
	weight_decay  f64
mut:
	sq_avg []&vtl.Tensor[T]
}

RMSPropParamGroup is a set of parameters sharing one RMSProp configuration, mirroring PyTorch parameter groups, with its own squared gradient averages.

struct ReduceLROnPlateau[T] {
	factor         f64
	patience       int
	threshold      f64
	epsilon        f64
	cooldown       int
	mode           ReduceLROnPlateauMode
	threshold_mode ReduceLROnPlateauThresholdMode
	min_lr         f64
mut:
	cooldown_remaining      int
	step_cooldown_remaining int
	step_bad_epochs         int
	step_best               f64
pub mut:
	wait       int
	current_lr f64
}

struct ReduceLROnPlateauConfig {
pub:
	factor         f64 = 0.1
	patience       int = 10
	threshold      f64 = 1e-4
	threshold_mode ReduceLROnPlateauThresholdMode = .rel
	mode           ReduceLROnPlateauMode          = .min
	epsilon        f64 = 1e-8
	cooldown       int
	min_lr         f64
}

ReduceLROnPlateauConfig defines a public data structure for this module.

struct SgdOptimizer[T] {
	learning_rate f64
	momentum      f64
	dampening     f64
	weight_decay  f64
	nesterov      bool
pub mut:
	params               []&autograd.Variable[T]
	momentum_buffers     []&vtl.Tensor[T]
	momentum_initialized []bool
	groups               []&SgdParamGroup[T]
}

SgdOptimizer implements Stochastic Gradient Descent with optional momentum and weight decay.

struct SgdOptimizerConfig {
pub:
	learning_rate f64 = 0.001
	momentum      f64
	dampening     f64
	weight_decay  f64
	nesterov      bool
}

SgdOptimizerConfig defines a public data structure for this module.

struct SgdParamGroup[T] {
pub mut:
	params        []&autograd.Variable[T]
	learning_rate f64
	momentum      f64
	dampening     f64
	weight_decay  f64
	nesterov      bool
mut:
	momentum_buffers     []&vtl.Tensor[T]
	momentum_initialized []bool
}

SgdParamGroup is a set of parameters sharing one SGD configuration, mirroring PyTorch parameter groups. Groups are updated with their own learning rate, momentum, dampening, weight decay and Nesterov settings, independently of the optimizer-wide defaults used by the flat params list.

struct StepLR[T] {
	step_size int
	gamma     f64
}

StepLR decays the learning rate by gamma every step_size steps.