INT8 quantization

vtl.quantization provides opt-in affine signed INT8 conversion for f32/f64 tensors. Quantization parameters use

q = clamp(round_to_even(x / scale + zero_point), qmin, qmax)
x = (q - zero_point) * scale

The initial API includes per-tensor and per-channel parameters, finite-value min/max calibration observers, and explicit f32/f64 quantize/dequantize entry points. Quantized results are ordinary Tensor[i8] values. This is a storage and calibration foundation; it does not yet provide integer Linear/Conv kernels, layer conversion, accelerator dispatch, or a performance claim. Those remain tracked by issue #792.

See the runnable INT8 quantization example.

Per-channel conversion accepts an axis and one scale/zero point per channel. Negative axes are normalized against the tensor rank. Conversion traverses logical tensor order, including strided views, and returns a row-major tensor with the same shape. Non-finite calibration or input values are errors rather than silently saturating NaNs or infinities.

Validation

Run from ~/.vmodules with a local memory cap:

VJOBS=2 systemd-run --user --scope -p MemoryMax=4G -p MemorySwapMax=0 -- v test ./vtl/quantization

fn affine_params(minimum f64, maximum f64, qmin i8, qmax i8) !QuantizationParams

affine_params derives asymmetric affine parameters from a finite value range.

fn dequantize_per_channel_f32(input &vtl.Tensor[i8], params PerChannelParams) !&vtl.Tensor[f32]

dequantize_per_channel_f32 converts signed INT8 values to f32 values.

fn dequantize_per_channel_f64(input &vtl.Tensor[i8], params PerChannelParams) !&vtl.Tensor[f64]

dequantize_per_channel_f64 converts signed INT8 values to f64 values.

fn dequantize_per_tensor_f32(input &vtl.Tensor[i8], params QuantizationParams) !&vtl.Tensor[f32]

dequantize_per_tensor_f32 converts signed INT8 values to f32 values.

fn dequantize_per_tensor_f64(input &vtl.Tensor[i8], params QuantizationParams) !&vtl.Tensor[f64]

dequantize_per_tensor_f64 converts signed INT8 values to f64 values.

fn new_params(scale f64, zero_point int, qmin i8, qmax i8) !QuantizationParams

new_params validates and creates signed INT8 affine quantization parameters.

fn new_per_channel_min_max_observer(axis int) PerChannelMinMaxObserver

new_per_channel_min_max_observer creates a per-channel calibration observer.

fn new_per_channel_params(scales []f64, zero_points []int, axis int, qmin i8, qmax i8) !PerChannelParams

new_per_channel_params validates per-channel INT8 affine parameters.

fn quantize_per_channel_f32(input &vtl.Tensor[f32], params PerChannelParams) !&vtl.Tensor[i8]

quantize_per_channel_f32 applies one scale and zero point per selected channel.

fn quantize_per_channel_f64(input &vtl.Tensor[f64], params PerChannelParams) !&vtl.Tensor[i8]

quantize_per_channel_f64 applies one scale and zero point per selected channel.

fn quantize_per_tensor_f32(input &vtl.Tensor[f32], params QuantizationParams) !&vtl.Tensor[i8]

quantize_per_tensor_f32 maps finite f32 values into signed INT8 values.

fn quantize_per_tensor_f64(input &vtl.Tensor[f64], params QuantizationParams) !&vtl.Tensor[i8]

quantize_per_tensor_f64 maps finite f64 values into signed INT8 values.

struct MinMaxObserver {
pub mut:
	minimum     f64
	maximum     f64
	initialized bool
}

MinMaxObserver accumulates finite scalar range statistics for calibration.

fn (mut observer MinMaxObserver) observe_f32(input &vtl.Tensor[f32]) !

observe_f32 adds finite f32 tensor values to the calibration range.

fn (mut observer MinMaxObserver) observe_f64(input &vtl.Tensor[f64]) !

observe_f64 adds finite f64 tensor values to the calibration range.

fn (observer &MinMaxObserver) params(qmin i8, qmax i8) !QuantizationParams

params derives affine quantization parameters from observed values.

struct PerChannelMinMaxObserver {
pub:
	axis int
pub mut:
	minimums    []f64
	maximums    []f64
	initialized []bool
}

PerChannelMinMaxObserver accumulates independent ranges along one tensor axis.

fn (mut observer PerChannelMinMaxObserver) observe_f32(input &vtl.Tensor[f32]) !

observe_f32 adds finite f32 values to matching channel ranges.

fn (mut observer PerChannelMinMaxObserver) observe_f64(input &vtl.Tensor[f64]) !

observe_f64 adds finite f64 values to matching channel ranges.

fn (observer &PerChannelMinMaxObserver) params(qmin i8, qmax i8) !PerChannelParams

params derives one affine parameter set per observed channel.

struct PerChannelParams {
pub:
	scales      []f64
	zero_points []int
	axis        int
	qmin        i8 = -128
	qmax        i8 = 127
}

PerChannelParams stores one affine scale and zero point for each channel.

struct QuantizationParams {
pub:
	scale      f64
	zero_point int
	qmin       i8 = -128
	qmax       i8 = 127
}

QuantizationParams describes affine signed INT8 quantization.