Skip to content

preprocessing #

VSL Preprocessing Module

The vsl.preprocessing module provides utilities for data preprocessing commonly used in machine learning pipelines.

Numeric transformers implement a shared fit/transform contract and can be composed with preprocessing.Pipeline. Fit the pipeline on training data only, then reuse its learned state for validation, test, and inference data.

Features

Scalers

  • StandardScaler: Standardizes features by removing the mean and scaling to unit variance (Z-score normalization)
  • MinMaxScaler: Transforms features by scaling to a given range (default [0, 1])
  • RobustScaler: Scales features using statistics robust to outliers (median and IQR)

Imputation

  • SimpleImputer: Learns per-feature mean, median, or constant replacements for NaN values from training data and reuses them during inference

Encoders

  • LabelEncoder: Encodes categorical labels as integers
  • OneHotEncoder: Encodes categorical features as one-hot numeric arrays
  • OrdinalEncoder: Encodes categorical features as ordinal integers

Binning

  • cut: Bins values into discrete intervals using equal-width binning
  • qcut: Bins values using quantile-based (equal-frequency) binning
  • Binner: Fitted binning transformer for consistent bin assignment

Polynomial Features

  • PolynomialFeatures: Generates polynomial powers and feature interactions

Numeric Pipelines

  • MatrixTransformer: Common interface for fitted numeric transformations
  • Pipeline: Fits named transformations in sequence and tracks output feature names

Quick Start

Standard Scaling (Z-Score Normalization)

import vsl.preprocessing

data := [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]

mut scaler := preprocessing.StandardScaler.new()
scaled := scaler.fit_transform(data)!

// Inverse transform to recover original values
recovered := scaler.inverse_transform(scaled)!

Min-Max Scaling

import vsl.preprocessing

data := [[0.0], [50.0], [100.0]]

// Scale to [0, 1] range
mut scaler := preprocessing.MinMaxScaler.new(0.0, 1.0)
scaled := scaler.fit_transform(data)!
// Result: [[0.0], [0.5], [1.0]]

// Or scale to custom range [-1, 1]
mut scaler2 := preprocessing.MinMaxScaler.new(-1.0, 1.0)
scaled2 := scaler2.fit_transform(data)!
// Result: [[-1.0], [0.0], [1.0]]

Robust Scaling

import vsl.preprocessing

// Data with outliers
data := [[1.0], [2.0], [3.0], [4.0], [5.0], [100.0]]

mut scaler := preprocessing.RobustScaler.new()
scaled := scaler.fit_transform(data)!
// Uses median and IQR, so outlier has less effect

Missing Values

SimpleImputer treats NaN as missing and learns its statistics only from the matrix passed to fit or fit_transform. Mean and median strategies return an error for a feature containing no observed values; constant imputation can fill such a feature.

import math
import vsl.preprocessing

missing := math.nan()
training := [[1.0, missing], [3.0, 4.0]]
mut imputer := preprocessing.SimpleImputer.new(.mean, 0.0)
_ := imputer.fit_transform(training)!

inference := imputer.transform([[missing, 10.0]])!
// The first feature uses the training mean (2), not an inference statistic.
assert inference == [[2.0, 10.0]]

Label Encoding

import vsl.preprocessing

labels := ['cat', 'dog', 'bird', 'cat', 'dog']

mut encoder := preprocessing.LabelEncoder.new()
encoded := encoder.fit_transform(labels)!
// Result: [0, 1, 2, 0, 1] (order depends on first occurrence)

// Inverse transform
decoded := encoder.inverse_transform(encoded)!
// Result: ['cat', 'dog', 'bird', 'cat', 'dog']

One-Hot Encoding

import vsl.preprocessing

data := [['red'], ['blue'], ['green'], ['red']]

mut encoder := preprocessing.OneHotEncoder.new(false)
encoded := encoder.fit_transform(data)!
// Result: [[1,0,0], [0,1,0], [0,0,1], [1,0,0]]

// Get feature names
names := encoder.get_feature_names(['color'])!
// Result: ['color_red', 'color_blue', 'color_green']

// Dummy encoding (drop first category)
mut dummy_encoder := preprocessing.OneHotEncoder.new(true)
dummy := dummy_encoder.fit_transform(data)!
// Result: [[0,0], [1,0], [0,1], [0,0]]

Binning

import vsl.preprocessing

values := [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0]

// Equal-width binning
result := preprocessing.cut(values, 2, ['low', 'high'])!

// Quantile binning (equal-frequency)
result2 := preprocessing.qcut(values, 4, ['Q1', 'Q2', 'Q3', 'Q4'])!

// Using Binner for consistent transformation
mut binner := preprocessing.Binner.new(3, .uniform, ['small', 'medium', 'large'])
binner.fit(values)!
binned := binner.transform(values)!

Polynomial Features

import vsl.preprocessing

data := [[2.0, 3.0], [4.0, 5.0]]
features := preprocessing.PolynomialFeatures{
    degree: 2
}
expanded := features.transform(data)!
// Columns: [bias, x1, x2, x1^2, x1*x2, x2^2]
// First row: [1, 2, 3, 4, 6, 9]

Set interaction_only: true to exclude repeated input features in a term, or include_bias: false to omit the leading column of ones.

Leakage-safe numeric pipeline

import math
import vsl.preprocessing

missing := math.nan()
training := [[1.0, missing], [3.0, 4.0]]
mut pipeline := preprocessing.Pipeline.new([
    preprocessing.PipelineStep{
        name:        'impute'
        transformer: preprocessing.SimpleImputer.new(.mean, 0.0)
    },
    preprocessing.PipelineStep{
        name:        'scale'
        transformer: preprocessing.StandardScaler.new()
    },
])!

scaled_training := pipeline.fit_transform(training, ['age', 'income'])!
inference := pipeline.transform([[missing, 10.0]])!
names := pipeline.get_feature_names_out()!

An empty feature-name slice asks the pipeline to generate x0, x1, etc. Transformers that change feature count, such as PolynomialFeatures, update the tracked names in their output order.

API Reference

StandardScaler

Method Description
new() Creates a new StandardScaler
fit(x) Computes mean and std from data
transform(x) Applies standardization
fit_transform(x) Fits and transforms in one step
inverse_transform(x) Reverses standardization

Attributes after fitting:

  • mean_: Mean of each feature
  • std_: Standard deviation of each feature

MinMaxScaler

Method Description
new(feature_min, feature_max) Creates scaler with desired range
fit(x) Computes min and max from data
transform(x) Applies min-max scaling
fit_transform(x) Fits and transforms in one step
inverse_transform(x) Reverses scaling

RobustScaler

Method Description
new() Creates a new RobustScaler
fit(x) Computes median and IQR from data
transform(x) Applies robust scaling
fit_transform(x) Fits and transforms in one step

SimpleImputer

Method Description
new(strategy, fill_value) Creates a NaN imputer
fit(x) Learns one replacement value per feature
transform(x) Replaces NaNs using learned training statistics
fit_transform(x) Fits and transforms in one step

ImputationStrategy:

  • .mean: Mean of observed values per feature
  • .median: Median of observed values per feature
  • .constant: The configured fill_value

fill_value is ignored for mean and median strategies. Matrices must be rectangular and contain at least one feature. Transformations must preserve the fitted feature count. Empty inference batches are accepted. Mean and median fitting fails if any feature contains only NaNs; constant imputation supports all-NaN features. Only NaN is considered missing. Fitted state is held in memory; serialization remains outside this pipeline slice.

LabelEncoder

Method Description
new() Creates a new LabelEncoder
fit(y) Learns unique classes
transform(y) Encodes labels to integers
fit_transform(y) Fits and transforms in one step
inverse_transform(y) Converts integers back to labels

OneHotEncoder

Method Description
new(drop_first) Creates encoder; drop_first for dummy encoding
fit(x) Learns categories for each feature
transform(x) Applies one-hot encoding
fit_transform(x) Fits and transforms in one step
get_feature_names(input_names) Returns output feature names

Binner

Method Description
new(n_bins, strategy, labels) Creates binner with specified strategy
fit(values) Computes bin edges
transform(values) Assigns values to bins
fit_transform(values) Fits and transforms in one step
transform_to_indices(values) Returns bin indices instead of labels

BinningStrategy:

  • .uniform: Equal-width bins
  • .quantile: Equal-frequency bins

PolynomialFeatures

Field Description
degree Maximum polynomial degree (default 2)
interaction_only Use each input feature at most once in each term
include_bias Add a leading column of ones (default true)
transform(data) Expands a rectangular matrix into polynomial features
get_feature_names_out(input_features) Returns names in transform output order

Numeric Pipeline

API Description
Pipeline.new(steps) Creates a pipeline from named PipelineStep values
PipelineStep{name, transformer} Associates a unique name with a MatrixTransformer
fit(data, feature_names) Fits steps in sequence; [] generates x0, x1, etc.
fit_transform(data, feature_names) Fits the steps and returns training features
transform(data) Reuses fitted state and validates input/output widths
get_feature_names_out() Returns the output names learned during fitting

MatrixTransformer requires fit, transform, and get_feature_names_out. StandardScaler, MinMaxScaler, RobustScaler, SimpleImputer, and PolynomialFeatures implement it. String encoders and serialization are not yet integrated with this numeric pipeline.

Integration with ML Pipeline

import vsl.preprocessing
import vsl.inout.csv
import vsl.ml

// Load data
csv_data := csv.read_csv('data.csv', csv.CsvReadConfig{ skip_header: true })!

// Scale features
mut scaler := preprocessing.StandardScaler.new()
scaled_data := scaler.fit_transform(csv_data.data)!

// Create ML Data object
mut data := ml.Data.from_raw_xy(scaled_data)!

// Train model
mut model := ml.LinReg.new(mut data, 'my_model')
model.train()

// For new predictions, use the same scaler
new_data := [[1.0, 2.0, 3.0]]
scaled_new := scaler.transform(new_data)!
prediction := model.predict(scaled_new[0])

See Also

fn cut #

fn cut(values []f64, n_bins int, labels []string) ![]string

cut bins values into discrete intervals using equal-width binning. Returns the bin labels for each value.

fn digitize #

fn digitize(values []f64, bins []f64) []int

digitize returns indices of the bins to which each value belongs. Similar to numpy.digitize

fn qcut #

fn qcut(values []f64, n_bins int, labels []string) ![]string

qcut bins values into discrete intervals using quantile-based binning. Returns the bin labels for each value.

fn Binner.new #

fn Binner.new(n_bins int, strategy BinningStrategy, labels []string) &Binner

Binner.new creates a new Binner

fn BinningStrategy.from #

fn BinningStrategy.from[W](input W) !BinningStrategy

from returns the BinningStrategy value of an integer or of a field name, or an error if there is none.

fn ImputationStrategy.from #

fn ImputationStrategy.from[W](input W) !ImputationStrategy

from returns the ImputationStrategy value of an integer or of a field name, or an error if there is none.

fn LabelEncoder.new #

fn LabelEncoder.new() &LabelEncoder

LabelEncoder.new creates a new LabelEncoder

fn MinMaxScaler.new #

fn MinMaxScaler.new(feature_min f64, feature_max f64) &MinMaxScaler

MinMaxScaler.new creates a new MinMaxScaler with optional range

fn OneHotEncoder.new #

fn OneHotEncoder.new(drop_first bool) &OneHotEncoder

OneHotEncoder.new creates a new OneHotEncoder

fn OrdinalEncoder.new #

fn OrdinalEncoder.new() &OrdinalEncoder

OrdinalEncoder.new creates a new OrdinalEncoder

fn Pipeline.new #

fn Pipeline.new(steps []PipelineStep) !&Pipeline

Pipeline.new creates a pipeline from named numeric transformers.

fn RobustScaler.new #

fn RobustScaler.new() &RobustScaler

RobustScaler.new creates a new RobustScaler

fn SimpleImputer.new #

fn SimpleImputer.new(strategy ImputationStrategy, fill_value f64) &SimpleImputer

SimpleImputer.new creates an imputer that treats NaN values as missing.

fn StandardScaler.new #

fn StandardScaler.new() &StandardScaler

StandardScaler.new creates a new StandardScaler

interface MatrixTransformer #

interface MatrixTransformer {
	// transform applies the learned transformation without mutating its state.
	transform(data [][]f64) ![][]f64

	// get_feature_names_out returns names for the transformed columns.
	get_feature_names_out(input_features []string) ![]string
	// fit learns transform state from the supplied training matrix.
mut:
	fit(data [][]f64) !
}

MatrixTransformer is a fitted transform for numeric feature matrices.

enum BinningStrategy #

enum BinningStrategy {
	uniform  // equal-width bins
	quantile // equal-frequency bins
}

BinningStrategy specifies how to create bin edges

enum ImputationStrategy #

enum ImputationStrategy {
	mean
	median
	constant
}

ImputationStrategy selects how a feature's missing values are filled.

struct Binner #

@[heap]
struct Binner {
mut:
	fitted bool
pub mut:
	edges_   []f64    // bin edges
	labels_  []string // bin labels
	n_bins   int
	strategy BinningStrategy
}

Binner defines a public data structure for this module.

fn (Binner) fit #

fn (mut b Binner) fit(values []f64) !

fit computes the bin edges based on the data

fn (Binner) transform #

fn (b &Binner) transform(values []f64) ![]string

transform applies the binning to the data

fn (Binner) fit_transform #

fn (mut b Binner) fit_transform(values []f64) ![]string

fit_transform fits and transforms in one step

fn (Binner) transform_to_indices #

fn (b &Binner) transform_to_indices(values []f64) ![]int

transform_to_indices returns bin indices instead of labels

struct LabelEncoder #

@[heap]
struct LabelEncoder {
mut:
	fitted bool
pub mut:
	classes_     []string       // unique classes in order
	class_to_idx map[string]int // mapping from class to index
}

LabelEncoder defines a public data structure for this module.

fn (LabelEncoder) fit #

fn (mut e LabelEncoder) fit(y []string) !

fit learns the unique classes from the data

fn (LabelEncoder) transform #

fn (e &LabelEncoder) transform(y []string) ![]int

transform encodes labels to integers

fn (LabelEncoder) fit_transform #

fn (mut e LabelEncoder) fit_transform(y []string) ![]int

fit_transform fits and transforms in one step

fn (LabelEncoder) inverse_transform #

fn (e &LabelEncoder) inverse_transform(y []int) ![]string

inverse_transform converts integer labels back to original labels

struct MinMaxScaler #

@[heap]
struct MinMaxScaler {
mut:
	fitted bool
pub mut:
	min_        []f64 // minimum of each feature
	max_        []f64 // maximum of each feature
	data_range_ []f64 // max - min for each feature
	n_features  int
	feature_min f64 = 0.0 // desired minimum of transformed feature
	feature_max f64 = 1.0 // desired maximum of transformed feature
}

MinMaxScaler defines a public data structure for this module.

fn (MinMaxScaler) fit #

fn (mut s MinMaxScaler) fit(x [][]f64) !

fit computes the min and max to be used for later scaling

fn (MinMaxScaler) transform #

fn (s &MinMaxScaler) transform(x [][]f64) ![][]f64

transform applies the min-max scaling to the data

fn (MinMaxScaler) fit_transform #

fn (mut s MinMaxScaler) fit_transform(x [][]f64) ![][]f64

fit_transform fits and transforms in one step

fn (MinMaxScaler) inverse_transform #

fn (s &MinMaxScaler) inverse_transform(x [][]f64) ![][]f64

inverse_transform reverses the min-max scaling

fn (MinMaxScaler) get_feature_names_out #

fn (s &MinMaxScaler) get_feature_names_out(input_features []string) ![]string

get_feature_names_out preserves names because scaling does not change columns.

struct OneHotEncoder #

@[heap]
struct OneHotEncoder {
mut:
	fitted bool
pub mut:
	categories_ [][]string // categories for each feature
	n_features  int
	drop_first  bool // whether to drop the first category (for dummy encoding)
}

OneHotEncoder defines a public data structure for this module.

fn (OneHotEncoder) fit #

fn (mut e OneHotEncoder) fit(x [][]string) !

fit learns the categories for each feature

fn (OneHotEncoder) transform #

fn (e &OneHotEncoder) transform(x [][]string) ![][]f64

transform applies one-hot encoding to the data

fn (OneHotEncoder) fit_transform #

fn (mut e OneHotEncoder) fit_transform(x [][]string) ![][]f64

fit_transform fits and transforms in one step

fn (OneHotEncoder) get_feature_names #

fn (e &OneHotEncoder) get_feature_names(input_names []string) ![]string

get_feature_names returns the names of the output features

struct OrdinalEncoder #

@[heap]
struct OrdinalEncoder {
mut:
	fitted bool
pub mut:
	categories_ [][]string // categories for each feature
	n_features  int
}

OrdinalEncoder defines a public data structure for this module.

fn (OrdinalEncoder) fit #

fn (mut e OrdinalEncoder) fit(x [][]string) !

fit learns the categories for each feature

fn (OrdinalEncoder) transform #

fn (e &OrdinalEncoder) transform(x [][]string) ![][]f64

transform applies ordinal encoding to the data

fn (OrdinalEncoder) fit_transform #

fn (mut e OrdinalEncoder) fit_transform(x [][]string) ![][]f64

fit_transform fits and transforms in one step

struct Pipeline #

@[heap]
struct Pipeline {
mut:
	steps                  []PipelineStep
	n_features_out_by_step []int
	fitted                 bool
pub mut:
	n_features_in_     int
	n_features_out_    int
	feature_names_in_  []string
	feature_names_out_ []string
}

Pipeline fits and applies numeric transformers in sequence.

fn (Pipeline) fit #

fn (mut p Pipeline) fit(data [][]f64, feature_names []string) !

fit learns each transform in sequence. An empty feature-name slice generates x0, x1, ...

fn (Pipeline) fit_transform #

fn (mut p Pipeline) fit_transform(data [][]f64, feature_names []string) ![][]f64

fit_transform fits each step on its input and returns the transformed data.

fn (Pipeline) transform #

fn (p &Pipeline) transform(data [][]f64) ![][]f64

transform applies the fitted steps without changing their learned state.

fn (Pipeline) get_feature_names_out #

fn (p &Pipeline) get_feature_names_out() ![]string

get_feature_names_out returns the output feature names learned during fit.

struct PipelineStep #

struct PipelineStep {
pub:
	name        string
	transformer MatrixTransformer
}

PipelineStep names one transformer in a numeric preprocessing pipeline.

struct PolynomialFeatures #

@[params]
struct PolynomialFeatures {
pub:
	degree           int = 2
	interaction_only bool
	include_bias     bool = true
}

PolynomialFeatures generates all polynomial combinations up to degree. Interaction-only mode uses each input feature at most once per term.

fn (PolynomialFeatures) fit #

fn (p PolynomialFeatures) fit(data [][]f64) !

fit validates the feature matrix before the transformer is used in a pipeline.

fn (PolynomialFeatures) transform #

fn (p PolynomialFeatures) transform(data [][]f64) ![][]f64

transform expands each row into polynomial features. Output columns are ordered by degree, then lexicographically by input feature index.

fn (PolynomialFeatures) get_feature_names_out #

fn (p PolynomialFeatures) get_feature_names_out(input_features []string) ![]string

get_feature_names_out returns names in the same order as transform.

struct RobustScaler #

@[heap]
struct RobustScaler {
mut:
	fitted bool
pub mut:
	median_    []f64 // median of each feature
	iqr_       []f64 // interquartile range of each feature
	n_features int
}

RobustScaler defines a public data structure for this module.

fn (RobustScaler) fit #

fn (mut s RobustScaler) fit(x [][]f64) !

fit computes the median and IQR to be used for later scaling

fn (RobustScaler) transform #

fn (s &RobustScaler) transform(x [][]f64) ![][]f64

transform applies the robust scaling to the data

fn (RobustScaler) fit_transform #

fn (mut s RobustScaler) fit_transform(x [][]f64) ![][]f64

fit_transform fits and transforms in one step

fn (RobustScaler) get_feature_names_out #

fn (s &RobustScaler) get_feature_names_out(input_features []string) ![]string

get_feature_names_out preserves names because scaling does not change columns.

struct SimpleImputer #

@[heap]
struct SimpleImputer {
mut:
	fitted bool
pub:
	strategy   ImputationStrategy
	fill_value f64
pub mut:
	statistics_ []f64
	n_features  int
}

SimpleImputer learns one replacement value per feature from training data.

fn (SimpleImputer) fit #

fn (mut imputer SimpleImputer) fit(data [][]f64) !

fit learns replacement values from the provided training matrix.

fn (SimpleImputer) transform #

fn (imputer &SimpleImputer) transform(data [][]f64) ![][]f64

transform replaces NaN values using statistics learned by fit.

fn (SimpleImputer) fit_transform #

fn (mut imputer SimpleImputer) fit_transform(data [][]f64) ![][]f64

fit_transform learns replacement values and applies them to the same matrix.

fn (SimpleImputer) get_feature_names_out #

fn (imputer &SimpleImputer) get_feature_names_out(input_features []string) ![]string

get_feature_names_out preserves names because imputation does not change columns.

struct StandardScaler #

@[heap]
struct StandardScaler {
mut:
	fitted bool
pub mut:
	mean_      []f64 // mean of each feature
	std_       []f64 // standard deviation of each feature
	n_features int
}

StandardScaler defines a public data structure for this module.

fn (StandardScaler) fit #

fn (mut s StandardScaler) fit(x [][]f64) !

fit computes the mean and std to be used for later scaling

fn (StandardScaler) transform #

fn (s &StandardScaler) transform(x [][]f64) ![][]f64

transform applies the standardization to the data

fn (StandardScaler) fit_transform #

fn (mut s StandardScaler) fit_transform(x [][]f64) ![][]f64

fit_transform fits and transforms in one step

fn (StandardScaler) inverse_transform #

fn (s &StandardScaler) inverse_transform(x [][]f64) ![][]f64

inverse_transform reverses the standardization

fn (StandardScaler) get_feature_names_out #

fn (s &StandardScaler) get_feature_names_out(input_features []string) ![]string

get_feature_names_out preserves names because scaling does not change columns.