preprocessing #
VSL Preprocessing Module
The vsl.preprocessing module provides utilities for data preprocessing commonly used in machine learning pipelines.
Numeric transformers implement a shared fit/transform contract and can be composed with preprocessing.Pipeline. Fit the pipeline on training data only, then reuse its learned state for validation, test, and inference data.
Features
Scalers
- StandardScaler: Standardizes features by removing the mean and scaling to unit variance (Z-score normalization)
- MinMaxScaler: Transforms features by scaling to a given range (default [0, 1])
- RobustScaler: Scales features using statistics robust to outliers (median and IQR)
Imputation
- SimpleImputer: Learns per-feature mean, median, or constant replacements for NaN values from training data and reuses them during inference
Encoders
- LabelEncoder: Encodes categorical labels as integers
- OneHotEncoder: Encodes categorical features as one-hot numeric arrays
- OrdinalEncoder: Encodes categorical features as ordinal integers
Binning
- cut: Bins values into discrete intervals using equal-width binning
- qcut: Bins values using quantile-based (equal-frequency) binning
- Binner: Fitted binning transformer for consistent bin assignment
Polynomial Features
- PolynomialFeatures: Generates polynomial powers and feature interactions
Numeric Pipelines
- MatrixTransformer: Common interface for fitted numeric transformations
- Pipeline: Fits named transformations in sequence and tracks output feature names
Quick Start
Standard Scaling (Z-Score Normalization)
import vsl.preprocessing
data := [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]
mut scaler := preprocessing.StandardScaler.new()
scaled := scaler.fit_transform(data)!
// Inverse transform to recover original values
recovered := scaler.inverse_transform(scaled)!
Min-Max Scaling
import vsl.preprocessing
data := [[0.0], [50.0], [100.0]]
// Scale to [0, 1] range
mut scaler := preprocessing.MinMaxScaler.new(0.0, 1.0)
scaled := scaler.fit_transform(data)!
// Result: [[0.0], [0.5], [1.0]]
// Or scale to custom range [-1, 1]
mut scaler2 := preprocessing.MinMaxScaler.new(-1.0, 1.0)
scaled2 := scaler2.fit_transform(data)!
// Result: [[-1.0], [0.0], [1.0]]
Robust Scaling
import vsl.preprocessing
// Data with outliers
data := [[1.0], [2.0], [3.0], [4.0], [5.0], [100.0]]
mut scaler := preprocessing.RobustScaler.new()
scaled := scaler.fit_transform(data)!
// Uses median and IQR, so outlier has less effect
Missing Values
SimpleImputer treats NaN as missing and learns its statistics only from the matrix passed to fit or fit_transform. Mean and median strategies return an error for a feature containing no observed values; constant imputation can fill such a feature.
import math
import vsl.preprocessing
missing := math.nan()
training := [[1.0, missing], [3.0, 4.0]]
mut imputer := preprocessing.SimpleImputer.new(.mean, 0.0)
_ := imputer.fit_transform(training)!
inference := imputer.transform([[missing, 10.0]])!
// The first feature uses the training mean (2), not an inference statistic.
assert inference == [[2.0, 10.0]]
Label Encoding
import vsl.preprocessing
labels := ['cat', 'dog', 'bird', 'cat', 'dog']
mut encoder := preprocessing.LabelEncoder.new()
encoded := encoder.fit_transform(labels)!
// Result: [0, 1, 2, 0, 1] (order depends on first occurrence)
// Inverse transform
decoded := encoder.inverse_transform(encoded)!
// Result: ['cat', 'dog', 'bird', 'cat', 'dog']
One-Hot Encoding
import vsl.preprocessing
data := [['red'], ['blue'], ['green'], ['red']]
mut encoder := preprocessing.OneHotEncoder.new(false)
encoded := encoder.fit_transform(data)!
// Result: [[1,0,0], [0,1,0], [0,0,1], [1,0,0]]
// Get feature names
names := encoder.get_feature_names(['color'])!
// Result: ['color_red', 'color_blue', 'color_green']
// Dummy encoding (drop first category)
mut dummy_encoder := preprocessing.OneHotEncoder.new(true)
dummy := dummy_encoder.fit_transform(data)!
// Result: [[0,0], [1,0], [0,1], [0,0]]
Binning
import vsl.preprocessing
values := [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0]
// Equal-width binning
result := preprocessing.cut(values, 2, ['low', 'high'])!
// Quantile binning (equal-frequency)
result2 := preprocessing.qcut(values, 4, ['Q1', 'Q2', 'Q3', 'Q4'])!
// Using Binner for consistent transformation
mut binner := preprocessing.Binner.new(3, .uniform, ['small', 'medium', 'large'])
binner.fit(values)!
binned := binner.transform(values)!
Polynomial Features
import vsl.preprocessing
data := [[2.0, 3.0], [4.0, 5.0]]
features := preprocessing.PolynomialFeatures{
degree: 2
}
expanded := features.transform(data)!
// Columns: [bias, x1, x2, x1^2, x1*x2, x2^2]
// First row: [1, 2, 3, 4, 6, 9]
Set interaction_only: true to exclude repeated input features in a term, or include_bias: false to omit the leading column of ones.
Leakage-safe numeric pipeline
import math
import vsl.preprocessing
missing := math.nan()
training := [[1.0, missing], [3.0, 4.0]]
mut pipeline := preprocessing.Pipeline.new([
preprocessing.PipelineStep{
name: 'impute'
transformer: preprocessing.SimpleImputer.new(.mean, 0.0)
},
preprocessing.PipelineStep{
name: 'scale'
transformer: preprocessing.StandardScaler.new()
},
])!
scaled_training := pipeline.fit_transform(training, ['age', 'income'])!
inference := pipeline.transform([[missing, 10.0]])!
names := pipeline.get_feature_names_out()!
An empty feature-name slice asks the pipeline to generate x0, x1, etc. Transformers that change feature count, such as PolynomialFeatures, update the tracked names in their output order.
API Reference
StandardScaler
| Method | Description |
|---|---|
new() |
Creates a new StandardScaler |
fit(x) |
Computes mean and std from data |
transform(x) |
Applies standardization |
fit_transform(x) |
Fits and transforms in one step |
inverse_transform(x) |
Reverses standardization |
Attributes after fitting:
mean_: Mean of each featurestd_: Standard deviation of each feature
MinMaxScaler
| Method | Description |
|---|---|
new(feature_min, feature_max) |
Creates scaler with desired range |
fit(x) |
Computes min and max from data |
transform(x) |
Applies min-max scaling |
fit_transform(x) |
Fits and transforms in one step |
inverse_transform(x) |
Reverses scaling |
RobustScaler
| Method | Description |
|---|---|
new() |
Creates a new RobustScaler |
fit(x) |
Computes median and IQR from data |
transform(x) |
Applies robust scaling |
fit_transform(x) |
Fits and transforms in one step |
SimpleImputer
| Method | Description |
|---|---|
new(strategy, fill_value) |
Creates a NaN imputer |
fit(x) |
Learns one replacement value per feature |
transform(x) |
Replaces NaNs using learned training statistics |
fit_transform(x) |
Fits and transforms in one step |
ImputationStrategy:
.mean: Mean of observed values per feature.median: Median of observed values per feature.constant: The configuredfill_value
fill_value is ignored for mean and median strategies. Matrices must be rectangular and contain at least one feature. Transformations must preserve the fitted feature count. Empty inference batches are accepted. Mean and median fitting fails if any feature contains only NaNs; constant imputation supports all-NaN features. Only NaN is considered missing. Fitted state is held in memory; serialization remains outside this pipeline slice.
LabelEncoder
| Method | Description |
|---|---|
new() |
Creates a new LabelEncoder |
fit(y) |
Learns unique classes |
transform(y) |
Encodes labels to integers |
fit_transform(y) |
Fits and transforms in one step |
inverse_transform(y) |
Converts integers back to labels |
OneHotEncoder
| Method | Description |
|---|---|
new(drop_first) |
Creates encoder; drop_first for dummy encoding |
fit(x) |
Learns categories for each feature |
transform(x) |
Applies one-hot encoding |
fit_transform(x) |
Fits and transforms in one step |
get_feature_names(input_names) |
Returns output feature names |
Binner
| Method | Description |
|---|---|
new(n_bins, strategy, labels) |
Creates binner with specified strategy |
fit(values) |
Computes bin edges |
transform(values) |
Assigns values to bins |
fit_transform(values) |
Fits and transforms in one step |
transform_to_indices(values) |
Returns bin indices instead of labels |
BinningStrategy:
.uniform: Equal-width bins.quantile: Equal-frequency bins
PolynomialFeatures
| Field | Description |
|---|---|
degree |
Maximum polynomial degree (default 2) |
interaction_only |
Use each input feature at most once in each term |
include_bias |
Add a leading column of ones (default true) |
transform(data) |
Expands a rectangular matrix into polynomial features |
get_feature_names_out(input_features) |
Returns names in transform output order |
Numeric Pipeline
| API | Description |
|---|---|
Pipeline.new(steps) |
Creates a pipeline from named PipelineStep values |
PipelineStep{name, transformer} |
Associates a unique name with a MatrixTransformer |
fit(data, feature_names) |
Fits steps in sequence; [] generates x0, x1, etc. |
fit_transform(data, feature_names) |
Fits the steps and returns training features |
transform(data) |
Reuses fitted state and validates input/output widths |
get_feature_names_out() |
Returns the output names learned during fitting |
MatrixTransformer requires fit, transform, and get_feature_names_out. StandardScaler, MinMaxScaler, RobustScaler, SimpleImputer, and PolynomialFeatures implement it. String encoders and serialization are not yet integrated with this numeric pipeline.
Integration with ML Pipeline
import vsl.preprocessing
import vsl.inout.csv
import vsl.ml
// Load data
csv_data := csv.read_csv('data.csv', csv.CsvReadConfig{ skip_header: true })!
// Scale features
mut scaler := preprocessing.StandardScaler.new()
scaled_data := scaler.fit_transform(csv_data.data)!
// Create ML Data object
mut data := ml.Data.from_raw_xy(scaled_data)!
// Train model
mut model := ml.LinReg.new(mut data, 'my_model')
model.train()
// For new predictions, use the same scaler
new_data := [[1.0, 2.0, 3.0]]
scaled_new := scaler.transform(new_data)!
prediction := model.predict(scaled_new[0])
See Also
fn cut #
fn cut(values []f64, n_bins int, labels []string) ![]string
cut bins values into discrete intervals using equal-width binning. Returns the bin labels for each value.
fn digitize #
fn digitize(values []f64, bins []f64) []int
digitize returns indices of the bins to which each value belongs. Similar to numpy.digitize
fn qcut #
fn qcut(values []f64, n_bins int, labels []string) ![]string
qcut bins values into discrete intervals using quantile-based binning. Returns the bin labels for each value.
fn Binner.new #
fn Binner.new(n_bins int, strategy BinningStrategy, labels []string) &Binner
Binner.new creates a new Binner
fn BinningStrategy.from #
fn BinningStrategy.from[W](input W) !BinningStrategy
from returns the BinningStrategy value of an integer or of a field name, or an error if there is none.
fn ImputationStrategy.from #
fn ImputationStrategy.from[W](input W) !ImputationStrategy
from returns the ImputationStrategy value of an integer or of a field name, or an error if there is none.
fn LabelEncoder.new #
fn LabelEncoder.new() &LabelEncoder
LabelEncoder.new creates a new LabelEncoder
fn MinMaxScaler.new #
fn MinMaxScaler.new(feature_min f64, feature_max f64) &MinMaxScaler
MinMaxScaler.new creates a new MinMaxScaler with optional range
fn OneHotEncoder.new #
fn OneHotEncoder.new(drop_first bool) &OneHotEncoder
OneHotEncoder.new creates a new OneHotEncoder
fn OrdinalEncoder.new #
fn OrdinalEncoder.new() &OrdinalEncoder
OrdinalEncoder.new creates a new OrdinalEncoder
fn Pipeline.new #
fn Pipeline.new(steps []PipelineStep) !&Pipeline
Pipeline.new creates a pipeline from named numeric transformers.
fn RobustScaler.new #
fn RobustScaler.new() &RobustScaler
RobustScaler.new creates a new RobustScaler
fn SimpleImputer.new #
fn SimpleImputer.new(strategy ImputationStrategy, fill_value f64) &SimpleImputer
SimpleImputer.new creates an imputer that treats NaN values as missing.
fn StandardScaler.new #
fn StandardScaler.new() &StandardScaler
StandardScaler.new creates a new StandardScaler
interface MatrixTransformer #
interface MatrixTransformer {
// transform applies the learned transformation without mutating its state.
transform(data [][]f64) ![][]f64
// get_feature_names_out returns names for the transformed columns.
get_feature_names_out(input_features []string) ![]string
// fit learns transform state from the supplied training matrix.
mut:
fit(data [][]f64) !
}
MatrixTransformer is a fitted transform for numeric feature matrices.
enum BinningStrategy #
enum BinningStrategy {
uniform // equal-width bins
quantile // equal-frequency bins
}
BinningStrategy specifies how to create bin edges
enum ImputationStrategy #
enum ImputationStrategy {
mean
median
constant
}
ImputationStrategy selects how a feature's missing values are filled.
struct Binner #
struct Binner {
mut:
fitted bool
pub mut:
edges_ []f64 // bin edges
labels_ []string // bin labels
n_bins int
strategy BinningStrategy
}
Binner defines a public data structure for this module.
fn (Binner) fit #
fn (mut b Binner) fit(values []f64) !
fit computes the bin edges based on the data
fn (Binner) transform #
fn (b &Binner) transform(values []f64) ![]string
transform applies the binning to the data
fn (Binner) fit_transform #
fn (mut b Binner) fit_transform(values []f64) ![]string
fit_transform fits and transforms in one step
fn (Binner) transform_to_indices #
fn (b &Binner) transform_to_indices(values []f64) ![]int
transform_to_indices returns bin indices instead of labels
struct LabelEncoder #
struct LabelEncoder {
mut:
fitted bool
pub mut:
classes_ []string // unique classes in order
class_to_idx map[string]int // mapping from class to index
}
LabelEncoder defines a public data structure for this module.
fn (LabelEncoder) fit #
fn (mut e LabelEncoder) fit(y []string) !
fit learns the unique classes from the data
fn (LabelEncoder) transform #
fn (e &LabelEncoder) transform(y []string) ![]int
transform encodes labels to integers
fn (LabelEncoder) fit_transform #
fn (mut e LabelEncoder) fit_transform(y []string) ![]int
fit_transform fits and transforms in one step
fn (LabelEncoder) inverse_transform #
fn (e &LabelEncoder) inverse_transform(y []int) ![]string
inverse_transform converts integer labels back to original labels
struct MinMaxScaler #
struct MinMaxScaler {
mut:
fitted bool
pub mut:
min_ []f64 // minimum of each feature
max_ []f64 // maximum of each feature
data_range_ []f64 // max - min for each feature
n_features int
feature_min f64 = 0.0 // desired minimum of transformed feature
feature_max f64 = 1.0 // desired maximum of transformed feature
}
MinMaxScaler defines a public data structure for this module.
fn (MinMaxScaler) fit #
fn (mut s MinMaxScaler) fit(x [][]f64) !
fit computes the min and max to be used for later scaling
fn (MinMaxScaler) transform #
fn (s &MinMaxScaler) transform(x [][]f64) ![][]f64
transform applies the min-max scaling to the data
fn (MinMaxScaler) fit_transform #
fn (mut s MinMaxScaler) fit_transform(x [][]f64) ![][]f64
fit_transform fits and transforms in one step
fn (MinMaxScaler) inverse_transform #
fn (s &MinMaxScaler) inverse_transform(x [][]f64) ![][]f64
inverse_transform reverses the min-max scaling
fn (MinMaxScaler) get_feature_names_out #
fn (s &MinMaxScaler) get_feature_names_out(input_features []string) ![]string
get_feature_names_out preserves names because scaling does not change columns.
struct OneHotEncoder #
struct OneHotEncoder {
mut:
fitted bool
pub mut:
categories_ [][]string // categories for each feature
n_features int
drop_first bool // whether to drop the first category (for dummy encoding)
}
OneHotEncoder defines a public data structure for this module.
fn (OneHotEncoder) fit #
fn (mut e OneHotEncoder) fit(x [][]string) !
fit learns the categories for each feature
fn (OneHotEncoder) transform #
fn (e &OneHotEncoder) transform(x [][]string) ![][]f64
transform applies one-hot encoding to the data
fn (OneHotEncoder) fit_transform #
fn (mut e OneHotEncoder) fit_transform(x [][]string) ![][]f64
fit_transform fits and transforms in one step
fn (OneHotEncoder) get_feature_names #
fn (e &OneHotEncoder) get_feature_names(input_names []string) ![]string
get_feature_names returns the names of the output features
struct OrdinalEncoder #
struct OrdinalEncoder {
mut:
fitted bool
pub mut:
categories_ [][]string // categories for each feature
n_features int
}
OrdinalEncoder defines a public data structure for this module.
fn (OrdinalEncoder) fit #
fn (mut e OrdinalEncoder) fit(x [][]string) !
fit learns the categories for each feature
fn (OrdinalEncoder) transform #
fn (e &OrdinalEncoder) transform(x [][]string) ![][]f64
transform applies ordinal encoding to the data
fn (OrdinalEncoder) fit_transform #
fn (mut e OrdinalEncoder) fit_transform(x [][]string) ![][]f64
fit_transform fits and transforms in one step
struct Pipeline #
struct Pipeline {
mut:
steps []PipelineStep
n_features_out_by_step []int
fitted bool
pub mut:
n_features_in_ int
n_features_out_ int
feature_names_in_ []string
feature_names_out_ []string
}
Pipeline fits and applies numeric transformers in sequence.
fn (Pipeline) fit #
fn (mut p Pipeline) fit(data [][]f64, feature_names []string) !
fit learns each transform in sequence. An empty feature-name slice generates x0, x1, ...
fn (Pipeline) fit_transform #
fn (mut p Pipeline) fit_transform(data [][]f64, feature_names []string) ![][]f64
fit_transform fits each step on its input and returns the transformed data.
fn (Pipeline) transform #
fn (p &Pipeline) transform(data [][]f64) ![][]f64
transform applies the fitted steps without changing their learned state.
fn (Pipeline) get_feature_names_out #
fn (p &Pipeline) get_feature_names_out() ![]string
get_feature_names_out returns the output feature names learned during fit.
struct PipelineStep #
struct PipelineStep {
pub:
name string
transformer MatrixTransformer
}
PipelineStep names one transformer in a numeric preprocessing pipeline.
struct PolynomialFeatures #
struct PolynomialFeatures {
pub:
degree int = 2
interaction_only bool
include_bias bool = true
}
PolynomialFeatures generates all polynomial combinations up to degree. Interaction-only mode uses each input feature at most once per term.
fn (PolynomialFeatures) fit #
fn (p PolynomialFeatures) fit(data [][]f64) !
fit validates the feature matrix before the transformer is used in a pipeline.
fn (PolynomialFeatures) transform #
fn (p PolynomialFeatures) transform(data [][]f64) ![][]f64
transform expands each row into polynomial features. Output columns are ordered by degree, then lexicographically by input feature index.
fn (PolynomialFeatures) get_feature_names_out #
fn (p PolynomialFeatures) get_feature_names_out(input_features []string) ![]string
get_feature_names_out returns names in the same order as transform.
struct RobustScaler #
struct RobustScaler {
mut:
fitted bool
pub mut:
median_ []f64 // median of each feature
iqr_ []f64 // interquartile range of each feature
n_features int
}
RobustScaler defines a public data structure for this module.
fn (RobustScaler) fit #
fn (mut s RobustScaler) fit(x [][]f64) !
fit computes the median and IQR to be used for later scaling
fn (RobustScaler) transform #
fn (s &RobustScaler) transform(x [][]f64) ![][]f64
transform applies the robust scaling to the data
fn (RobustScaler) fit_transform #
fn (mut s RobustScaler) fit_transform(x [][]f64) ![][]f64
fit_transform fits and transforms in one step
fn (RobustScaler) get_feature_names_out #
fn (s &RobustScaler) get_feature_names_out(input_features []string) ![]string
get_feature_names_out preserves names because scaling does not change columns.
struct SimpleImputer #
struct SimpleImputer {
mut:
fitted bool
pub:
strategy ImputationStrategy
fill_value f64
pub mut:
statistics_ []f64
n_features int
}
SimpleImputer learns one replacement value per feature from training data.
fn (SimpleImputer) fit #
fn (mut imputer SimpleImputer) fit(data [][]f64) !
fit learns replacement values from the provided training matrix.
fn (SimpleImputer) transform #
fn (imputer &SimpleImputer) transform(data [][]f64) ![][]f64
transform replaces NaN values using statistics learned by fit.
fn (SimpleImputer) fit_transform #
fn (mut imputer SimpleImputer) fit_transform(data [][]f64) ![][]f64
fit_transform learns replacement values and applies them to the same matrix.
fn (SimpleImputer) get_feature_names_out #
fn (imputer &SimpleImputer) get_feature_names_out(input_features []string) ![]string
get_feature_names_out preserves names because imputation does not change columns.
struct StandardScaler #
struct StandardScaler {
mut:
fitted bool
pub mut:
mean_ []f64 // mean of each feature
std_ []f64 // standard deviation of each feature
n_features int
}
StandardScaler defines a public data structure for this module.
fn (StandardScaler) fit #
fn (mut s StandardScaler) fit(x [][]f64) !
fit computes the mean and std to be used for later scaling
fn (StandardScaler) transform #
fn (s &StandardScaler) transform(x [][]f64) ![][]f64
transform applies the standardization to the data
fn (StandardScaler) fit_transform #
fn (mut s StandardScaler) fit_transform(x [][]f64) ![][]f64
fit_transform fits and transforms in one step
fn (StandardScaler) inverse_transform #
fn (s &StandardScaler) inverse_transform(x [][]f64) ![][]f64
inverse_transform reverses the standardization
fn (StandardScaler) get_feature_names_out #
fn (s &StandardScaler) get_feature_names_out(input_features []string) ![]string
get_feature_names_out preserves names because scaling does not change columns.
- README
- fn cut
- fn digitize
- fn qcut
- fn Binner.new
- fn BinningStrategy.from
- fn ImputationStrategy.from
- fn LabelEncoder.new
- fn MinMaxScaler.new
- fn OneHotEncoder.new
- fn OrdinalEncoder.new
- fn Pipeline.new
- fn RobustScaler.new
- fn SimpleImputer.new
- fn StandardScaler.new
- interface MatrixTransformer
- enum BinningStrategy
- enum ImputationStrategy
- struct Binner
- struct LabelEncoder
- struct MinMaxScaler
- struct OneHotEncoder
- struct OrdinalEncoder
- struct Pipeline
- struct PipelineStep
- struct PolynomialFeatures
- struct RobustScaler
- struct SimpleImputer
- struct StandardScaler