model_selection #
VSL Model Selection Module
The vsl.model_selection module provides utilities for splitting data and performing cross-validation for machine learning model evaluation.
Features
Train-Test Splitting
- train_test_split: Split arrays into random train and test subsets
- train_val_test_split: Three-way split into train, validation, and test
- Stratified splitting: Preserve class proportions in splits
Cross-Validation
- KFold: K-Fold cross-validation iterator
- StratifiedKFold: Stratified K-Fold (preserves class balance)
- LeaveOneOut: Leave-one-out cross-validation
- ShuffleSplit: Random permutation cross-validator
Quick Start
Basic Train-Test Split
import vsl.model_selection
x := [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0], [7.0, 8.0], [9.0, 10.0]]
y := [0.0, 1.0, 0.0, 1.0, 0.0]
result := model_selection.train_test_split(x, y, model_selection.TrainTestSplitConfig{
test_size: 0.2
shuffle: true
random_seed: 42
})!
println('Train size: ${result.x_train.len}')
println('Test size: ${result.x_test.len}')
Stratified Split (Preserve Class Proportions)
import vsl.model_selection
x := [[1.0], [2.0], [3.0], [4.0], [5.0], [6.0], [7.0], [8.0], [9.0], [10.0]]
y := [0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0]
result := model_selection.train_test_split(x, y, model_selection.TrainTestSplitConfig{
test_size: 0.3
stratify: true // Maintains 50/50 class balance in both sets
})!
Stratification treats each distinct finite f64 label as a separate class. NaN and infinite labels return an error. The requested overall test size is preserved while per-class counts are allocated by largest remainder; setting shuffle: false preserves the original sample order.
Train-Validation-Test Split
import vsl.model_selection
result := model_selection.train_val_test_split(x, y, model_selection.TrainValTestConfig{
val_size: 0.1 // 10% validation
test_size: 0.2 // 20% test (70% train)
stratify: true
})!
println('Train: ${result.x_train.len}')
println('Val: ${result.x_val.len}')
println('Test: ${result.x_test.len}')
K-Fold Cross-Validation
import vsl.model_selection
kf := model_selection.KFold.new(5, true, 42)
folds := kf.split(100)! // 100 samples
for i, fold in folds {
println('Fold ${i + 1}:')
println(' Train indices: ${fold.train_indices.len}')
println(' Test indices: ${fold.test_indices.len}')
// Get actual data for this fold
x_train, y_train, x_test, y_test := model_selection.get_fold_data(x, y, fold)
// Train and evaluate your model here
}
Stratified K-Fold
import vsl.model_selection
y := [0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
skf := model_selection.StratifiedKFold.new(5, true, 42)
folds := skf.split(y)!
// Each fold maintains class proportions
Leave-One-Out Cross-Validation
import vsl.model_selection
loo := model_selection.LeaveOneOut.new()
folds := loo.split(10) // Returns 10 folds, each with 1 test sample
for fold in folds {
// fold.test_indices has exactly 1 element
// fold.train_indices has n-1 elements
}
Shuffle Split (Multiple Random Splits)
import vsl.model_selection
ss := model_selection.ShuffleSplit.new(10, 0.2, 42) // 10 splits, 20% test
folds := ss.split(100)!
// Creates 10 different random splits
API Reference
TrainTestSplitConfig
| Field | Type | Default | Description |
|---|---|---|---|
test_size |
f64 |
0.25 |
Proportion for test set (0.0-1.0) |
shuffle |
bool |
true |
Whether to shuffle before splitting |
stratify |
bool |
false |
Preserve class proportions |
random_seed |
u32 |
42 |
Random seed for reproducibility |
TrainValTestConfig
| Field | Type | Default | Description |
|---|---|---|---|
val_size |
f64 |
0.1 |
Proportion for validation set |
test_size |
f64 |
0.2 |
Proportion for test set |
shuffle |
bool |
true |
Whether to shuffle |
stratify |
bool |
false |
Preserve class proportions |
random_seed |
u32 |
42 |
Random seed |
Cross-Validators
| Class | Description |
|---|---|
KFold |
Standard K-Fold CV |
StratifiedKFold |
Stratified K-Fold (for classification) |
LeaveOneOut |
Leave-one-out CV (n folds for n samples) |
ShuffleSplit |
Random permutation splits |
cross_val_score(x, y, folds, predictor, scorer) runs one prediction and scoring pass per supplied fold and returns one score per fold. The predictor receives x_train, y_train, and x_test (not the test labels), so it should create and fit a fresh estimator on each call. The scorer receives y_test and the predictions. Invalid, overlapping, duplicate, or out-of-range fold indices and prediction-length mismatches are returned as errors.
Scores are returned in the supplied fold order; this API does not average, negate, or otherwise transform them. For model selection, use a higher-is-better scorer (negate a loss when appropriate) and aggregate the fold scores explicitly. Labels and predictions are passed as []f64 without class encoding, thresholding, positive-class selection, or sample weighting; the scorer defines those conventions.
For clearer estimator lifecycle separation, cross_val_score_with_factory accepts a factory that receives only the current fold's training data and returns a predictor for held-out features. The factory runs once per fold and must create independently fitted model state:
import vsl.model_selection
x := [[0.0], [1.0], [2.0], [3.0]]
y := [0.0, 1.0, 3.0, 7.0]
folds := model_selection.KFold.new(2, false, 42).split(y.len)!
factory := fn (_x_train [][]f64, y_train []f64) !model_selection.FoldPrediction {
mut mean := 0.0
for value in y_train {
mean += value
}
mean /= f64(y_train.len)
return fn [mean] (x_test [][]f64) ![]f64 {
mut predictions := []f64{cap: x_test.len}
for _ in x_test {
predictions << mean
}
return predictions
}
}
scorer := fn (_y_true []f64, y_pred []f64) !f64 {
mut mean := 0.0
for value in y_pred {
mean += value
}
return mean / f64(y_pred.len)
}
scores := model_selection.cross_val_score_with_factory(x, y, folds, factory, scorer)!
cross_val_compare evaluates a bounded list of named factories sequentially on the same folds. It returns each candidate's fold scores and unweighted arithmetic mean in caller-supplied order without ranking them. Use cross_val_search when selection is desired; pass .maximize or .minimize explicitly so the library does not guess the scorer's direction. It returns all evaluations and the selected candidate index, keeps the first candidate on ties, and rejects NaN means because they cannot be ordered. The caller supplies candidate factories (which may capture hyperparameters); automatic Cartesian-grid construction and random sampling are not provided. Factories must fit independent state for each fold. The ml_cross_validation example compares logistic regression and Gaussian Naive Bayes and selects the best mean accuracy.
search := model_selection.cross_val_search(x, y, folds, candidates, 20,
.maximize, scorer)!
best := search.candidate_scores[search.best_candidate_index]
println('Best candidate: ${best.name} (${best.mean_score:.3f})')
Fold Structure
pub struct Fold {
pub:
train_indices []int // Indices for training
test_indices []int // Indices for testing
}
Helper Functions
| Function | Description |
|---|---|
get_fold_data(x, y, fold) |
Extract actual data arrays from fold indices |
cross_val_score(x, y, folds, predictor, scorer) |
Run a predictor and scorer for each validated fold |
cross_val_score_with_factory(x, y, folds, factory, scorer) |
Fit a new predictor from each fold's training data |
cross_val_compare(x, y, folds, candidates, max_candidates, scorer) |
Compare bounded candidate factories on shared folds |
cross_val_search(x, y, folds, candidates, max_candidates, direction, scorer) |
Select a bounded candidate using an explicit score direction |
Integration with ML Pipeline
import vsl.model_selection
import vsl.metrics
import vsl.ml
// Prepare data
x := your_features
y := your_labels
// Cross-validation
kf := model_selection.KFold.new(5, true, 42)
folds := kf.split(x.len)!
predictor := fn (x_train [][]f64, y_train []f64, x_test [][]f64) ![]f64 {
mut train_data := ml.Data.from_raw_xy_sep(x_train, y_train)!
mut model := ml.LogReg.new(mut train_data, 'cv_model')
model.train(ml.LogRegTrainConfig{})
mut predictions := []f64{}
for row in x_test {
predictions << model.predict(row)
}
return predictions
}
scorer := fn (y_true []f64, y_pred []f64) !f64 {
return metrics.accuracy_score(y_true, y_pred)
}
scores := model_selection.cross_val_score(x, y, folds, predictor, scorer)!
// Report average score
mut avg := 0.0
for s in scores {
avg += s
}
avg /= f64(scores.len)
println('Average CV Accuracy: ${avg}')
Best Practices
- Use stratified splitting for classification - Ensures representative class distribution
- Set random_seed for reproducibility - Same seed = same splits
- K=5 or K=10 for cross-validation - Common choices balancing bias-variance
- Use LeaveOneOut for small datasets - Maximizes training data
- Use ShuffleSplit for large datasets - Faster than full K-Fold
Notes
- Stratified methods treat each distinct finite
f64label as a class; NaN and infinite labels are rejected. - Cross-validation returns indices, use
get_fold_datafor actual arrays - Train and test indices are always disjoint within each fold
- Random seed ensures reproducibility across runs
See Also
fn cross_val_compare #
fn cross_val_compare(x [][]f64, y []f64, folds []Fold, candidates []CrossValCandidate, max_candidates int, scorer FoldScorer) ![]CrossValCandidateScore
cross_val_compare evaluates each named factory sequentially on the same validated folds. max_candidates bounds the number of candidate fits; scores remain in candidate/fold order and are not ranked or direction-transformed. mean_score is the unweighted arithmetic mean of that candidate's fold scores.
fn cross_val_score #
fn cross_val_score(x [][]f64, y []f64, folds []Fold, predictor FoldPredictor, scorer FoldScorer) ![]f64
cross_val_score evaluates a predictor independently on each supplied fold. The predictor receives training labels but only test features, avoiding target leakage; create an unfitted estimator inside each invocation. Scores are returned in fold order without aggregation or direction changes; label and higher-is-better conventions belong to the supplied scorer.
fn cross_val_score_with_factory #
fn cross_val_score_with_factory(x [][]f64, y []f64, folds []Fold, factory FoldEstimatorFactory, scorer FoldScorer) ![]f64
cross_val_score_with_factory calls a model factory once for every fold with only that fold's training data, then scores predictions on held-out features. The factory must return a predictor backed by new, independently fitted state.
fn cross_val_search #
fn cross_val_search(x [][]f64, y []f64, folds []Fold, candidates []CrossValCandidate, max_candidates int, direction CrossValScoreDirection, scorer FoldScorer) !CrossValSearchResult
cross_val_search evaluates a bounded caller-supplied candidate list and selects the highest or lowest unweighted mean score. Ties keep the earliest candidate. Scorers that return NaN cannot define a reliable ordering.
fn get_fold_data #
fn get_fold_data(x [][]f64, y []f64, fold Fold) ([][]f64, []f64, [][]f64, []f64)
get_fold_data extracts the actual data for a fold
fn train_test_split #
fn train_test_split(x [][]f64, y []f64, config TrainTestSplitConfig) !TrainTestResult
train_test_split splits arrays into random train and test subsets
fn train_val_test_split #
fn train_val_test_split(x [][]f64, y []f64, config TrainValTestConfig) !TrainValTestResult
train_val_test_split splits data into train, validation, and test sets
fn CrossValScoreDirection.from #
fn CrossValScoreDirection.from[W](input W) !CrossValScoreDirection
from returns the CrossValScoreDirection value of an integer or of a field name, or an error if there is none.
fn KFold.new #
fn KFold.new(n_splits int, shuffle bool, random_seed u32) &KFold
KFold.new creates a new KFold cross-validator
fn LeaveOneOut.new #
fn LeaveOneOut.new() &LeaveOneOut
LeaveOneOut.new creates a new LeaveOneOut cross-validator
fn ShuffleSplit.new #
fn ShuffleSplit.new(n_splits int, test_size f64, random_seed u32) &ShuffleSplit
ShuffleSplit.new creates a new ShuffleSplit cross-validator
fn StratifiedKFold.new #
fn StratifiedKFold.new(n_splits int, shuffle bool, random_seed u32) &StratifiedKFold
StratifiedKFold.new creates a new StratifiedKFold cross-validator
type FoldEstimatorFactory #
type FoldEstimatorFactory = fn (x_train [][]f64, y_train []f64) !FoldPrediction
FoldEstimatorFactory fits and returns a predictor using only one fold's training data. It is called once per fold and must not reuse fitted state.
type FoldPrediction #
type FoldPrediction = fn (x_test [][]f64) ![]f64
FoldPrediction predicts test labels using a model fitted for one fold.
type FoldPredictor #
type FoldPredictor = fn (x_train [][]f64, y_train []f64, x_test [][]f64) ![]f64
FoldPredictor fits a fresh estimator from one fold's training data and returns predictions for that fold's test features.
type FoldScorer #
type FoldScorer = fn (y_true []f64, y_pred []f64) !f64
FoldScorer computes one score from test labels and corresponding predictions.
enum CrossValScoreDirection #
enum CrossValScoreDirection {
maximize
minimize
}
CrossValScoreDirection specifies how a scorer's mean should be optimized.
struct CrossValCandidate #
struct CrossValCandidate {
pub:
name string
factory FoldEstimatorFactory @[required]
}
CrossValCandidate names one estimator configuration for bounded comparison. Capture its hyperparameters in factory; the factory must fit fresh state.
struct CrossValCandidateScore #
struct CrossValCandidateScore {
pub:
name string
fold_scores []f64
mean_score f64
}
CrossValCandidateScore contains fold-ordered scores and their unweighted arithmetic mean for one candidate.
struct CrossValSearchResult #
struct CrossValSearchResult {
pub:
candidate_scores []CrossValCandidateScore
best_candidate_index int
}
CrossValSearchResult contains all evaluations and the selected candidate's index in candidate_scores.
struct Fold #
struct Fold {
pub mut:
train_indices []int
test_indices []int
}
Fold represents a single fold in cross-validation
struct KFold #
struct KFold {
pub:
n_splits int = 5
shuffle bool = true
random_seed u32 = 42
}
KFold defines a public data structure for this module.
fn (KFold) split #
fn (kf &KFold) split(n_samples int) ![]Fold
split generates indices to split data into training and test sets
struct LeaveOneOut #
struct LeaveOneOut {
}
LeaveOneOut defines a public data structure for this module.
fn (LeaveOneOut) split #
fn (loo &LeaveOneOut) split(n_samples int) []Fold
split generates indices for leave-one-out CV
struct ShuffleSplit #
struct ShuffleSplit {
pub:
n_splits int = 5
test_size f64 = 0.2
random_seed u32 = 42
}
ShuffleSplit defines a public data structure for this module.
fn (ShuffleSplit) split #
fn (ss &ShuffleSplit) split(n_samples int) ![]Fold
split generates random train/test indices
struct StratifiedKFold #
struct StratifiedKFold {
pub:
n_splits int = 5
shuffle bool = true
random_seed u32 = 42
}
StratifiedKFold defines a public data structure for this module.
fn (StratifiedKFold) split #
fn (skf &StratifiedKFold) split(y []f64) ![]Fold
split generates stratified indices to split data
struct TrainTestResult #
struct TrainTestResult {
pub:
x_train [][]f64
x_test [][]f64
y_train []f64
y_test []f64
}
TrainTestResult holds the result of a train-test split
struct TrainTestSplitConfig #
struct TrainTestSplitConfig {
pub:
test_size f64 = 0.25 // proportion of data for test set
shuffle bool = true // whether to shuffle before splitting
stratify bool // whether to preserve class proportions
random_seed u32 = 42 // random seed for reproducibility
}
TrainTestSplitConfig holds configuration for train-test split
struct TrainValTestConfig #
struct TrainValTestConfig {
pub:
val_size f64 = 0.1 // proportion for validation
test_size f64 = 0.2 // proportion for test
shuffle bool = true
stratify bool
random_seed u32 = 42
}
TrainValTestConfig holds configuration for train-val-test split
struct TrainValTestResult #
struct TrainValTestResult {
pub:
x_train [][]f64
x_val [][]f64
x_test [][]f64
y_train []f64
y_val []f64
y_test []f64
}
TrainValTestResult holds the result of a train-val-test split
- README
- fn cross_val_compare
- fn cross_val_score
- fn cross_val_score_with_factory
- fn cross_val_search
- fn get_fold_data
- fn train_test_split
- fn train_val_test_split
- fn CrossValScoreDirection.from
- fn KFold.new
- fn LeaveOneOut.new
- fn ShuffleSplit.new
- fn StratifiedKFold.new
- type FoldEstimatorFactory
- type FoldPrediction
- type FoldPredictor
- type FoldScorer
- enum CrossValScoreDirection
- struct CrossValCandidate
- struct CrossValCandidateScore
- struct CrossValSearchResult
- struct Fold
- struct KFold
- struct LeaveOneOut
- struct ShuffleSplit
- struct StratifiedKFold
- struct TrainTestResult
- struct TrainTestSplitConfig
- struct TrainValTestConfig
- struct TrainValTestResult