VTL Datasets

VTL provides dataset loaders and batching utilities for ML examples and tests.

Available datasets

DataLoader returns a view when a batch's sample indices form a contiguous range. Shuffled, non-contiguous batches allocate tensors to preserve the requested sample order. A non-positive batch_size produces an empty loader.

DataLoaderConfig defaults to batch_size: 32, shuffle: true, and drop_last: true. These differ from PyTorch's defaults (1, false, and false), so set all three explicitly when porting a training pipeline. Call reset() between epochs to advance the deterministic shuffle order from the configured seed.

For custom sample order, new_data_loader_with_sampler_checked and new_data_loader_with_labels_and_sampler_checked accept samplers from vtl.nn.data. Built-in options include sequential, random, subset-random, weighted, and fixed batch sampling. Wrap a sampler with FixedBatchSampler to choose batch boundaries and drop_last explicitly. Call reset_checked() between epochs when using a sampler; it reports errors from custom sampler implementations.

For custom batch construction, install a CollateFn[T] with DataLoader.set_collate_fn and call collated_batch(i) or for_each_collated(...). The callback receives the source feature tensor, an optional labels tensor, and a copy of the sample indices selected for that batch. It returns a CollatedBatch[T] containing the output features and optional labels. This supports dense tensor collation such as transforms, padding, filtering, and reordering; it does not model arbitrary Python objects or nested sample structures. See the custom collation example.

load_imdb() returns the full 25,000 review training and test splits. Use load_imdb_with_config for a smaller, class-balanced subset. Review features remain raw strings; labels are binary (1 positive, 0 negative).

For simple NLP pipelines, tokenize_text lowercases and splits on Unicode whitespace and ASCII punctuation while preserving UTF-8 text. build_text_vocabulary creates deterministic frequency-ranked IDs; <pad> is ID 0 and <unk> is ID 1. max_size includes those reserved IDs, and 0 means no size limit. This is a small preprocessing utility; it does not replace language-specific tokenizers or normalization.

import vtl.datasets

tokens := datasets.tokenize_text('A joyful, moving review!')
assert tokens == ['a', 'joyful', 'moving', 'review']

vocabulary := datasets.build_text_vocabulary([
    'joyful moving review',
    'moving review',
], 10000, 1)!
ids := vocabulary.encode('Joyful review')
assert vocabulary.decode(ids) == ['joyful', 'review']

import vtl.datasets

imdb := datasets.load_imdb_with_config(datasets.ImdbConfig{
    train_count: 1000
    test_count:  200
})!
assert imdb.train_features.shape == [1000]
assert imdb.test_features.shape == [200]

Examples

Run from ~/.vmodules:

v run vtl/examples/datasets_mnist/main.v
v run vtl/examples/datasets_imdb/main.v
v run ./vtl/examples/nn_cifar10_tiny_synth/main.v

Use synthetic examples (nn_cifar10_tiny_synth, nn_cifar10_f32_tiny_synth) for CI and quick local checks. Real dataset examples may download/cache data and should be treated as local integration tests.

Related docs

• Examples catalog
• Neural network tutorial
• Lightweight development commands

fn build_text_vocabulary(texts []string, max_size int, min_frequency int) !TextVocabulary

build_text_vocabulary builds a frequency-ranked vocabulary from texts. max_size includes the reserved and entries; zero means unlimited. Tokens with frequency below min_frequency are omitted. Equal-frequency tokens are ordered lexicographically to make the result deterministic.

fn class_names() []string

class_names returns the CIFAR-10 class names.

fn load_cifar10() !Cifar10Dataset

load_cifar10 loads the CIFAR-10 dataset. Returns train/test features (normalized [0,1]) and one-hot encoded labels.

fn load_cifar10_with_config(cfg Cifar10Config) !Cifar10Dataset

load_cifar10_with_config loads CIFAR-10 with custom configuration.

fn load_imdb() !ImdbDataset

load_imdb loads the IMDB dataset.

fn load_imdb_with_config(cfg ImdbConfig) !ImdbDataset

load_imdb_with_config loads a balanced subset of the IMDB dataset.

fn load_mnist() !MnistDataset

load_mnist loads the MNIST dataset.

fn new_data_loader[T](dataset &vtl.Tensor[T], config DataLoaderConfig) &DataLoader[T]

new_data_loader creates a DataLoader from a dataset tensor (features only).

fn new_data_loader_with_batch_sampler_checked[T](dataset &vtl.Tensor[T], config DataLoaderConfig, batch_sampler nn_data.BatchSampler) !&DataLoader[T]

new_data_loader_with_batch_sampler_checked creates a loader using explicit sample batches from batch_sampler.

fn new_data_loader_with_labels[T](dataset &vtl.Tensor[T], labels &vtl.Tensor[T], config DataLoaderConfig) &DataLoader[T]

new_data_loader_with_labels creates a DataLoader with both features and labels tensors. Both tensors must have the same first dimension (number of samples). This compatibility constructor panics for invalid shapes; prefer the checked variant when input dimensions are not already trusted.

fn new_data_loader_with_labels_and_batch_sampler_checked[T](dataset &vtl.Tensor[T], labels &vtl.Tensor[T], config DataLoaderConfig, batch_sampler nn_data.BatchSampler) !&DataLoader[T]

new_data_loader_with_labels_and_batch_sampler_checked creates a labeled loader using explicit sample batches and preserves feature-label alignment.

fn new_data_loader_with_labels_and_sampler_checked[T](dataset &vtl.Tensor[T], labels &vtl.Tensor[T], config DataLoaderConfig, sampler nn_data.Sampler) !&DataLoader[T]

new_data_loader_with_labels_and_sampler_checked creates a labeled loader with a custom sample order and validates the data and sampler indices.

fn new_data_loader_with_labels_checked[T](dataset &vtl.Tensor[T], labels &vtl.Tensor[T], config DataLoaderConfig) !&DataLoader[T]

new_data_loader_with_labels_checked creates a labeled DataLoader and reports invalid dataset or label shapes as an error instead of deferring failure until a batch is read.

fn new_data_loader_with_sampler_checked[T](dataset &vtl.Tensor[T], config DataLoaderConfig, sampler nn_data.Sampler) !&DataLoader[T]

new_data_loader_with_sampler_checked creates a loader whose sample order is supplied by sampler. The sampler is invoked again by reset_checked each epoch.

fn tokenize_text(text string) []string

tokenize_text lowercases text and splits on whitespace and ASCII punctuation. Non-ASCII characters are preserved so UTF-8 words remain intact.

type CollateFn[T] = fn (dataset &vtl.Tensor[T], labels &vtl.Tensor[T], indices []int) !CollatedBatch[T]

CollateFn builds tensors for the selected sample indices. labels is nil for feature-only loaders. The callback owns the collation policy, including output shapes and ordering.

fn (mut it DataLoaderIterator[T]) next[T]() ?&vtl.Tensor[T]

next returns the next batch. Returns none when exhausted.

fn (dl &DataLoader[T]) batch(i int) ?&vtl.Tensor[T]

batch returns a batch at index i. Contiguous indices return a view into the dataset; other index orders return a copy. Mutating a returned view also mutates the corresponding dataset elements. Returns none for invalid indices or a non-positive batch size.

fn (dl &DataLoader[T]) batch_with_labels(i int) ?(&vtl.Tensor[T], &vtl.Tensor[T])

batch_with_labels returns both features and labels for batch i. Both tensors are extracted using the same index set. Returns none if the index is out of range or if no labels tensor was provided.

fn (mut dl DataLoader[T]) set_collate_fn(collate_fn CollateFn[T])

set_collate_fn installs a callback used by collated_batch. Existing batch accessors keep their default tensor slicing and gathering behavior.

fn (dl &DataLoader[T]) collated_batch(i int) !CollatedBatch[T]

collated_batch builds the batch selected by i. A custom callback receives the source tensors and a copy of the selected indices, so it can implement custom padding, filtering, transforms, or ordering without changing loader state. Without a callback, this returns the same default tensors as batch and batch_with_labels.

fn (dl &DataLoader[T]) len() int

len returns the number of batches (drop_last affects count).

fn (dl &DataLoader[T]) total_samples() int

total_samples returns the number of samples in the dataset.

fn (mut dl DataLoader[T]) reset()

reset re-shuffles the indices (call between epochs). Uses seed if set.

fn (mut dl DataLoader[T]) reset_checked() !

reset_checked advances the loader to a new epoch and reports sampler errors.

fn (dl &DataLoader[T]) iter[T]() DataLoaderIterator[T]

iter returns an iterator over the DataLoader batches.

fn (dl &DataLoader[T]) for_each(fn_each fn (batch &vtl.Tensor[T]) bool) int

for_each applies fn to each batch until exhaustion or fn returns false.

fn (dl &DataLoader[T]) for_each_with_labels(fn_each fn (features &vtl.Tensor[T], labels &vtl.Tensor[T]) bool) int

for_each_with_labels applies fn to each (features, labels) batch. Only works if the DataLoader was created with labels.

fn (dl &DataLoader[T]) for_each_collated(fn_each fn (batch CollatedBatch[T]) bool) !int

for_each_collated applies the custom collation callback to every batch and passes each result to fn_each. Callback errors stop iteration and return to the caller.

fn (dl &DataLoader[T]) split(val_fraction f64) !(&DataLoader[T], &DataLoader[T])

split splits a DataLoader into train and validation DataLoaders. val_fraction must be finite and in the inclusive range [0, 1].

fn (dl &DataLoader[T]) mini_batch_count() int

mini_batch_count returns the total number of batches in one epoch.

struct Cifar10Config {
pub:
	channels    int = 3
	height      int = 32
	width       int = 32
	num_classes int = 10
	train_count int = 50000
	test_count  int = 10000
}

Cifar10Config holds configuration for CIFAR-10 dataset loading.

struct Cifar10Dataset {
pub:
	train_features &vtl.Tensor[f64] = unsafe { nil }
	train_labels   &vtl.Tensor[f64] = unsafe { nil }
	test_features  &vtl.Tensor[f64] = unsafe { nil }
	test_labels    &vtl.Tensor[f64] = unsafe { nil }
}

Cifar10Dataset holds the CIFAR-10 dataset.

struct CollatedBatch[T] {
pub:
	features &vtl.Tensor[T]
	labels   &vtl.Tensor[T] = unsafe { nil }
}

CollatedBatch contains the tensors produced for one selected batch. Labels are nil when the source loader has no labels or the collator omits them.

struct DataLoader[T] {
pub:
	dataset    &vtl.Tensor[T]
	labels     &vtl.Tensor[T] = unsafe { nil }
	batch_size int
	shuffle    bool
	drop_last  bool
	seed       u64
pub mut:
	indices []int
	epoch   int
mut:
	sampler            nn_data.Sampler
	uses_sampler       bool
	batch_sampler      nn_data.BatchSampler
	uses_batch_sampler bool
	batches            [][]int
	collate_fn         CollateFn[T] = unsafe { nil }
}

DataLoader defines a public data structure for this module.

struct DataLoaderConfig {
pub:
	batch_size int  = 32
	shuffle    bool = true
	drop_last  bool = true
	seed       u64
}

DataLoaderConfig holds configuration for creating a DataLoader.

struct DataLoaderIterator[T] {
mut:
	loader  &DataLoader[T]
	current int
}

DataLoaderIterator provides a for-in loop interface over a DataLoader.

struct ImdbConfig {
pub:
	train_count int = 25000
	test_count  int = 25000
}

ImdbConfig configures the number of balanced examples returned per split.

struct ImdbDataset {
pub:
	train_features &vtl.Tensor[string] = unsafe { nil }
	train_labels   &vtl.Tensor[int]    = unsafe { nil }
	test_features  &vtl.Tensor[string] = unsafe { nil }
	test_labels    &vtl.Tensor[int]    = unsafe { nil }
}

ImdbDataset is a dataset for sentiment analysis.

struct MnistDataset {
pub:
	train_features &vtl.Tensor[u8] = unsafe { nil }
	train_labels   &vtl.Tensor[u8] = unsafe { nil }
	test_features  &vtl.Tensor[u8] = unsafe { nil }
	test_labels    &vtl.Tensor[u8] = unsafe { nil }
}

MnistDataset is a dataset of MNIST handwritten digits.

struct TextVocabulary {
pub:
	tokens []string
mut:
	token_to_index map[string]int
}

TextVocabulary maps normalized text tokens to stable integer IDs. The first IDs are reserved for padding and out-of-vocabulary tokens.

fn (v &TextVocabulary) index(token string) int

index returns the ID for token, or the unknown-token ID when it is absent.

fn (v &TextVocabulary) encode(text string) []int

encode tokenizes text and returns its vocabulary IDs.

fn (v &TextVocabulary) decode(indices []int) []string

decode converts IDs back to tokens. Invalid IDs decode as .