Natural Language Processing Tools

This submodule offers tools for Natural Language Processing.

Examples

Here you can see a full set of examples.

fn count_vectorize(ngrams [][]string, most_frequent [][]string) []int

count_vectorize will give you an array of occurrences of each ngram from ngrams in most_frequent. Assume ng := [['hello'], ['hello'], ['hi']]. nlp.count_vectorize(ng, nlp.most_frequent_ngrams(ng, 0)) should return [2, 1]. See most_frequent_ngrams for more details on how it works.

fn inverse_document_frequencies(document [][][]string) !map[string]f64

inverse_document_frequencies will return the IDF of each term by calling term_idf on each unique ngram. Check term_idf for more details about the parameter document.

fn most_frequent_ngrams(ngrams [][]string, n_features int) ![][]string

most_frequent_ngrams returns an array with up to n_features elements denoting the most frequent ngrams in ngrams. Since V does not support map of arrays, the ngrams are joined by "NGRAMSEP". If n_features is <= 0, it will be set to ngrams.len.

fn ngrams(tokens []string, n int) ![][]string

ngrams will return an array of grams containing n elements from tokens. Example: ngrams('the apple is red'.split(' '), 3) will return [['the', 'apple', 'is'], ['apple', 'is', 'red']]

fn remove_punctuation(x string) string

remove_punctuation will remove the following characters from the string: ,.[]()[]-=_*;:+><\\"´~^!?@#$%¨&/|'`

fn remove_stopwords(tokens []string, stopwords []string, ignore_case bool) []string

remove_stopwords will remove all tokens included in stopwords. If ignore_case is true, "THIS" will be considered as "this".

fn remove_stopwords_en(tokens []string, ignore_case bool) []string

remove_stopwords_en is a wrapper for remove_stopwords, passing a default array of some English stopwords.

fn term_frequencies(ngrams_sentence [][]string) !map[string]f64

term_frequencies will return the frequency of each term in a sentence. However, since in VSL NLP tools we deal with ngrams instead of regular "words" (1gram tokens), our sentence is actually a collection of ngrams and our words/terms are the ngrams themselves. Keep in mind that, since V does NOT support maps of arrays, ngrams are joined by the constant ngram_sep, which can be found in ml/tokenizer.v. This is why it returns a map[string]f64 and not a map[[]string]f64.

fn term_idf(term []string, document [][][]string) !f64

term_idf will return the IDF of a single term term. However, since in VSL NLP tools we deal with ngrams instead of regular "words" (1gram tokens), our document is actually a collection of ngrams and our words/terms are the ngrams themselves. This means that a document (a collection of sentences) is actually a [][][]string (array of sentences, which by themselves are arrays of ngrams, which are arrays of string). Keep in mind that, since V does NOT support maps of arrays, ngrams are joined by the constant ngram_sep, which can be found in ml/tokenizer.v. This is why it returns a map[string]f64 and not a map[[]string]f64.

fn tf_idf(ngram []string, sentence [][]string, document [][][]string) !f64

tf_idf will return the TF * IDF for any given ngram, in a sentence, in a document.

fn tokenize(x string) []string

tokenize will return an array of tokens from the string x.

fn LancasterStemmer.new(strip_prefix bool) LancasterStemmer

LancasterStemmer.new returns a LancasterStemmer struct with a predefined set of stemming rules.

struct LancasterStemmer {
mut:
	rule_map map[string][]string
pub mut:
	strip_prefix bool
	rules        []string = [
		'ai*2.',
		'a*1.',
		'bb1.',
		'city3s.',
		'ci2>',
		'cn1t>',
		'dd1.',
		'dei3y>',
		'deec2ss.',
		'dee1.',
		'de2>',
		'dooh4>',
		'e1>',
		'feil1v.',
		'fi2>',
		'gni3>',
		'gai3y.',
		'ga2>',
		'gg1.',
		'ht*2.',
		'hsiug5ct.',
		'hsi3>',
		'i*1.',
		'i1y>',
		'ji1d.',
		'juf1s.',
		'ju1d.',
		'jo1d.',
		'jeh1r.',
		'jrev1t.',
		'jsim2t.',
		'jn1d.',
		'j1s.',
		'lbaifi6.',
		'lbai4y.',
		'lba3>',
		'lbi3.',
		'lib2l>',
		'lc1.',
		'lufi4y.',
		'luf3>',
		'lu2.',
		'lai3>',
		'lau3>',
		'la2>',
		'll1.',
		'mui3.',
		'mu*2.',
		'msi3>',
		'mm1.',
		'nois4j>',
		'noix4ct.',
		'noi3>',
		'nai3>',
		'na2>',
		'nee0.',
		'ne2>',
		'nn1.',
		'pihs4>',
		'pp1.',
		're2>',
		'rae0.',
		'ra2.',
		'ro2>',
		'ru2>',
		'rr1.',
		'rt1>',
		'rei3y>',
		'sei3y>',
		'sis2.',
		'si2>',
		'ssen4>',
		'ss0.',
		'suo3>',
		'su*2.',
		's*1>',
		's0.',
		'tacilp4y.',
		'ta2>',
		'tnem4>',
		'tne3>',
		'tna3>',
		'tpir2b.',
		'tpro2b.',
		'tcud1.',
		'tpmus2.',
		'tpec2iv.',
		'tulo2v.',
		'tsis0.',
		'tsi3>',
		'tt1.',
		'uqi3.',
		'ugo1.',
		'vis3j>',
		'vie0.',
		'vi2>',
		'ylb1>',
		'yli3y>',
		'ylp0.',
		'yl2>',
		'ygo1.',
		'yhp1.',
		'ymo1.',
		'ypo1.',
		'yti3>',
		'yte3>',
		'ytl2.',
		'yrtsi5.',
		'yra3>',
		'yro3>',
		'yfi3.',
		'ycn2t>',
		'yca3>',
		'zi2>',
		'zy1s.',
	]
}

All credits go to the respective authors of NLTK's LancasterStemmer implementation.

fn (mut stemmer LancasterStemmer) set_rules(rules []string) !

set_rules redefines the rules of stemmer and parses it.

fn (mut stemmer LancasterStemmer) stem(word string) !string

stem serves as a wrapper for do_stemming, which is private.