lab_2_tokenize_by_bpe package

Submodules

Lab 2.

BPE and machine translation evaluation

lab_2_tokenize_by_bpe.main.calculate_bleu(actual, reference, max_order=3)

Compare two sequences by virtue of BLEU metric.

Parameters:

  • actual (str | None) – Predicted sequence

  • reference (str) – Expected sequence

  • max_order (int, optional) – Max length of n-gram to consider for comparison. Defaults to 3

Returns:

A value of BLEU metric

Return type:

float | None

In case of corrupt input arguments or functions used return None, None is returned

lab_2_tokenize_by_bpe.main.calculate_geo_mean(precisions, max_order)

Compute geometric mean of sequence of values.

Parameters:

  • precisions (Sequence[float]) – A sequence of Precision values

  • max_order (int) – Maximum length of n-gram considered

Returns:

A value of geometric mean of Precision metric

Return type:

float | None

In case of corrupt input arguments, None is returned

lab_2_tokenize_by_bpe.main.calculate_precision(actual, reference)

Compare two sequences by virtue of Precision metric.

Parameters:

  • actual (Sequence[tuple[str, ...]]) – Predicted sequence of n-grams

  • reference (Sequence[tuple[str, ...]]) – Expected sequence of n-grams

Returns:

Value of Precision metric

Return type:

float | None

In case of corrupt input arguments, None is returned.

lab_2_tokenize_by_bpe.main.collect_frequencies(text, start_of_word, end_of_word)

Count number of occurrences of each word.

Parameters:

  • text (str) – Original text with no preprocessing

  • start_of_word (str | None) – A token that signifies the start of word

  • end_of_word (str) – A token that signifies the end of word

Returns:

Dictionary where key - preprocessed word,

value - number of occurrences

Return type:

dict[tuple[str, …], int] | None

In case of corrupt input arguments or functions used return None, None is returned

lab_2_tokenize_by_bpe.main.collect_ngrams(text, order)

Extract n-grams from the given sequence.

Parameters:

  • text (str) – Original text

  • order (int) – Required number of elements in a single n-gram

Returns:

A sequence of n-grams

Return type:

list[tuple[str, …]] | None

In case of corrupt input arguments, None is returned

lab_2_tokenize_by_bpe.main.count_tokens_pairs(word_frequencies)

Count number of occurrences of each pair of subsequent tokens.

Parameters:

word_frequencies (dict[tuple[str, ...], int]) – A dictionary where key - preprocessed word, value - number of occurrences

Returns:

A dictionary where

key - token pair, value - number of occurrences

Return type:

dict[tuple[str, str], int] | None

In case of corrupt input arguments, None is returned

lab_2_tokenize_by_bpe.main.decode(encoded_text, vocabulary, end_of_word_token)

Translate encoded sequence into decoded one.

Parameters:

  • encoded_text (Sequence[int] | None) – A sequence of token identifiers

  • vocabulary (dict[str, int] | None) – A dictionary where key - token, value - identifier

  • end_of_word_token (str | None) – An end-of-word token

Returns:

Decoded sequence

Return type:

str | None

In case of corrupt input arguments, None is returned

lab_2_tokenize_by_bpe.main.encode(original_text, vocabulary, start_of_word_token, end_of_word_token, unknown_token)

Translate original text into a sequence of token identifiers.

Parameters:

  • original_text (str) – Original text

  • vocabulary (dict[str, int] | None) – A dictionary where key - token, value - identifier

  • start_of_word_token (str | None) – A start-of-word token

  • end_of_word_token (str | None) – An end-of-word token

  • unknown_token (str) – A token that signifies unknown sequence

Returns:

A list of token identifiers

Return type:

list[int] | None

In case of corrupt input arguments or functions used return None, None is returned

lab_2_tokenize_by_bpe.main.get_vocabulary(word_frequencies, unknown_token)

Establish correspondence between tokens and its integer identifier.

Parameters:

  • word_frequencies (dict[tuple[str, ...], int]) – A dictionary where key - preprocessed word, value - number of occurrences

  • unknown_token (str) – A token to signify an unknown token

Returns:

A dictionary where key - token, value - identifier

Return type:

dict[str, int] | None

In case of corrupt input arguments, None is returned

lab_2_tokenize_by_bpe.main.load_vocabulary(vocab_path)

Read and retrieve dictionary of type <token: identifier>.

Parameters:

vocab_path (str) – A path to the saved vocabulary

Returns:

A dictionary where key - token, value - identifier

Return type:

dict[str, int] | None

In case of corrupt input arguments, None is returned

lab_2_tokenize_by_bpe.main.merge_tokens(word_frequencies, pair)

Update word frequency dictionary by replacing a pair of tokens with a merged one.

Parameters:

  • word_frequencies (dict[tuple[str, ...], int]) – A dictionary where key - preprocessed word, value - number of occurrences

  • pair (tuple[str, str]) – A pair of tokens to be merged

Returns:

A dictionary where

key - preprocessed word, value - number of occurrences

Return type:

dict[tuple[str, …], int] | None

In case of corrupt input arguments, None is returned

lab_2_tokenize_by_bpe.main.prepare_word(raw_word, start_of_word, end_of_word)

Tokenize a word into characters and attach optional boundary tokens.

Parameters:

  • raw_word (str) – Original word

  • start_of_word (str | None) – A token that signifies the start of word

  • end_of_word (str | None) – A token that signifies the end of word

Returns:

Preprocessed word

Return type:

tuple[str, …] | None

In case of corrupt input arguments, None is returned

lab_2_tokenize_by_bpe.main.tokenize_word(word, vocabulary, end_of_word, unknown_token)

Split word into tokens.

Parameters:

  • word (tuple[str, ...]) – Preprocessed word

  • vocabulary (dict[str, int]) – A dictionary where key - token, value - identifier

  • end_of_word (str | None) – An end-of-word token

  • unknown_token (str) – A token that signifies unknown sequence

Returns:

A list of token identifiers

Return type:

list[int] | None

In case of corrupt input arguments, None is returned

lab_2_tokenize_by_bpe.main.train(word_frequencies, num_merges)

Create required number of new tokens by merging existing ones.

Parameters:

  • word_frequencies (dict[tuple[str, ...], int] | None) – A dictionary where key - preprocessed word, value - number of occurrences

  • num_merges (int) – Required number of new tokens

Returns:

A dictionary where

key - preprocessed word, value - number of occurrences

Return type:

dict[tuple[str, …], int] | None

In case of corrupt input arguments or functions used return None, None is returned