lab_3_generate_by_ngrams package

Submodules

Lab 3.

Beam-search and natural language generation evaluation

class lab_3_generate_by_ngrams.main.BackOffGenerator(language_models, text_processor)

Bases: object

Language model for back-off based text generation.

Parameters:

language_models (tuple[NGramLanguageModel, ...])
text_processor (TextProcessor)

_language_models

Language models for next token prediction

Type:: dict[int, NGramLanguageModel]

_text_processor

A TextProcessor instance to handle text processing

Type:: TextProcessor

__init__(language_models, text_processor)

Initializes an instance of BackOffGenerator.

Parameters:

language_models (tuple[NGramLanguageModel, ...]) – Language models to use for text generation
text_processor (TextProcessor) – A TextProcessor instance to handle text processing

Return type:

None

_get_next_token(sequence_to_continue)

Retrieve next tokens for sequence continuation.

Parameters:: sequence_to_continue (tuple[int, ...]) – Sequence to continue
Returns:: Next tokens for sequence continuation
Return type:: dict[int, float] | None

In case of corrupt input arguments return None.

run(prompt, seq_len)

Generate sequence based on NGram language model and prompt provided.

Parameters:

prompt (str) – Beginning of sequence
seq_len (int) – Number of tokens to generate

Returns:

Generated sequence

Return type:

str | None

In case of corrupt input arguments or methods used return None, None is returned

class lab_3_generate_by_ngrams.main.BeamSearchTextGenerator(language_model, text_processor, beam_width)

Bases: object

Class for text generation with BeamSearch.

Parameters:

language_model (NGramLanguageModel)
text_processor (TextProcessor)
beam_width (int)

_language_model

Language model for next token prediction

Type:: NGramLanguageModel

_text_processor

A TextProcessor instance to handle text processing

Type:: TextProcessor

_beam_width

Beam width parameter for generation

Type:: int

beam_searcher

Searcher instance for the language model

Type:: BeamSearcher

__init__(language_model, text_processor, beam_width)

Initializes an instance of BeamSearchTextGenerator.

Parameters:

language_model (NGramLanguageModel) – Language model to use for text generation
text_processor (TextProcessor) – A TextProcessor instance to handle text processing
beam_width (int) – Beam width parameter for generation

Return type:

None

_get_next_token(sequence_to_continue)

Retrieve next tokens for sequence continuation.

Parameters:: sequence_to_continue (tuple[int, ...]) – Sequence to continue
Returns:: Next tokens for sequence continuation
Return type:: list[tuple[int, float]] | None

In case of corrupt input arguments return None.

run(prompt, seq_len)

Generate sequence based on NGram language model and prompt provided.

Parameters:

prompt (str) – Beginning of sequence
seq_len (int) – Number of tokens to generate

Returns:

Generated sequence

Return type:

str | None

In case of corrupt input arguments or methods used return None, None is returned

class lab_3_generate_by_ngrams.main.BeamSearcher(beam_width, language_model)

Bases: object

Beam Search algorithm for diverse text generation.

Parameters:

beam_width (int)
language_model (NGramLanguageModel)

_beam_width

Number of candidates to consider at each step

Type:: int

_model

A language model to use for next token prediction

Type:: NGramLanguageModel

__init__(beam_width, language_model)

Initialize an instance of BeamSearcher.

Parameters:

beam_width (int) – Number of candidates to consider at each step
language_model (NGramLanguageModel) – A language model to use for next token prediction

Return type:

None

continue_sequence(sequence, next_tokens, sequence_candidates)

Generate new sequences from the base sequence with next tokens provided.

The base sequence is deleted after continued variations are added.

Parameters:

sequence (tuple[int, ...]) – Base sequence to continue
next_tokens (list[tuple[int, float]]) – Token for sequence continuation
sequence_candidates (dict[tuple[int, ...], float]) – Storage with all sequences generated

Returns:

Updated sequence candidates

Return type:

dict[tuple[int, …], float] | None

In case of corrupt input arguments or unexpected behaviour of methods used return None.

get_next_token(sequence)

Retrieve candidate tokens for sequence continuation.

Candidates are sorted by probability and then by token in descending order. Number of tokens retrieved must not be bigger than beam width parameter.

The return value has the following format: [(token, probability), …].

Parameters:: sequence (tuple[int, ...]) – Base sequence to continue
Returns:: Tokens to use for base sequence continuation
Return type:: list[tuple[int, float]] | None

In case of corrupt input arguments or methods used return None.

prune_sequence_candidates(sequence_candidates)

Remove those sequence candidates that do not make top-N most probable sequences.

Parameters:: sequence_candidates (dict[tuple[int, ...], float]) – Current candidate sequences
Returns:: Pruned sequences
Return type:: dict[tuple[int, …], float] | None

In case of corrupt input arguments return None.

class lab_3_generate_by_ngrams.main.GreedyTextGenerator(language_model, text_processor)

Bases: object

Greedy text generation by N-grams.

Parameters:

language_model (NGramLanguageModel)
text_processor (TextProcessor)

_model

A language model to use for text generation

Type:: NGramLanguageModel

_text_processor

A TextProcessor instance to handle text processing

Type:: TextProcessor

__init__(language_model, text_processor)

Initialize an instance of GreedyTextGenerator.

Parameters:

language_model (NGramLanguageModel) – A language model to use for text generation
text_processor (TextProcessor) – A TextProcessor instance to handle text processing

Return type:

None

run(prompt, seq_len)

Generate sequence based on NGram language model and prompt provided.

Parameters:

prompt (str) – Beginning of sequence
seq_len (int) – Number of tokens to generate

Returns:

Generated sequence

Return type:

str | None

In case of corrupt input arguments or methods used return None, None is returned

class lab_3_generate_by_ngrams.main.NGramLanguageModel(encoded_corpus, n_gram_size)

Bases: object

Store language model by n_grams, predict the next token.

Parameters:

encoded_corpus (tuple | None)
n_gram_size (int)

__n_gram_size

A size of n-grams to use for language modelling

Type:: int

_n_gram_frequencies

Frequencies for n-grams

Type:: dict

_encoded_corpus

Encoded text

Type:: tuple

__init__(encoded_corpus, n_gram_size)

Initialize an instance of NGramLanguageModel.

Parameters:

encoded_corpus (tuple | None) – Encoded text
n_gram_size (int) – A size of n-grams to use for language modelling

Return type:

None

_extract_n_grams(encoded_corpus)

Split encoded sequence into n-grams.

Parameters:: encoded_corpus (tuple[int, ...]) – A tuple of encoded tokens
Returns:: A tuple of extracted n-grams
Return type:: tuple[tuple[int, …], …] | None

In case of corrupt input arguments, None is returned

build()

Fill attribute _n_gram_frequencies from encoded corpus.

Encoded corpus is stored in the attribute _encoded_corpus

Returns:: True if attribute is filled successfully, otherwise False
Return type:: bool

In case of corrupt input arguments or methods used return None, False is returned

generate_next_token(sequence)

Retrieve tokens that can continue the given sequence along with their probabilities.

Parameters:: sequence (tuple[int, ...]) – A sequence to match beginning of NGrams for continuation
Returns:: Possible next tokens with their probabilities
Return type:: dict | None

In case of corrupt input arguments, None is returned

get_n_gram_size()

Retrieve value stored in self.__n_gram_size attribute.

Returns:: Size of stored n_grams
Return type:: int

set_n_grams(frequencies)

Setter method for n-gram frequencies.

Parameters:: frequencies (dict) – Computed in advance frequencies for n-grams
Return type:: None

class lab_3_generate_by_ngrams.main.NGramLanguageModelReader(json_path, eow_token)

Bases: object

Factory for loading language models ngrams from external JSON.

Parameters:

json_path (str)
eow_token (str)

_json_path

Local path to assets file

Type:: str

_eow_token

Special token for text processor

Type:: str

_text_processor

A TextProcessor instance to handle text processing

Type:: TextProcessor

__init__(json_path, eow_token)

Initialize reader instance.

Parameters:

json_path (str) – Local path to assets file
eow_token (str) – Special token for text processor

Return type:

None

get_text_processor()

Get method for the processor created for the current JSON file.

Returns:: processor created for the current JSON file.
Return type:: TextProcessor

load(n_gram_size)

Fill attribute _n_gram_frequencies from dictionary with N-grams.

The N-grams taken from dictionary must be cleaned from digits and punctuation, their length must match n_gram_size, and spaces must be replaced with EoW token.

Parameters:: n_gram_size (int) – Size of ngram
Returns:: Built language model.
Return type:: NGramLanguageModel | None

In case of corrupt input arguments or unexpected behaviour of methods used, None is returned.

class lab_3_generate_by_ngrams.main.TextProcessor(end_of_word_token)

Bases: object

Handle text tokenization, encoding and decoding.

Parameters:: end_of_word_token (str)

_end_of_word_token

A token denoting word boundary

Type:: str

_storage

Dictionary in the form of <token: identifier>

Type:: dict

__init__(end_of_word_token)

Initialize an instance of TextProcessor.

Parameters:: end_of_word_token (str) – A token denoting word boundary
Return type:: None

_decode(corpus)

Decode sentence by replacing ids with corresponding letters.

Parameters:: corpus (tuple[int, ...]) – A tuple of encoded tokens
Returns:: Sequence with decoded tokens
Return type:: tuple[str, …] | None

In case of corrupt input arguments, None is returned. In case any of methods used return None, None is returned.

_postprocess_decoded_text(decoded_corpus)

Convert decoded sentence into the string sequence.

Special symbols are replaced with spaces (no multiple spaces in a row are allowed). The first letter is capitalized, resulting sequence must end with a full stop.

Parameters:: decoded_corpus (tuple[str, ...]) – A tuple of decoded tokens
Returns:: Resulting text
Return type:: str | None

In case of corrupt input arguments, None is returned

_put(element)

Put an element into the storage, assign a unique id to it.

Parameters:: element (str) – An element to put into storage
Return type:: None

In case of corrupt input arguments or invalid argument length, an element is not added to storage

_tokenize(text)

Tokenize text into unigrams, separating words with special token.

Punctuation and digits are removed. EoW token is appended after the last word in two cases: 1. It is followed by punctuation 2. It is followed by space symbol

Parameters:: text (str) – Original text
Returns:: Tokenized text
Return type:: tuple[str, …] | None

In case of corrupt input arguments, None is returned. In case any of methods used return None, None is returned.

decode(encoded_corpus)

Decode and postprocess encoded corpus by converting integer identifiers to string.

Special symbols are replaced with spaces (no multiple spaces in a row are allowed). The first letter is capitalized, resulting sequence must end with a full stop.

Parameters:: encoded_corpus (tuple[int, ...]) – A tuple of encoded tokens
Returns:: Resulting text
Return type:: str | None

In case of corrupt input arguments, None is returned. In case any of methods used return None, None is returned.

encode(text)

Encode text.

Tokenize text, assign each symbol an integer identifier and replace letters with their ids.

Parameters:: text (str) – An original text to be encoded
Returns:: Processed text
Return type:: tuple[int, …] | None

In case of corrupt input arguments, None is returned. In case any of methods used return None, None is returned.

fill_from_ngrams(content)

Fill internal storage with letters from external JSON.

Parameters:: content (dict) – ngrams from external JSON
Return type:: None

get_end_of_word_token()

Retrieve value stored in self._end_of_word_token attribute.

Returns:: EoW token
Return type:: str

get_id(element)

Retrieve a unique identifier of an element.

Parameters:: element (str) – String element to retrieve identifier for
Returns:: Integer identifier that corresponds to the given element
Return type:: int | None

In case of corrupt input arguments or arguments not included in storage, None is returned

get_token(element_id)

Retrieve an element by unique identifier.

Parameters:: element_id (int) – Identifier to retrieve identifier for
Returns:: Element that corresponds to the given identifier
Return type:: str | None

In case of corrupt input arguments or arguments not included in storage, None is returned

class lab_3_generate_by_ngrams.main.WordProcessor(end_of_word_token)

Bases: TextProcessor

Handle text tokenization, encoding and decoding on the level of words.

Parameters:: end_of_word_token (str)

_end_of_word_token

A token denoting sentence boundary

Type:: str

_storage

Dictionary in the form of <token: identifier>

Type:: dict

_postprocess_decoded_text(decoded_corpus)

Convert decoded sequence of words into the string sequence.

Words are separated with spaces, special tokens are replaced with full stops. Each sentence is capitalized, resulting sequence must end with a full stop.

Parameters:: decoded_corpus (tuple[str, ...]) – A tuple of decoded tokens
Returns:: Resulting text
Return type:: str | None

In case of corrupt input arguments, None is returned. In case decoded corpus does not contain any words, None is returned.

_put(element)

Put a word into the storage, assign a unique id to it.

Parameters:: element (str) – A word to put into storage
Return type:: None

In case of corrupt input arguments, an element is not added to storage

_tokenize(text)

Tokenize text into words, separating sentences with special token.

Punctuation and digits are removed. Special token is appended after the word that ends a sentence, i.e. is followed by ‘.’, ‘!’ or ‘?’.

Parameters:: text (str) – Original text
Returns:: Tokenized text
Return type:: tuple[str, …] | None

In case of corrupt input arguments, None is returned. In case text does not contain any words, None is returned.