lab_1_classify_profile package

Submodules

Lab 1.

Language detection

lab_1_classify_profile.main.FreqDictType

Frequency dictionary. Contains pairs of token and its frequency.

alias of dict[str, float]

lab_1_classify_profile.main.ProfileType

Language profile of a text. Contains language name, frequency dictionary and number of tokens.

alias of tuple[str, dict[str, float], int]

lab_1_classify_profile.main.calculate_frequencies(tokens)

Calculates frequencies of given tokens

Parameters:

tokens (Sequence[str]) – Sequence of tokens

Returns:

Dictionary with frequencies. Returns None in case of incorrect input types.

Return type:

dict[str, float] | None

lab_1_classify_profile.main.calculate_mse(predicted, actual)

Calculates mean squared error between predicted and actual values.

Parameters:

  • predicted (Sequence[float]) – Sequence of predicted values

  • actual (Sequence[float]) – Sequence of actual values

Returns:

The score Returns None in case of incorrect input types or mismatched length. In case of empty inputs, returns 0.0.

Return type:

float | None

lab_1_classify_profile.main.check_profile(profile)

Checks profile structure

Parameters:

profile (ProfileType) – Profile to check

Returns:

Returns True if the profile has right structure and types, otherwise returns False.

Return type:

bool

lab_1_classify_profile.main.collect_profiles(paths_to_profiles)

Collects profiles for a given path.

Parameters:

paths_to_profiles (Sequence[str]) – Sequence of paths to the profiles

Returns:

Sequence of loaded profiles. Returns None in case of incorrect input types.

Return type:

Sequence[ProfileType] | None

lab_1_classify_profile.main.compare_profiles_by_mse(unknown_profile, profile_to_compare)

Compares two language profiles using the MSE metric.

Parameters:

  • unknown_profile (ProfileType) – Unknown profile

  • profile_to_compare (ProfileType) – Profile to compare the unknown profile with

Returns:

The distance between the profiles. In case of corrupt input arguments or invalid profile structure, None is returned.

Return type:

float | None

lab_1_classify_profile.main.compare_profiles_by_top_n(unknown_profile, profile_to_compare, top_n)

Compares profiles and calculates the distance using top n words

Parameters:

  • unknown_profile (ProfileType) – Unknown profile

  • profile_to_compare (ProfileType) – Profile of a known language

  • top_n (int) – Number of the most common words

Returns:

The distance between profiles. Returns None in case of incorrect input types.

Return type:

float | None

lab_1_classify_profile.main.create_language_profile(language, text, stop_words)

Creates a language profile

Parameters:

  • language (str) – Language name

  • text (str) – Text

  • stop_words (Sequence[str]) – Sequence of stop words (can be empty)

Returns:

Language profile. Returns None in case of incorrect input types.

Return type:

ProfileType | None

lab_1_classify_profile.main.detect_language_advanced(unknown_profile, known_profiles, top_n)

Detects the language of an unknown profile.

Parameters:

  • unknown_profile (ProfileType) – Profile to determine the language of

  • known_profiles (Sequence[ProfileType]) – Known profiles

  • top_n (int) – Number of popular words

Returns:

Sorted sequence of tuples containing a language and a distance via both metrics. The sequence is sorted by best MSE value, then by best Top-N value. Returns None in case of incorrect input types.

Return type:

Sequence[tuple[str, dict[str, float]]] | None

lab_1_classify_profile.main.detect_language_by_mse(unknown_profile, profile_1, profile_2)

Detects the language of an unknown profile.

Parameters:

  • unknown_profile (ProfileType) – Profile to determine the language of

  • profile_1 (ProfileType) – Known profile

  • profile_2 (ProfileType) – Another known profile

Returns:

Unknown profile language. Returns None in case of incorrect input types.

Return type:

str | None

lab_1_classify_profile.main.detect_language_by_top_n(unknown_profile, profile_1, profile_2, top_n)

Detects the language of an unknown profile

Parameters:

  • unknown_profile (ProfileType) – Unknown profile

  • profile_1 (ProfileType) – Profile for comparison

  • profile_2 (ProfileType) – Another profile for comparison

  • top_n (int) – Number of the most common words

Returns:

Unknown profile language. Returns None in case of incorrect input types.

Return type:

str | None

lab_1_classify_profile.main.get_top_n_words(freq_dict, top_n)

Finds the most common words

Parameters:

  • freq_dict (dict[str, float]) – Dictionary with frequencies

  • top_n (int) – Number of the most common words

Returns:

Sequence of the most common words. Returns None in case of incorrect input types or non-positive top_n.

Return type:

Sequence[str] | None

lab_1_classify_profile.main.load_profile(path_to_file)

Loads a language profile.

Parameters:

path_to_file (str) – Path to the language profile

Returns:

Loaded profile. Returns None in case of incorrect input types.

Return type:

ProfileType | None

lab_1_classify_profile.main.print_report(unknown_profile, metrics_stats, top_n)

Prints report for detection of language.

Parameters:

  • unknown_profile (ProfileType) – Profile

  • metrics_stats (Sequence[tuple[str, dict[str, float]]]) – Sequence with distances for available language comparison and metrics

  • top_n (int) – Number of popular words

Return type:

None

In case of incorrect type inputs, does not print anything.

lab_1_classify_profile.main.remove_stop_words(tokens, stop_words)

Removes stop words

Parameters:

  • tokens (Sequence[str]) – Sequence of tokens

  • stop_words (Sequence[str]) – Sequence of stop words (can be empty)

Returns:

Sequence of tokens without stop words. Returns None in case of incorrect input types.

Return type:

Sequence[str] | None

lab_1_classify_profile.main.save_profile(profile, save_path)

Saves a language profile

Parameters:

  • profile (ProfileType) – Profile

  • save_path (str) – Path to the folder to save profile

Returns:

False in case of incorrect input types or if the profile is missing obligatory keys. True if the profile is saved.

Return type:

bool

lab_1_classify_profile.main.tokenize(text)

Splits a text into tokens, converts the tokens into lowercase, removes punctuation and other symbols from words

Parameters:

text (str) – Text

Returns:

Sequence of lower-cased tokens without punctuation. Returns None if input text is not a string.

Return type:

Sequence[str] | None