core_utils package

Submodules

ConfigDTO class implementation: stores the configuration information.

class core_utils.ctlr.config_dto.ConfigDTO(seed_urls, total_articles_to_find_and_parse, headers, encoding, timeout, should_verify_certificate, headless_mode)

Bases: object

Type annotations for configurations.

Parameters:

  • seed_urls (list[str]) –

  • total_articles_to_find_and_parse (int) –

  • headers (dict[str, str]) –

  • encoding (str) –

  • timeout (int) –

  • should_verify_certificate (bool) –

  • headless_mode (bool) –

__init__(seed_urls, total_articles_to_find_and_parse, headers, encoding, timeout, should_verify_certificate, headless_mode)

Initializes an instance of the ConfigDTO class.

Parameters:

  • seed_urls (list[str]) – Seed urls

  • total_articles_to_find_and_parse (int) – Number of total articles

  • headers (dict[str, str]) – Headers

  • encoding (str) – Encoding

  • timeout (int) – Number of seconds to wait for response

  • should_verify_certificate (bool) – Should verify certificate or not

  • headless_mode (bool) – Require headless mode or not

Return type:

None

encoding: str

Encoding

headers: dict[str, str]

Headers

headless_mode: bool

Require headless mode or not

seed_urls: list[str]

List of seed urls

should_verify_certificate: bool

Should verify certificate or not

timeout: int

Number of seconds to wait for response

total_articles: int

Number of total articles

Interface definitions for text processing pipelines.

class core_utils.ctlr.pipeline.LibraryWrapper(*args, **kwargs)

Bases: Protocol

Interface definition for text analyzers.

__init__(*args, **kwargs)
_abc_impl = <_abc._abc_data object>
_analyzer: None
_bootstrap()

Bootstrap analyzer with required models and settings.

Returns:

Instance of analyzer.

Return type:

Language

_is_protocol = True
analyze(texts)

Analyze given texts.

Parameters:

texts (list[str]) – Texts to analyze.

Returns:

Collection of processed documents.

Return type:

list[str]

from_conllu(article)

Load ConLLU content from article stored on disk.

Parameters:

article (Article) – Article to load

Returns:

Document ready for parsing

Return type:

Doc

to_conllu(article)

Write ConLLU content to a file.

Parameters:

article (Article) – Article to save

Return type:

None

class core_utils.ctlr.pipeline.PipelineProtocol(*args, **kwargs)

Bases: Protocol

Interface definition for pipeline.

__init__(*args, **kwargs)
_abc_impl = <_abc._abc_data object>
_is_protocol = True
run()

Key API method.

Return type:

None

class core_utils.ctlr.pipeline.TreeNode(upos, text, children)

Bases: object

Interface definition for node in the graph.

Parameters:
__init__(upos, text, children)
Parameters:
Return type:

None

children: list[TreeNode]
text: str
upos: str

Visualizer module for visualizing PosFrequencyPipeline results.

core_utils.ctlr.visualizer.show_graph(graph, graph_path)

Visualization for debug.

Parameters:

  • graph (DiGraph) – Graph to check

  • graph_path (str) – Path to graph

Return type:

None

core_utils.ctlr.visualizer.visualize(article, path_to_save)

Visualize PosFrequencyPipeline results.

Parameters:
Return type:

None