=wg NdZddlddlddlddlddlddlddly)aClasses and functions for turning a piece of text into an indexable stream of "tokens" (usually equivalent to words). There are three general classes involved in analysis: * Tokenizers are always at the start of the text processing pipeline. They take a string and yield Token objects (actually, the same token object over and over, for performance reasons) corresponding to the tokens (words) in the text. Every tokenizer is a callable that takes a string and returns an iterator of tokens. * Filters take the tokens from the tokenizer and perform various transformations on them. For example, the LowercaseFilter converts all tokens to lowercase, which is usually necessary when indexing regular English text. Every filter is a callable that takes a token generator and returns a token generator. * Analyzers are convenience functions/classes that "package up" a tokenizer and zero or more filters into a single unit. For example, the StandardAnalyzer combines a RegexTokenizer, LowercaseFilter, and StopFilter. Every analyzer is a callable that takes a string and returns a token iterator. (So Tokenizers can be used as Analyzers if you don't need any filtering). You can compose tokenizers and filters together using the ``|`` character:: my_analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter() The first item must be a tokenizer and the rest must be filters (you can't put a filter first or a tokenizer after the first item). )*N)__doc__whoosh.analysis.acorewhoosh.analysis.tokenizerswhoosh.analysis.filterswhoosh.analysis.morphwhoosh.analysis.intrawordwhoosh.analysis.ngramswhoosh.analysis.analyzersO/var/www/horilla/myenv/lib/python3.12/site-packages/whoosh/analysis/__init__.pyrs"8!F$(%#'$'r