=wg , >ddlmZmZddlmZddlmZddlmZmZddl m Z ddl m Z ddlm Z ddlmZdd lmZdd lmZdd lmZdd lmZGd deZGddeZddZd dZd!dZe dfdZe edddfdZe edddeddfdZdedddddddf dZe ddfdZy)") ComposableCompositionError) Tokenizer)LowercaseFilter) StopFilter STOP_WORDS) StemFilter)IntraWordFilter)default_pattern)CommaSeparatedTokenizer) IDTokenizerRegexTokenizer)SpaceSeparatedTokenizer)stemc(eZdZdZdZdZdZdZy)Analyzerz( Abstract base class for analyzers. c4d|jjzS)Nz%s()) __class____name__selfs P/var/www/horilla/myenv/lib/python3.12/site-packages/whoosh/analysis/analyzers.py__repr__zAnalyzer.__repr__0s////cp|xr3|j|juxr|j|jk(SN)r__dict__rothers r__eq__zAnalyzer.__eq__3s44NNeoo54MMU^^3 5rc tr)NotImplementedError)rvaluekwargss r__call__zAnalyzer.__call__8s!!rcyrrs rcleanzAnalyzer.clean;s rN)r __module__ __qualname____doc__rr!r&r)r(rrrr,s05 " rrc>eZdZdZdZd dZdZdZdZdZ dZ y ) CompositeAnalyzerc:g|_|D]S}t|tr&|jj|j9|jj |U|jddD]*}t|t st d|jzy)Nz;Only one tokenizer allowed at the start of the analyzer: %r)items isinstancer.extendappendrr)r composablescompitems r__init__zCompositeAnalyzer.__init__@s  (D$ 12 !!$**- !!$'  (JJqrN LD$ *&(>@D (KLL Lrcz|jjddjd|jDdS)N(z, c32K|]}t|ywr)repr.0r7s r z-CompositeAnalyzer.__repr__..Ss$GDT$Z$Gs))rrjoinr1rs rrzCompositeAnalyzer.__repr__Qs1>>2299$GDJJ$GGI Irc |j}|d|fi|}|ddD]%}|rt|dr |jr||}'|S)Nrr0is_morph)r1hasattrrC)rr$no_morphr%r1genr7s rr&zCompositeAnalyzer.__call__UsW eAhu''!"I Dz!:t}}3i  rc8|jj|Sr)r1 __getitem__rr7s rrHzCompositeAnalyzer.__getitem___szz%%d++rc,t|jSr)lenr1rs r__len__zCompositeAnalyzer.__len__bs4::rcp|xr3|j|juxr|j|jk(Sr)rr1rs rr!zCompositeAnalyzer.__eq__es4.NNeoo5.JJ%++- /rc`|jD]}t|ds|j!y)Nr))r1rDr)rIs rr)zCompositeAnalyzer.cleanjs(JJ DtW%  rc:td|jDS)Nc34K|]}|jywr)rCr=s rr?z.CompositeAnalyzer.has_morph..ps8T4==8s)anyr1rs r has_morphzCompositeAnalyzer.has_morphos8TZZ888rNF) rr*r+r8rr&rHrLr!r)rRr(rrr.r.?s,L"I,/  9rr.Fc8t}|r |tz}|S)zYDeprecated, just use an IDTokenizer directly, with a LowercaseFilter if desired. )r r) lowercase tokenizers r IDAnalyzerrWus!  I 11 rcR|r t}n t}|r |tz}|S)a]Parses whitespace- or comma-separated tokens. >>> ana = KeywordAnalyzer() >>> [token.text for token in ana("Hello there, this is a TEST")] ["Hello", "there,", "this", "is", "a", "TEST"] :param lowercase: whether to lowercase the tokens. :param commas: if True, items are separated by commas rather than whitespace. )r rr)rUcommasrVs rKeywordAnalyzerrZs-+- +-  11 rct||S)z4Deprecated, just use a RegexTokenizer directly.  expressiongapsrr\s r RegexAnalyzerr_s Zd ;;rc2t||tzS)aComposes a RegexTokenizer with a LowercaseFilter. >>> ana = SimpleAnalyzer() >>> [token.text for token in ana("Hello there, this is a TEST")] ["hello", "there", "this", "is", "a", "test"] :param expression: The regular expression pattern to use to extract tokens. :param gaps: If True, the tokenizer *splits* on the expression, rather than matching on the expression. r\)rrr\s rSimpleAnalyzerras Zd ;o>O OOrNc`t||}|tz}||t|||z}|S)aComposes a RegexTokenizer with a LowercaseFilter and optional StopFilter. >>> ana = StandardAnalyzer() >>> [token.text for token in ana("Testing is testing and testing")] ["testing", "testing", "testing"] :param expression: The regular expression pattern to use to extract tokens. :param stoplist: A list of stop words. Set this to None to disable the stop word filter. :param minsize: Words smaller than this are removed from the stream. :param maxsize: Words longer that this are removed from the stream. :param gaps: If True, the tokenizer *splits* on the expression, rather than matching on the expression. r\stoplistminsizemaxsize)rrr)r]rerfrgr^retchains rStandardAnalyzerrjs@$ JT :C /# #E Hg+244 LriPc~t||}|tz} || t|||z} | t|||zS)aComposes a RegexTokenizer with a lower case filter, an optional stop filter, and a stemming filter. >>> ana = StemmingAnalyzer() >>> [token.text for token in ana("Testing is testing and testing")] ["test", "test", "test"] :param expression: The regular expression pattern to use to extract tokens. :param stoplist: A list of stop words. Set this to None to disable the stop word filter. :param minsize: Words smaller than this are removed from the stream. :param maxsize: Words longer that this are removed from the stream. :param gaps: If True, the tokenizer *splits* on the expression, rather than matching on the expression. :param ignore: a set of words to not stem. :param cachesize: the maximum number of stemmed words to cache. The larger this number, the faster stemming will be but the more memory it will use. Use None for no cache, or -1 for an unbounded cache. r\rd)stemfnignore cachesize)rrrr ) r]rerfrgr^rlrmrnrhris rStemmingAnalyzerrosV. JT :C /# #E Hg+244 :VF(13 33rz\s+Tc nt||t||||ztzt||zS)aComposes a RegexTokenizer with an IntraWordFilter, LowercaseFilter, and StopFilter. >>> ana = FancyAnalyzer() >>> [token.text for token in ana("Should I call getInt or get_real?")] ["should", "call", "getInt", "get", "int", "get_real", "get", "real"] :param expression: The regular expression pattern to use to extract tokens. :param stoplist: A list of stop words. Set this to None to disable the stop word filter. :param minsize: Words smaller than this are removed from the stream. :param maxsize: Words longer that this are removed from the stream. :param gaps: If True, the tokenizer *splits* on the expression, rather than matching on the expression. r\) splitwords splitnums mergewords mergenums)rerf)rr rr) r]rerfrgr^rqrrrsrts r FancyAnalyzerrusI& jt <y)3yJ J (G<  =rcddlm}m}t||t z} |t |z} |t ||z}|S#|$rYwxYw#|$rY|SwxYw)aaConfigures a simple analyzer for the given language, with a LowercaseFilter, StopFilter, and StemFilter. >>> ana = LanguageAnalyzer("es") >>> [token.text for token in ana("Por el mar corren las liebres")] ['mar', 'corr', 'liebr'] The list of available languages is in `whoosh.lang.languages`. You can use :func:`whoosh.lang.has_stemmer` and :func:`whoosh.lang.has_stopwords` to check if a given language has a stemming function and/or stop word list available. :param expression: The regular expression pattern to use to extract tokens. :param gaps: If True, the tokenizer *splits* on the expression, rather than matching on the expression. :param cachesize: the maximum number of stemmed words to cache. The larger this number, the faster stemming will be but the more memory it will use. r) NoStemmer NoStopWordsr\)lang)ryrn) whoosh.langrwrxrrrr )ryr]r^rnrwrxris rLanguageAnalyzerr{s},3z = !E  --    BB L      L s"AAA  A AArS)FF)z \w+(\.?\w+)*F)whoosh.analysis.acorerrwhoosh.analysis.tokenizersrwhoosh.analysis.filtersrrrwhoosh.analysis.morphr whoosh.analysis.intrawordr r r r rrwhoosh.lang.porterrrr.rWrZr_rarjrorur{r(rrrs8?03:,56>25>#  z &1919l*<.E P!0*54!0*5 E3@$j!Tdd"e6'6E$(r