=wgT"ddlmZddlmZddlmZddlmZmZddlm Z m Z Gdde Z Gdd eZ d d Z dd Zy )) text_type)xrange)Token)FilterLowercaseFilter) TokenizerRegexTokenizerc>eZdZdZeeeZddZdZ ddZ y) NgramTokenizera7Splits input text into N-grams instead of words. >>> ngt = NgramTokenizer(4) >>> [token.text for token in ngt("hi there")] ["hi t", "i th", " the", "ther", "here"] Note that this tokenizer does NOT use a regular expression to extract words, so the grams emitted by it will contain whitespace, punctuation, etc. You may want to massage the input or add a custom filter to this tokenizer's output. Alternatively, if you only want sub-word grams without whitespace, you could combine a RegexTokenizer with NgramFilter instead. minsizemaxsizeNc(||_|xs||_y)z :param minsize: The minimum size of the N-grams. :param maxsize: The maximum size of the N-grams. If you omit this parameter, maxsize == minsize. N)minmax)selfr rs M/var/www/horilla/myenv/lib/python3.12/site-packages/whoosh/analysis/ngrams.py__init__zNgramTokenizer.__init__7s%gc|j|jur3|j|jk(r|j|jk(ryy)NTF __class__rrrothers r__eq__zNgramTokenizer.__eq__As7 >>U__ ,xx599$UYY)>rc +Kt|ts Jd|zt|} t||||} |} |dk(rt |j | } t d| | z dzD]Y}|| z}|| kDr|||| _|r| j| _d| _ |r| | _ |r||z| _ ||z| _ | | dz } [yt d| |jz dzD]}t |j|j dzD]T} || z}|| kDr|||| _|r| j| _d| _ |r| | _ |r||z| _ ||z| _ | V| dz } yw)Nz%r is not unicode) removestopsmodequeryrF) isinstancerlenrrrrtextoriginalstoppedpos startcharendchar)rvalue positionschars keeporiginalr start_pos start_charrkwargsinlentr&sizestartends r__call__zNgramTokenizer.__call__Gs%+H-@5-HH+E  )U $ G 7?txx'D54>> rext = RegexTokenizer() >>> stream = rext("hello there") >>> ngf = NgramFilter(4) >>> [token.text for token in ngf(stream)] ["hell", "ello", "ther", "here"] r Ncj||_|xs||_d|_|dk(rd|_y|dk(rd|_yy)a :param minsize: The minimum size of the N-grams. :param maxsize: The maximum size of the N-grams. If you omit this parameter, maxsize == minsize. :param at: If 'start', only take N-grams from the start of each word. if 'end', only take N-grams from the end of each word. Otherwise, take all N-grams from the word (the default). rr3r4r N)rrat)rr rrEs rrzNgramFilter.__init__s?%g =DG 5[DGrc|xrN|j|juxr4|j|jk(xr|j|jk(Sr6rrs rrzNgramFilter.__eq__sH<5??:< HH !<&*hh%))&; $A66D4y488#GGEKK vv 488S[18!%4[AF$-$4 G1W!!d()_AF&'ii$&6 G!'3t9t+;a+?!@ !%eEDL!9 *3e*;AK(1E(9D(@AI  8#d)4E &txx ; !%et (1D(8AI  1W-.[[*3t9txx#78E#E3t9txx+?!+CD !%ab * $sJJ)NNr8r@rrrBrBws%c2M$<A$rrBNc2t||tzS)zComposes an NgramTokenizer and a LowercaseFilter. >>> ana = NgramAnalyzer(4) >>> [token.text for token in ana("hi there")] ["hi t", "i th", " the", "ther", "here"] )r)r rr s r NgramAnalyzerrOs '7 3o6G GGrcR|s t}|tzt|||zS)N)rE)r rrB)r r tokenizerrEs rNgramWordAnalyzerrRs) "$ ( (;wB+O OOrr6)NNN) whoosh.compatrrwhoosh.analysis.acorerwhoosh.analysis.filtersrrwhoosh.analysis.tokenizersrr r rBrOrRr@rrrWs@8$ ';@ MYMdc$&c$PHPr