=wg'ddlmZddlmZddlmZddlmZddlm Z m Z GddeZ Gdd e Z Gd d eZ y ) )Filter) integer_types)double_metaphone)stem) lfu_cache unbound_cachec\eZdZdZeeeZdZe dddfdZ dZ dZ d Z d Zd Zd Zy) StemFilteraHStems (removes suffixes from) the text of tokens using the Porter stemming algorithm. Stemming attempts to reduce multiple forms of the same root word (for example, "rendering", "renders", "rendered", etc.) to a single word in the index. >>> stemmer = RegexTokenizer() | StemFilter() >>> [token.text for token in stemmer("fundamentally willows")] ["fundament", "willow"] You can pass your own stemming function to the StemFilter. The default is the Porter stemming algorithm for English. >>> stemfilter = StemFilter(stem_function) You can also use one of the Snowball stemming functions by passing the `lang` keyword argument. >>> stemfilter = StemFilter(lang="ru") The list of available languages is in `whoosh.lang.languages`. You can use :func:`whoosh.lang.has_stemmer` to check if a given language has a stemming function available. By default, this class wraps an LRU cache around the stemming function. The ``cachesize`` keyword argument sets the size of the cache. To make the cache unbounded (the class caches every input), use ``cachesize=-1``. To disable caching, use ``cachesize=None``. If you compile and install the py-stemmer library, the :class:`PyStemmerFilter` provides slightly easier access to the language stemmers in that library. )stemfnignoreTNPc||_||_| tn t||_||_|j y)a  :param stemfn: the function to use for stemming. :param lang: if not None, overrides the stemfn with a language stemmer from the ``whoosh.lang.snowball`` package. :param ignore: a set/list of words that should not be stemmed. This is converted into a frozenset. If you omit this argument, all tokens are stemmed. :param cachesize: the maximum number of words to cache. Use ``-1`` for an unbounded cache, or ``None`` for no caching. N)r lang frozensetr cachesizeclear)selfr rr rs L/var/www/horilla/myenv/lib/python3.12/site-packages/whoosh/analysis/morph.py__init__zStemFilter.__init__Is6  %+^ik69J " czt|jDcgc]}|dk7r||j|fc}Scc}wN_stemdict__dict__rks r __getstate__zStemFilter.__getstate__\sCDMM'qgq)*'( ('8cd|vrd|_d|vr |d|_nd|vrt|_d|vrd|_d|vr|d=|jj ||j y)Nrr ignoresr rcache)rr rrrupdaterrstates r __setstate__zStemFilter.__setstate__bsq e #"DN   *DK U "#+DK  DI e g U# rch|jrddlm}||j}n |j}t |j t r`|j dk7rQ|j dkrt||_y|j dkDr!t|j ||_yy||_y)Nr)stemmer_for_language) r whoosh.langr)r isinstancerrrrr)rr)r s rrzStemFilter.clearts 99 8)$))4F[[F dnnm 419L~~!*62 !#6Yt~~6v> $ DJrcV|jdkry|jjS)Nr*)rr cache_infors rr.zStemFilter.cache_infos$ >>Q zz$$&&rcp|xr3|j|juxr|j|jk(SN) __class__r rothers r__eq__zStemFilter.__eq__s30$..EOO;0KK5<</ 1rc#K|j}|j}|D]/}|js|j}||vr |||_|1ywr1)rr stoppedtext)rtokensr r tr8s r__call__zStemFilter.__call__sO A99vvv%#D\AFG  sAA)__name__ __module__ __qualname____doc__robjectlist __inittypes__is_morphrrrr'rr.r5r;rrr r #sGBt4MH"de&( $  ' 1 rr c6eZdZdZd dZdZdZdZdZdZ y) PyStemmerFilterzThis is a simple subclass of StemFilter that works with the py-stemmer third-party library. You must have the py-stemmer library installed to use this filter. >>> PyStemmerFilter("spanish") Nc||_| tn t||_||_|j |_y)a :param lang: a string identifying the stemming algorithm to use. You can get a list of available algorithms by with the :meth:`PyStemmerFilter.algorithms` method. The identification strings are directly from the py-stemmer library. :param ignore: a set/list of words that should not be stemmed. This is converted into a frozenset. If you omit this argument, all tokens are stemmed. :param cachesize: the maximum number of words to cache. N)rrr r_get_stemmer_fnr)rrr rs rrzPyStemmerFilter.__init__s6 %+^ik69J "))+ rc*ddl}|jS)zZReturns a list of stemming algorithms provided by the py-stemmer library. rN)Stemmer algorithms)rrJs rrKzPyStemmerFilter.algorithmss !!##rcyr1rDr/s rr.zPyStemmerFilter.cache_infosrczddl}|j|j}|j|_|jS)Nr)rJrr maxCacheSizestemWord)rrJstemmers rrHzPyStemmerFilter._get_stemmer_fns///$)),#~~rczt|jDcgc]}|dk7r||j|fc}Scc}wrrrs rrzPyStemmerFilter.__getstate__sCDMM&qW q)*&' '&r cd|vrd|_d|vr |d|_nd|vrt|_d|vr|d=|jj ||j |_y)Nr'r"r r#)rr rrr$rHrr%s rr'zPyStemmerFilter.__setstate__si e #"DN   *DK U "#+DK e g U#))+ r)englishNrS) r<r=r>r?rrKr.rHrr'rDrrrFrFs%,"$ ' ,rrFc(eZdZdZdZddZdZdZy)DoubleMetaphoneFilteraeTransforms the text of the tokens using Lawrence Philips's Double Metaphone algorithm. This algorithm attempts to encode words in such a way that similar-sounding words reduce to the same code. This may be useful for fields containing the names of people and places, and other uses where tolerance of spelling differences is desireable. Tc.||_||_||_y)ad :param primary_boost: the boost to apply to the token containing the primary code. :param secondary_boost: the boost to apply to the token containing the secondary code, if any. :param combine: if True, the original unencoded tokens are kept in the stream, preceding the encoded tokens. N) primary_boostsecondary_boostcombine)rrXrYrZs rrzDoubleMetaphoneFilter.__init__s+. rcp|xr3|j|juxr|j|jk(Sr1)r2rXr3s rr5zDoubleMetaphoneFilter.__eq__s8>NNeoo5>&&%*=*== ?rc#K|j}|j}|j}|D][}|r|t|j\}}|j }|r||_||z|_||sG||_||z|_|]ywr1)rXrYrZrr8boost) rr9rXrYrZr:primary secondarybs rr;zDoubleMetaphoneFilter.__call__s** ..,, A!1!&&!9 GYA m+"o- s A-B0BN)g?g?F)r<r=r>r?rCrr5r;rDrrrVrVsH ? rrVN)whoosh.analysis.filtersr whoosh.compatrwhoosh.lang.dmetaphonerwhoosh.lang.porterrwhoosh.util.cacherrr rFrVrDrrrfs@8+'3#6rrj@,j@,F0F0r