=wgdZddlmZddlmZddlmZddlmZddl m Z ddl m Z dZ d2d ZGd d eZd ZGddeZGddeZeZGddeZGddeZGddeZGddeZGddeZdZdZdZdZd ZGd!d"eZGd#d$eZ Gd%d&eZ!Gd'd(eZ"Gd)d*eZ#d3d,Z$d-d d+ed.fd/Z%Gd0d1eZ&y )4aRThe highlight module contains classes and functions for displaying short excerpts from hit documents in the search results you present to the user, with query terms highlighted. The highlighting system has four main elements. * **Fragmenters** chop up the original text into __fragments__, based on the locations of matched terms in the text. * **Scorers** assign a score to each fragment, allowing the system to rank the best fragments by whatever criterion. * **Order functions** control in what order the top-scoring fragments are presented to the user. For example, you can show the fragments in the order they appear in the document (FIRST) or show higher-scoring fragments first (SCORE) * **Formatters** turn the fragment objects into human-readable output, such as an HTML string. See :doc:`/highlight` for more information. )division)deque)nlargest)groupby htmlescape)TokeniNc||r|djnd}||r|djn t|}td||z }t t|||z}t ||||S)zfReturns a :class:`Fragment` object based on the :class:`analysis.Token` objects in ``tokens`. r) startcharendcharlenmaxminFragment)texttokensr r charsbefore charsafters G/var/www/horilla/myenv/lib/python3.12/site-packages/whoosh/highlight.pymkfragrBsp +1F1I''q (.&*$$CIAy;./I#d)Wz12G D&)W 55c6eZdZdZd dZdZdZdZdZdZ y) raXRepresents a fragment (extract) from a hit document. This object is mainly used to keep track of the start and end points of the fragment and the "matched" character ranges inside; it does not contain the text of the fragment or do much else. The useful attributes are: ``Fragment.text`` The entire original text from which this fragment is taken. ``Fragment.matches`` An ordered list of objects representing the matched terms in the fragment. These objects have ``startchar`` and ``endchar`` attributes. ``Fragment.startchar`` The index of the first character in the fragment. ``Fragment.endchar`` The index of the last character in the fragment. ``Fragment.matched_terms`` A ``set`` of the ``text`` of the matched terms in the fragment (if available). c||_||_|dk(r t|}||_||_t |_|D]4}t|ds|j j|j6y)a :param text: the source text of the fragment. :param matches: a list of objects which have ``startchar`` and ``endchar`` attributes, and optionally a ``text`` attribute. :param startchar: the index into ``text`` at which the fragment starts. The default is 0. :param endchar: the index into ``text`` at which the fragment ends. The default is -1, which is interpreted as the length of ``text``. r rN) rmatchesrr r set matched_termshasattradd)selfrrr r ts r__init__zFragment.__init__msk  b=$iG"  U /Aq&!""&&qvv. /rc`d|j|jt|jfzS)Nz)r r rrr s r__repr__zFragment.__repr__s,$ (+DLL(9(;; ;rc4|j|jz SNr r r$s r__len__zFragment.__len__s||dnn,,rc|j}|j}|j}|j}||cxkxr|kncxs||cxkxr|kScSr'r r r fragmentscecfscfecs roverlapszFragment.overlapssJ ^^ \\  S 2 12=b=1=1rc|j}|j}|j}|j}t||t||z Sr')r r rrr,s roverlapped_lengthzFragment.overlapped_lengthsA ^^ \\  2s|c"cl**rc0t|t|kSr')id)r others r__lt__zFragment.__lt__s$x"U)##rN)rr ) __name__ __module__ __qualname____doc__r"r%r)r2r4r8rrrrSs%2/0;-2+$rrc#HK|D]}|j|v|_|ywr')rmatched)rtermsetr!s rset_matched_filterrAs* FFg% s "ceZdZdZdZdZy) Fragmentercy)aReturns True if this fragmenter requires retokenized text. If this method returns True, the fragmenter's ``fragment_tokens`` method will be called with an iterator of ALL tokens from the text, with the tokens for matched terms having the ``matched`` attribute set to True. If this method returns False, the fragmenter's ``fragment_matches`` method will be called with a LIST of matching tokens. Tr=r$s rmust_retokenizezFragmenter.must_retokenizesrct)zYields :class:`Fragment` objects based on the tokenized text. :param text: the string being highlighted. :param all_tokens: an iterator of :class:`analysis.Token` objects from the string. NotImplementedError)r r all_tokenss rfragment_tokenszFragmenter.fragment_tokenss "!rct)aYields :class:`Fragment` objects based on the text and the matched terms. :param text: the string being highlighted. :param matched_tokens: a list of :class:`analysis.Token` objects representing the term matches in the string. rG)r rmatched_tokenss rfragment_matcheszFragmenter.fragment_matchess "!rN)r9r:r;rErJrMr=rrrCrCs " "rrCc eZdZdZefdZdZy)WholeFragmenteraDoesn't fragment the token stream. This object just returns the entire entire stream as one "fragment". This is useful if you want to highlight the entire text. Note that even if you use the `WholeFragmenter`, the highlight code will return no fragment if no terms matched in the given field. To return the whole fragment even in that case, call `highlights()` with `minscore=0`:: # Query where no terms match in the "text" field q = query.Term("tag", "new") r = mysearcher.search(q) r.fragmenter = highlight.WholeFragmenter() r.formatter = highlight.UppercaseFormatter() # Since no terms in the "text" field matched, we get no fragments back assert r[0].highlights("text") == "" # If we lower the minimum score to 0, we get a fragment even though it # has no matching terms assert r[0].highlights("text", minscore=0) == "This is the text field." c||_yr') charlimit)r rQs rr"zWholeFragmenter.__init__s "rc|j}g}|D]A}|r|j|kDrn.|js#|j|j Ct ||gSr')rQr r?appendcopyr)r rrrQrr!s rrJzWholeFragmenter.fragment_tokenss[NN  )AQYY2yyqvvx(  ) w'((rNr9r:r;r<DEFAULT_CHARLIMITr"rJr=rrrOrOs."3#)rrOc$eZdZdZddefdZdZy)SentenceFragmentera|Breaks the text up on sentence end punctuation characters (".", "!", or "?"). This object works by looking in the original text for a sentence end as the next character after each token's 'endchar'. When highlighting with this fragmenter, you should use an analyzer that does NOT remove stop words, for example:: sa = StandardAnalyzer(stoplist=None) z.!?c@||_t||_||_y)zf :param maxchars: The maximum number of characters allowed in a fragment. N)maxchars frozenset sentencecharsrQ)r r[r]rQs rr"zSentenceFragmenter.__init__s! &}5"rc#K|j}|j}|j}t|}d}g}d} d} |D]} | j} | j } |r| |kDrn||| }d} | | z } | | z } | j r|j| j| |kse|| |vsm| dz|kr || dz|vr|r| |krt|||| g}d}d} |rt|||| yyw)Nrr+) r[r]rQrr r r?rSrTr)r rrr[r]rQtextlenfirsttksr currentlenr!r tlengths rrJz"SentenceFragmenter.fragment_tokens s&==** NN d)  A IiiGWy0}!  )G ' !Jyy 1668$ T']m%CQ;(T'A+->--O:1 seWMM ? F seWE E sBC1!C1)AC1NrUr=rrrXrXs!$5, #2FrrXc$eZdZdZddefdZdZy)ContextFragmenterzTLooks for matched terms and aggregates them with their surrounding context. rYc.||_||_||_y)a :param maxchars: The maximum number of characters allowed in a fragment. :param surround: The number of extra characters of context to add both before the first matched term and after the last matched term. N)r[surroundrQ)r r[rirQs rr"zContextFragmenter.__init__Gs!   "rc#K|j}|j}|j}d}t}d}g} d} d} |D]} | j} | j } | | z }|r| |kDrn|dkrI| j s=|j| |rs| |dz |kDrh|j|rV| |dz |kDrnI| |z|kDrd}n>| j r2|}||r|d}n| }||z }| j| j|dk\s| |z } ||z}|dkst|| || g} t}d}d} | rt|| || yyw)Nr rr+) r[rirQrr r r?rSpopleftrTr)r rrr[rirQrafirsts countdownrbr rcr!r rds rrJz!ContextFragmenter.fragment_tokensSs====NN   - #A IiiG )GWy01}QYY i(6!9!4x!?NN$6!9!4x!?g%0 $ = &q  )!X-  1668$A~g% W$ > seWMMC"WF E!"J[- #` seWE E sB0E 3AE E 8E NrUr=rrrfrfBs!$b"QIJJsIw7 q=G   Ix'7'7':'D'DEI'8#3#3B#7#?#?@G&"rc#nK|j}|j}|j}|j}d}t |D]\}} ||k\r |}| j } | j } |r| |kDry| | z } |t|dz kr[| |krV||dz} | j }|| z |kr"|| z |kr|dz }|} | || j z z } nn|t|dz kr| |krVtd| |z } tt|| |z} t||||dz| | }|r|j||yw)Nr r_r) r[rirqrQ enumerater r rrrrrz)r rrr[rirqrQjir!leftrightrcnextr/r-s rrMz#PinpointFragmenter.fragment_matchessT======NN f% DAqAvA;;DIIEUY.Jc&kAo%*x*?a!e}\\:)b4i8.CFAE2#67Jc&kAo%*x*?q$/*DD 58#34EfQq1uotUCHx(N3 s CD5 AD5N) r9r:r;r<rVr"rErJ staticmethodrzrMr=rrroros;!$b5,#$4##&!rroc eZdZy)FragmentScorerN)r9r:r;r=rrrrsrrceZdZdZy)BasicFragmentScorerc|td|jD}|t|jdzxsdz}|S)Nc34K|]}|jywr')boost).0r!s r z/BasicFragmentScorer.__call__..s/AGG/sdr_)sumrrr)r fscores r__call__zBasicFragmentScorer.__call__s:/QYY// #aoo&,22 rN)r9r:r;rr=rrrrsrrcy)z#Sorts higher scored passages first.r_r=r-s rSCORErs rc|jS)z2Sorts passages from earlier in the document first.r rs rFIRSTr s   rcdt|z S)zSorts longer passages first.rrrs rLONGERrs s8} rct|S)zSort shorter passages first.rrs rSHORTERrs x=rcP|r |jS||j|jS)aConvenience function for getting the text to use for a match when formatting. If ``replace`` is False, returns the part of ``original`` between ``token.startchar`` and ``token.endchar``. If ``replace`` is True, returns ``token.text``. )rr r )originaltokenreplaces rget_textrs&zz 66rc8eZdZdZdZdZd dZd dZd dZdZ y) FormatteraBase class for formatters. For highlighters that return strings, it is usually only necessary to override :meth:`Formatter.format_token`. Use the :func:`get_text` function as a convenience to get the token text:: class MyFormatter(Formatter): def format_token(text, token, replace=False): ttext = get_text(text, token, replace) return "[%s]" % ttext ...c|Sr'r=r rs r_textzFormatter._text9s rct)amReturns a formatted version of the given "token" object, which should have at least ``startchar`` and ``endchar`` attributes, and a ``text`` attribute if ``replace`` is True. :param text: the original fragment text being highlighted. :param token: an object having ``startchar`` and ``endchar`` attributes and optionally a ``text`` attribute (if ``replace`` is True). :param replace: if True, the original text between the token's ``startchar`` and ``endchar`` indices will be replaced with the value of the token's ``text`` attribute. rGr rrrs r format_tokenzFormatter.format_token<s "!rcg}|j}|j}|jD]}|j|j|kr |j|kDr-|j|j |||j|j|j ||||j }|j|j |||j dj|}|S)aReturns a formatted version of the given text, using the "token" objects in the given :class:`Fragment`. :param fragment: a :class:`Fragment` object representing a list of matches in the text. :param replace: if True, the original text corresponding to each match will be replaced with the value of the token object's ``text`` attribute. )r rrrSrrr join)r r-routputindexrr! out_strings rformat_fragmentzFormatter.format_fragmentKs""}}!! A{{"{{U"{{U" djjeAKK)@AB MM$++D!W= >IIE   djjeH,<,?WWV_ rc|Dcgc]}|j||}}|jj|Scc}w)zjReturns a formatted version of the given text, using a list of :class:`Fragment` objects. r)rbetweenr)r fragmentsrr formatteds rformatzFormatter.formathsK ()))!W)=) )||  ++)s<c$|j|Sr')r)r rrs rrzFormatter.__call__qs{{9%%rNF) r9r:r;r<rrrrrrr=rrrr)s' G ":,&rrceZdZdZddZy) NullFormatterz/Formatter that does not modify the string. ct|||Sr')rrs rrzNullFormatter.format_tokenzseW--rNr)r9r:r;r<rr=rrrrvs .rrc eZdZdZddZddZy)UppercaseFormatterzBReturns a string in which the matched terms are in UPPERCASE. c||_y)zD :param between: the text to add between fragments. N)r)r rs rr"zUppercaseFormatter.__init__s  rc<t|||}|jSr')rupper)r rrrttxts rrzUppercaseFormatter.format_tokenseW-zz|rN)rr)r9r:r;r<r"rr=rrrr~srrc6eZdZdZdZ ddZdZd dZdZy) HtmlFormattera:Returns a string containing HTML formatting around the matched terms. This formatter wraps matched terms in an HTML element with two class names. The first class name (set with the constructor argument ``classname``) is the same for each match. The second class name (set with the constructor argument ``termclass`` is different depending on which term matched. This allows you to give different formatting (for example, different background colors) to the different terms in the excerpt. >>> hf = HtmlFormatter(tagname="span", classname="match", termclass="term") >>> hf(mytext, myfragments) "The template geometry is..." This object maintains a dictionary mapping terms to HTML class names (e.g. ``term0`` and ``term1`` above), so that multiple excerpts will use the same class for the same term. If you want to re-use the same HtmlFormatter object with different searches, you should call HtmlFormatter.clear() between searches to clear the mapping. z6<%(tag)s class=%(q)s%(cls)s%(tn)s%(q)s>%(t)sc||_||_||_||_||_||_i|_dj|j|jf|_y)a :param tagname: the tag to wrap around matching terms. :param between: the text to add between fragments. :param classname: the class name to add to the elements wrapped around matching terms. :param termclass: the class name prefix for the second class which is different for each matched term. :param maxclasses: the maximum number of term classes to produce. This limits the number of classes you have to define in CSS by recycling term class names. For example, if you set maxclasses to 3 and have 5 terms, the 5 terms will use the CSS classes ``term0``, ``term1``, ``term2``, ``term0``, ``term1``. ruN) rtagname classname termclass attrquote maxclassesseenr htmlclass)r rrrrrrs rr"zHtmlFormatter.__init__sR"  """$ 4>>4>>"BCrct|dS)NF)quoterrs rrzHtmlFormatter._texts$e,,rc|j}|jt|||}||vr||}nt||jz}|||<|j |j |j|j||dzS)N)tagqclsr!tn) rrrrrtemplaterrr)r rrrrttexttermnums rrzHtmlFormatter.format_tokens{yy 8D%9: D=5kG$i$//1G!DK}}t||$..'+~~E&- // /rci|_y)z@Clears the dictionary mapping terms to HTML classnames. N)rr$s rcleanzHtmlFormatter.cleans  rN)strongrmatchterm"r) r9r:r;r<rr"rrrr=rrrrs-(HH16ABD4- /rrc6eZdZdZddZdZd dZd dZd dZy) GenshiFormatterz[Returns a Genshi event stream containing HTML formatting around the matched terms. c||_||_ddlm}m}m}ddlm}m}|||c|_|_|_||c|_|_y)z :param qname: the QName for the tag to wrap around matched terms. :param between: the text to add between fragments. r)STARTENDTEXT)AttrsStreamN)qnamer genshi.corerrrrr)r rrrrrrrs rr"zGenshiFormatter.__init__sA   00-*/d' DHdi"' DKrc|r7|dd|jk(r"|j|dd|z|ddf|d<y|j|j|dfy)Nr rr_Nr r )rrS)r rrs r _add_textzGenshiFormatter._add_textsX fRjmtyy0))VBZ]T%96":a=IF2J MM499dN; :<= =rc\g}|j}|j}|jD]P}|j|kDr|j|||j||j |||f|j }R|t |kr|j||d||j|Sr')r rrrrSr rr)r r-rrrrr!s rrzGenshiFormatter.format_fragments""}}!! A{{U"tE!++6? MM4G, -IIE   3t9  NN4< 0{{6""rcg}d}|D]8}|s|j|j|||j||z }d}:|j|S)NTrF)rrrr)r rrrrar-s rrzGenshiFormatter.formats`! Ht||V4 d**8W*E EFE   {{6""rN)rrr) r9r:r;r<r"rrrrr=rrrrs  0= = ##rrr_cfd|D}t||}|Dcgc] \}}||k\s |}}}|j||Scc}}w)Nc32K|]}||fywr'r=)rrscorers rrz top_fragments..s:1A:key)rsort) rcountrorderminscorescored_fragmentsrsfbest_fragmentss ` r top_fragmentsrsW: :'78*:PYUBex>ObPNPE" Qs AAqueryc \| t}t|tur|}t|tur|}t|tur|}| t}t|} ||d| d} t| | } |j || } t | ||||} ||| S)NTF)charsmode removestops)rtyper\rArJr) rtermsanalyzer fragmenter formattertoprrrrr@rrs r highlightr s~$& J4\  I$K  F|t ~$&G d$Tu EF  0F**48IifeXFI T9 %%rcLeZdZddddefdZdZedZedZddZ y) HighlighterNFc|xs t|_|xs t|_|xs t d|_||_||_y)Nb)r)rfrrrrrralways_retokenize)r rrrrrs rr"zHighlighter.__init__1sB$;(9(;5 3 5 "@mC&@ !2rc|jry|jsy|jjry|jj |}|j dS)NF characters)rhas_matched_termsrrEsearcherschemasupports)r results fieldnamefields rcan_load_charszHighlighter.can_load_chars9sY  ! !((* ?? * * ,  '' 2~~l++rcix|j|<}td|jD}|D]}i||< |D]}||}|jj ||} t |j ||f} |D]D}|| vs| j|| j|k(sJ| jd|||<Fy)Nc3&K|] \}}| ywr'r=)r_docnums rrz*Highlighter._load_chars..RsByq&FBsr) _char_cachesortedtop_nrpostingsrtermdocsskip_tor6value_as) rrtextsto_bytescache sorted_idsrrbtextmdocsets r _load_charszHighlighter._load_charsLs 243I&BGMMBB   FE&M  CDTNE  )))U;A))9e*<=>F$ CV#IIf%446V+++*+**\*BE&M$'  C  Crc#Kd}|D]}|js ||d}|||j}/|j|jkre|j|jkDsb|xj|j|j|jz dz c_|j|_|d}|||yywr')r?rTr r r)rrr!s r_merge_matched_tokensz!Highlighter._merge_matched_tokensas  A99$K E} -99u}},JJ!&&qyy)@)A"BBJ$%IIEM % (  K s A&C)ACc |j}|jj}|}|j} |j||vrt dz|}|j rfd|jD} n|jd} tfd| D} |j|r |jvr|j|| | fd|jD} |j|j} g}|jj}| D]9}| |}|D]-\}}}|r||kDr|j!t#||||/;|j%d t'|d Dcgc]\}}t)|d }}}|jj+||}ni|jjj,}||ddd d }t/|| }|j1|}|jj3||}t5|||j6|j8|}|j:j=|}|Scc}}w)NzField %r is not stored.c32K|]}|dk(r|yw)rNr=)rrrs rrz,Highlighter.highlight_hit..s$/taI-/rT)expandrc34K|]}|dyw)r_Nr=)rr from_bytess rrz,Highlighter.highlight_hit..sA$*T!W-Asc3DK|]}|dk(r |dyw)rr_Nr=)rrrr3s rrz,Highlighter.highlight_hit..s-17i/#47+1s )rposr r c|jSr'rr!s rz+Highlighter.highlight_hit..s akkrrc|jSr'rr7s rr8z+Highlighter.highlight_hit..s !++rc4|j|jz Sr'r(r7s rr8z+Highlighter.highlight_hit..sqyy1;;/FrrF) positionsrrr)r)rrrr&r3KeyErrorrr query_termsr\rrr,rrrQrSr rrrrMrrAr.rJrrrrr)r hitobjrrr rrrrr&btermswordshittermscmaprrQwordrr5r r rgrouprrrr3s ` @r highlight_hitzHighlighter.highlight_hits..!!((y!>>%% <&89DEE)$D  $ $ &/w'<'<'>/F(( (JFA&AA   w 2 3 33  )UHE18L8L8N1H&&y1&--@DF11I  OT /4O+CG Wy%8MM%Ts2;W#NOO O KK1K 2(/8M(NP$#u%%FGPFP88vFI''..y9BBHdd$W*/1F(6F//7F77fEI!)S$++tzz+35 &&y1 !PsI2)Nrr_) r9r:r;rr"rrr,r.rEr=rrr r 0sG"&tt#(3,&CC(:;rr )NNrr)r_)'r< __future__r collectionsrheapqr itertoolsr whoosh.compatrwhoosh.analysisr rVrobjectrrArCrO NullFragmeterrXrfrorrrrrrrrrrrrrr r r=rrrNs58. $! 26%&6"H$vH$Z""""J#)j#)N HFHFVUF UFpSSp V  .     7J&J&Z.I.   DIDN6#i6#vABAU&0J&Jr