=wg]dZddlmZddlmZddlmZddlmZm Z m Z m Z ddl m Z ddlmZddlmZGd d eZGd d eZGd deZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGdd eZGd!d"eZ Gd#d$eZ!Gd%d&eZ"y')(zC This module contains base classes/interfaces for "codec" objects. ) bisect_right)columns)lev)abstractmethodizipunichrxrange)CompoundStorage) emptybytes random_namec eZdZy)OutOfOrderErrorN)__name__ __module__ __qualname__H/var/www/horilla/myenv/lib/python3.12/site-packages/whoosh/codec/base.pyrr,srrceZdZdZedZedZed dZed dZdZ edZ ed Z ed Z y) CodecTctNNotImplementedErrorselfstoragesegments rper_document_writerzCodec.per_document_writer7!!rctrrrs r field_writerzCodec.field_writer=r!rctrrrdbfilebyteidss rpostings_writerzCodec.postings_writerCr!rNctrrrr&terminfoformat_termscorers rpostings_readerzCodec.postings_readerGr!rctSr)Automatars rautomatazCodec.automataMs zrctrrrs r terms_readerzCodec.terms_readerPr!rctrrrs rper_document_readerzCodec.per_document_readerTr!rctrrrr indexnames r new_segmentzCodec.new_segmentZr!rFNN) rrr length_statsrr r#r(r/r2r4r6r:rrrrr2sL"" "" """" """" ""rrcFeZdZdZdZdZd dZd dZdZdZ d Z d Z y) WrappingCodecc||_yr_childrchilds r__init__zWrappingCodec.__init__`  rc:|jj||Sr)rBr rs rr z!WrappingCodec.per_document_writerc{{..w@@rc:|jj||Sr)rBr#rs rr#zWrappingCodec.field_writerf{{''99rc<|jj||S)N)r')rBr(r%s rr(zWrappingCodec.postings_writeris{{**67*CCrNcB|jj|||||S)N)r-r.)rBr/r*s rr/zWrappingCodec.postings_readerls){{**68W428+: :rc:|jj||Sr)rBr2rs rr2zWrappingCodec.automataps{{##GW55rc:|jj||Sr)rBr4rs rr4zWrappingCodec.terms_readersrJrc:|jj||Sr)rBr6rs rr6z!WrappingCodec.per_document_readervrHrc:|jj||Sr)rBr:r8s rr:zWrappingCodec.new_segmentys{{&&w ::rr;r<) rrrrEr r#r(r/r2r4r6r:rrrr?r?_s2A:D:6:A;rr?c^eZdZedZedZedZedZdZdZ dZ y) PerDocumentWriterctrrrdocnums r start_doczPerDocumentWriter.start_docr!rctrr)r fieldnamefieldobjvaluelengths r add_fieldzPerDocumentWriter.add_fieldr!rctd)Nz(Codec does not implement writing columnsr)rrX columnobjrZs radd_column_valuez"PerDocumentWriter.add_column_values!"LMMrctrr)rrXrYitemss radd_vector_itemsz"PerDocumentWriter.add_vector_itemsr!rc@fd}|j|||y)Nc3KjrYj}j}j}|||fj jrXyywr) is_activeidweightrZnext)textrg valuestringvmatchers r readitemsz7PerDocumentWriter.add_vector_matcher..readitemssZ$$&{{}!*&nn. V[11  $$&s A(A.,A.)rb)rrXrYrkrls ` radd_vector_matcherz$PerDocumentWriter.add_vector_matchers  i9;?rcyrrrs r finish_doczPerDocumentWriter.finish_doc rcyrrros rclosezPerDocumentWriter.closerqrN) rrrrrVr\r_rbrmrprsrrrrRrRsb""""NN""@  rrRcdeZdZdZedZedZedZdZedZ dZ dZ y ) FieldWriterc 6|j}|j}|j}|j}|j}|r |j } nd} d} d} d} d} |D]\}}}}}| || krt d| d||| k(r| r|| krt d| d| d|d||| k7r)| || || k7r|||} ||| |} d} |dk(rx|| k7r| ||||} | ||}|t}|||||| || |yy)NcyNrr)rUrXs rz*FieldWriter.add_postings..srzField z .. zTerm :) start_field start_termadd finish_term finish_fielddoc_field_lengthrr )rschemalengthsrar|r}r~rrdfllastfnlasttext lastspellrYrXbtextrUrgrZr[s r add_postingszFieldWriter.add_postingss && __ hh&& (( **C-C 7<- / 3Iuffe!i&&8%&)&LMMF"xEH4D%)/9e'MNN F"'M%)v*= N!),Ix0"| 'M5! +F}" v .[- /^   M   N rctrrrrXrYs rr|zFieldWriter.start_fieldr!rctrr)rris rr}zFieldWriter.start_termr!rctrr)rrUrgvbytesr[s rr~zFieldWriter.addr!rctrrrrXris radd_spell_wordzFieldWriter.add_spell_word!!rctrrros rrzFieldWriter.finish_termr!rcyrrros rrzFieldWriter.finish_fieldrqrcyrrros rrszFieldWriter.closerqrN) rrrrrr|r}r~rrrrsrrrrurushJX"""""""""  rrucDeZdZedZeddZdZedZy)PostingsWriterctrr)rr,r+s rstart_postingszPostingsWriter.start_postings r!rNctrr)rid_rgrr[s r add_postingzPostingsWriter.add_postingr!rcyrrros rfinish_postingszPostingsWriter.finish_postingsrqrct)zAReturns True if this object has already written to disk. rros rwrittenzPostingsWriter.writtens "!rr)rrrrrrrrrrrrr sA"""" ""rrc$eZdZdZdZdZdZy) FieldCursorctrrros rfirstzFieldCursor.first!rrctrr)rstrings rfindzFieldCursor.find$rrctrrros rrhzFieldCursor.next'rrctrrros rr-zFieldCursor.term*rrN)rrrrrrhr-rrrrr s""""rrceZdZedZedZedZedZedZedZ edZ edZ ed Z edd Z ed Zd Zy ) TermsReaderctrrrr-s r __contains__zTermsReader.__contains__/r!rctrrrs rcursorzTermsReader.cursor3r!rctrrros rtermszTermsReader.terms7r!rctrrrrXprefixs r terms_fromzTermsReader.terms_from;r!rctrrros rrazTermsReader.items?r!rctrrrs r items_fromzTermsReader.items_fromCr!rctrrrs r term_infozTermsReader.term_infoGr!rcB|j||jSr)rrgrs r frequencyzTermsReader.frequencyKs~~i.5577rcB|j||jSr)r doc_frequencyrs rrzTermsReader.doc_frequencyOs~~i.<<>>rNctrr)rrXrir,r.s rmatcherzTermsReader.matcherSr!rctrrros rindexed_field_nameszTermsReader.indexed_field_namesWr!rcyrrros rrszTermsReader.close[rqrr)rrrrrrrrrarrrrrrrsrrrrr.s""""""""""""""88??"""" rrc6eZdZeddZedZddZy)r1cLtj|||jSr)rlevenshtein_automatonto_dfa)utermmaxdistrs rlevenshtein_dfazAutomata.levenshtein_dfa`s ((@GGIIrc#Ktd}|j}|y|j|}|rG|j||j}|y||k(r |||z }|j|}|rFyywrx)rrinext_valid_stringr)dfacurunullr-matchs r find_matcheszAutomata.find_matchesds}q xxz < %%d+ HHUO88:D|}  ))$/Es A7A<:A<cL|j|||}|j||Sr)rr)rfieldcurrrrrs r terms_withinzAutomata.terms_withinws)""5'6:  h//rNr)rrr staticmethodrrrrrrr1r1_s/JJ00$0rr1ceZdZdZedZedZedZedZedZ dZ dZ d Z d Z d Zd Zd ZeddZedZedZedZdZdZedZdZy)PerDocumentReadercyrrros rrszPerDocumentReader.closerqrctrrros r doc_countzPerDocumentReader.doc_countr!rctrrros r doc_count_allzPerDocumentReader.doc_count_allr!rctrrros r has_deletionszPerDocumentReader.has_deletionsr!rctrrrTs r is_deletedzPerDocumentReader.is_deletedr!rctrrros r deleted_docszPerDocumentReader.deleted_docsr!rc`|jfdt|jDS)zT Returns an iterator of all (undeleted) document IDs in the reader. c32K|]}|s|ywrr).0rUrs r z0PerDocumentReader.all_doc_ids..s +6!&)+s)rr r)rrs @r all_doc_idszPerDocumentReader.all_doc_idss- __ +VD,>,>,@%A+ +rc#`K|jD]}||j|fywrr stored_fieldsrTs r iter_docszPerDocumentReader.iter_docss4&&( 5F$,,V44 4 5s,.cyNFrros rsupports_columnsz"PerDocumentReader.supports_columnsrcyrrrrXs r has_columnzPerDocumentReader.has_columnrrctrrros r list_columnszPerDocumentReader.list_columnsrrctrrrrXcolumns r column_readerzPerDocumentReader.column_readerrrcyrrrs r field_docszPerDocumentReader.field_docsrctrr)rrUrXdefaults rrz"PerDocumentReader.doc_field_lengthr!rctrrrs r field_lengthzPerDocumentReader.field_lengthr!rctrrrs rmin_field_lengthz"PerDocumentReader.min_field_lengthr!rctrrrs rmax_field_lengthz"PerDocumentReader.max_field_lengthr!rcyrr)rrUrXs r has_vectorzPerDocumentReader.has_vectorrrctrr)rrUrXr,s rvectorzPerDocumentReader.vectorrrctrrrTs rrzPerDocumentReader.stored_fieldsr!rc#\K|jD]}|j|ywrrrTs rall_stored_fieldsz#PerDocumentReader.all_stored_fieldss/&&( -F$$V, , -s*,Nr)rrrrsrrrrrrrrrrrrrrrrrrr rr rrrrr~s """" """"""+5 ""  """""""" " ""-rrceZdZdZdZdZeddZdZdZ dZ dZ d Z d Z d Zd Zd ZdZdZedZdZdZdZedZedZeddZedZdZy)SegmentaDo not instantiate this object directly. It is used by the Index object to hold information about a segment. A list of objects of this class are pickled as part of the TOC file. The TOC file stores a minimal amount of information -- mostly a list of Segment objects. Segments are the real reverse indexes. Having multiple segments allows quick incremental indexing: just create a new segment for the new documents, and have the index overlay the new segment over previous ones for purposes of reading/search. "Optimizing" the index combines the contents of existing segments into one (removing any deleted documents along the way). z.segcJ||_|j|_d|_yr)r9 _random_idsegidcompound)rr9s rrEzSegment.__init__s"__&  rct|S)N)sizer )clsrs rrzSegment._random_ids %%rcXd|jjd|jdS)N< >) __class__r segment_idros r__repr__zSegment.__repr__s NN33T__5FGGrctrrros rcodecz Segment.codecrrc|jSr)r9ros r index_namezSegment.index_name ~~rcpt|dr |jS|jd|jS)Nname_)hasattrr#r rros rrzSegment.segment_ids, 4 99 "oo/< .0s MDt}}T%6%67Ms%(T)r/anyr1r:r assemble delete_filer)rrsegfilescfiler#s` rcreate_compound_filezSegment.create_compound_file.ss??7+MHMMMM  $*;*;<  : &D    % & rc|j|j}|j|}t||jS)N)use_mmap)r+r:r6r supports_mmap)rrr#r&s ropen_compound_filezSegment.open_compound_file7s;!!$"3"34""4(v0E0EFFrct)zg Returns the total number of documents, DELETED OR UNDELETED, in this segment. rros rrzSegment.doc_count_all>s "!rcD|j|jz S)zN Returns the number of (undeleted) documents in this segment. )r deleted_countros rrzSegment.doc_countGs! !!#d&8&8&:::rctrrrdoccounts r set_doc_countzSegment.set_doc_countNrrc(|jdkDS)zL Returns True if any documents in this segment are deleted. r)rGros rrzSegment.has_deletionsQs !!#a''rct)zP Returns the total number of deleted documents in this segment. rros rrGzSegment.deleted_countX "!rctrrros rrzSegment.deleted_docs`r!rct)zDeletes the given document number. The document is not actually removed from the index until it is optimized. :param docnum: The document number to delete. :param delete: If False, this undeletes a deleted document. rrrUdeletes rdelete_documentzSegment.delete_documentds "!rct)zG Returns True if the given document number is deleted. rrTs rrzSegment.is_deletedorNrcyNTrros rshould_assemblezSegment.should_assemblewrrN)T)rrr__doc__r:rE classmethodrrrr rr'r+r/r1r6r@rDrrrrKrrGrrSrrWrrrrrs L  &&H"=1L42G"";"(""""""""rrceZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zdd ZdZdZdZdZdZdZdZy)WrappingSegmentc||_yrrArCs rrEzWrappingSegment.__init__~rFrc6|jjSr)rBrros rrzWrappingSegment.codecs{{  ""rc6|jjSr)rBr ros rr zWrappingSegment.index_name{{%%''rc6|jjSr)rBrros rrzWrappingSegment.segment_idrarc6|jjSr)rBr'ros rr'zWrappingSegment.is_compounds{{&&((rc6|jjSr)rBrWros rrWzWrappingSegment.should_assembles{{**,,rc8|jj|Sr)rBr+r)s rr+zWrappingSegment.make_filenames{{((--rc8|jj|Sr)rBr/rrs rr/zWrappingSegment.list_filess{{%%g..rc >|jj||fi|Sr)rBr1rrr*r3s rr1zWrappingSegment.create_files &t{{&&w>v>>rc >|jj||fi|Sr)rBr6ris rr6zWrappingSegment.open_files $t{{$$Wc.s<1??$<r;ryros rrz$MultiPerDocumentReader.has_deletionss->? &IAv..* &vo% & &AAc#Kt|j|jD]!\}}|jD] }||z #ywr)rryrzrrs rrz"MultiPerDocumentReader.all_doc_idssIdmmT->->? &IAv--/ &vo% & &rc@tfd|jDS)Nc3@K|]}|jywr)r)rrrXs rrz4MultiPerDocumentReader.has_column..sBq1<< *Bsrrs `rrz!MultiPerDocumentReader.has_columnsBDMMBBBrc|j|std||j}g}|jD][}|j|r|j ||}n$t j ||j}|j|]t|dk(r|dSt j|S)Nz No column rr) r ValueError default_valueryrrEmptyColumnReaderrr|lenMultiColumnReader)rrXrr colreadersrcrs rrz$MultiPerDocumentReader.column_readersy)y:; ;&&(  "A||I&__Y7..w8IJ   b !  " z?a a= ,,Z8 8rcj|j|\}}|j|j|||Sr)rryr)rrUrXrrrs rrz'MultiPerDocumentReader.doc_field_length s4&&v.1}}Q00IwGGrcTd}|jD]}||j|z }|Srx)ryr)rrXrrs rrz#MultiPerDocumentReader.field_lengths2 /A Q^^I. .E / rc:td|jDS)Nc3<K|]}|jywr)rrs rrz:MultiPerDocumentReader.min_field_length..?A1%%'?r)minryros rrz'MultiPerDocumentReader.min_field_length????rc:td|jDS)Nc3<K|]}|jywr)rrs rrz:MultiPerDocumentReader.max_field_length..rr)rryros rrz'MultiPerDocumentReader.max_field_lengthrrNr)rrrrErsrrrrrrrrrrrrrrrrrrwrwsW   C%=.& &C9(H @@rrwcHeZdZdZedZedZedZdZy)PerDocWriterWithColumnscXtj|d|_d|_d|_yr)rRrE_storage_segment_docnumros rrEz PerDocWriterWithColumns.__init__s%""4(   rctrrrs r _has_columnz#PerDocWriterWithColumns._has_column&r!rctrrrs r_create_columnz&PerDocWriterWithColumns._create_column*r!rctrrrs r _get_columnz#PerDocWriterWithColumns._get_column.r!rc|j|s|j|||j|j|j|yr)rrrr~r)rrXrrZs rr_z(PerDocWriterWithColumns.add_column_value2s? *    6 2 #'' errcyrrros rrhzEmptyCursor.nextArrcyrrros rrizEmptyCursor.textDrrcyrrros rrzEmptyCursor.term_infoGrrcyrrros ris_validzEmptyCursor.is_validJrrN) rrrrrrhrirrrrrrr:s rrN)#rZbisectrwhooshrwhoosh.automatar whoosh.compatrrrr whoosh.filedb.compoundr whoosh.systemr whoosh.utilr Exceptionrobjectrr?rRrurrrr1rrr]rwrrrrrrs8 >>2$#  i *"F*"Z;E;@  Dd &d R"V", "& ". &. b0v0>^-^-FWfWx;+g;+@\@.\@B=/=8+r