Pmjd} dZddlZddlZddlZddlZddlZddlmZddlm Z m Z ddl m Z e rddl mZejeZGddZGd d Zejd ejZejd ejZejd ejZejdejZejdZejdZejdejZejdZejdZde de fdZ!GddZ"de de fdZ#ejdZ$de de%fdZ&de de'e fdZ(de'e de fd Z)de de fd!Z*ddddd"d#e d$e dzd%e dzd&d'd(d)dd*f d+Z+de de%fd,Z,de de%fd-Z-de de%fd.Z.de de%fd/Z/ejd0Z0dBd1Z1de dd2fd3Z2d4e d5e de fd6Z3dCd8Z4dCd9Z5dDd;Z6 dBdd?Z7dBd@Z8dAZ9dS)EzShared helper classes for gateway platform adapters. Extracts common patterns that were duplicated across 5-7 adapters: message deduplication, text batch aggregation, markdown stripping, and thread participation tracking. N)Path) TYPE_CHECKINGDict)atomic_json_write) MessageEventcZeZdZdZddedefdZdedefd Z dedefd Z dedd fd Z d Z d S)MessageDeduplicatora|TTL-based message deduplication cache. Replaces the identical ``_seen_messages`` / ``_is_duplicate()`` pattern previously duplicated in discord, slack, dingtalk, wecom, weixin, mattermost, and feishu adapters. Usage:: self._dedup = MessageDeduplicator() # In message handler: if self._dedup.is_duplicate(msg_id): return ,max_size ttl_secondsc0i|_||_||_dSN)_seen _max_size_ttl)selfr r s ?/home/thesage/.hermes/hermes-agent/gateway/platforms/helpers.py__init__zMessageDeduplicator.__init__+s') ! msg_idreturnc|sdStj}||jvr#||j|z |jkrdS|j|=||j|<t|j|jkr||jz fd|jD|_t|j|jkrKt |jd|j d}t||_dS)z?Return True if *msg_id* was already seen within the TTL window.FTc(i|]\}}|k ||Sr).0kvcutoffs r z4MessageDeduplicator.is_duplicate..=s$LLL41aV!Qrc|dS)Nr)items rz2MessageDeduplicator.is_duplicate..Ds T!Wr)keyN)timerrlenritemssorteddict)rrnownewestrs @r is_duplicatez MessageDeduplicator.is_duplicate0s 5ikk TZ  TZ''$)33t 6"  6 tz??T^ + +49_FLLLL4:+;+;+=+=LLLDJ4:// J$$&&,,>/""$"&\\ urc|sdS|j|}|dStj|z |jkrdS|j|=dS)zBReturn whether *msg_id* is live in the cache without inserting it.FNT)rgetr&r)rrseen_ats rcontainszMessageDeduplicator.containsIsX 5*..(( ?5 9;; 49 , ,4 Jv urNc<|j|ddS)z 0).r)rGr7s r is_enabledzTextBatchAggregator.is_enableds 1$$reventrr%Nct|jpd}|j|}|s||_||j|<n|jd|j|_||_|j|}|r(|s|tj | ||j|<dS)z+Add *event* to the pending batch for *key*. N) r'textrJr/_last_chunk_lenrKdonecancelasyncio create_task_flush)rrOr% chunk_lenexistingpriors renqueuezTextBatchAggregator.enqueues (b)) =$$S)) 1$-E !!&DM#  '}<< <K|j|}|j|}|rt|ddnd}||jkr|jn|j}tj|d{V|j |d}|rH | |d{Vn+#t$rt d|YnwxYw|j||ur|j |ddSdS)z/Wait then dispatch the batched event for *key*.rTrNz<[TextBatchAggregator] Error dispatching batched event for %s)rKr/rJgetattrrIrHrGrWsleepr3rF Exceptionlogger exception)rr% current_taskpendinglast_lendelayrOs rrYzTextBatchAggregator._flushs]*..s33 -##C((=DK77$5q999!&.1F%F%F!!DL]mE""""""""" !!#t,,  f fmmE********** f f f  !_adeeeee f   " "3 ' '< 7 7   # #C . . . . . 8 7sB99%C! C!c|jD]*}|s|+|j|jdS)zCancel all pending flush tasks.N)rKvaluesrUrVr6rJ)rtasks r cancel_allzTextBatchAggregator.cancel_allsm'..00  D99;;   !!### rrN) r8r9r:r;r=r<rr?rNr>r]rYrkrrrrArAas.! # : : : :  :  : : : :%D%%%%I^I#I$IIII"//////(rrAz \*\*(.+?)\*\*z \*(.+?)\*z \b__(?![\s_])(.+?)(?r thread_ids rr z7ThreadParticipationTracker.__init__..s-* * * %.C NND* * * r) _platform _max_tracked_load_threads)rrrs rrz#ThreadParticipationTracker.__init__s=&'* * 26**,,* * *  rrc8ddlm}||jdz S)Nr)get_hermes_homez _threads.json)hermes_constantsrr)rrs r _state_pathz&ThreadParticipationTracker._state_paths2444444  dn#C#C#CCCrc |}|r[ tj|d}t |t r d|DSn#t$rYnwxYwgS)Nzutf-8)encodingc,g|]}t|Srrrs r z4ThreadParticipationTracker._load..sAAAyC NNAAAr)rexistsjsonloads read_text isinstancelistra)rpathdatas rrz ThreadParticipationTracker._loads!! ;;==  z$..'."B"BCCdD))BAADAAAAB     sAA44 BBNc|}t|j}t||jkr/||j d}t ||_t||ddS)N)indent)rrrr'rr*fromkeysr)rr thread_lists r_savez ThreadParticipationTracker._saves|!!4=)) {  d/ / /%t'8&8&9&9:K MM+66DM$ D999999rrcX||jvr d|j|<|dSdS)z-Mark *thread_id* as participated and persist.N)rrrrs rmarkzThreadParticipationTracker.mark s4 DM ) )'+DM) $ JJLLLLL * )rc||jvSr)rrs r __contains__z'ThreadParticipationTracker.__contains__sDM))rc8|jdSr)rr6r7s rr6z ThreadParticipationTracker.clears r)r~rl)r8r9r:r; _MAX_TRACKEDr>r<rrrrrrrr?rr6rrrr}r}s$L  c     DTDDDD tCy    ::::cd *c*d****rr}phonec|sdSt|dkr-t|dkr|dddz|ddzndS|dddz|ddzS) zRedact a phone number for logging, preserving country code and last 4. Replaces the identical ``_redact_phone()`` functions in signal.py, sms.py, and bluebubbles.py. zNz****)r')rs r redact_phonerss x 5zzQ25e**q..uRaRy6!E"##J..fL !9v bcc **rz0^\s*\|?\s*:?-+:?\s*(?:\|\s*:?-+:?\s*){1,}\|?\s*$linecP|}t|od|vS)z:Return True if *line* could plausibly be a table data row.|)ryr?rstrippeds r is_table_rowr7s$zz||H >> -cXo-rc$ddlm}||S)a(Split a GFM table row into stripped cell values. Thin delegate to the canonical implementation in :mod:`agent.markdown_tables` (``split_table_row``) so the three formerly byte-identical copies (here, ``agent/markdown_tables.py``, ``weixin._split_table_row``) share one body. r)split_table_row)agent.markdown_tablesr)rrs rsplit_markdown_table_rowr=s&655555 ?4  r table_blockcpt|dkrd|St|d}t|dkrd|St|dkrt|dng}t|t|dzk}g}t|dddD]S\}}t|}|r"|r|dr|dnd|}|dd} nt d |Dd|}|} t| t|kr7| d gt|t| z zn7t| t|kr| dt|} g} t || D])\} } |s| |kr| d | d | *d |d g| } |d| Ud|S)u4Render a detected GFM table as bold-heading + bullet groups. Uses the same alignment logic as Telegram's renderer: for non-row-label tables, ``data_cells = cells`` (the full row) and the bullet whose value duplicates the heading is skipped. This keeps header→value alignment correct. rRrrr"N)startzRow c3K|]}||V dSrr)rcells r z&_render_table_block..gs';;Td;D;;;;;;rrQu• z: z**rn)r'joinr enumeratenextextendzipappend)rheadersfirst_data_rowhas_row_label_colrendered_groupsindexrowcellsheading data_cellsbulletsheadervalue group_liness r_render_table_blockrJsd ;!yy%%%&{1~66G 7||ayy%%% {  a   !Q000  N++s7||a/??!#O ABBq99977 s(--  "'HE!HHeAhh...GqrrJJ;;U;;;^E^^LLGJ z??S\\ ) )   rdc'llS__&DE F F F F __s7|| + +#Nc'llN3J *55 5 5MFE$ ')9)9 NN3&33E33 4 4 4 4'G'''2'2 tyy556666 ;; ' ''rcd|vsd|vr|S|d}g}d}d}|t|kr||}|}|dr| }|||dz }c|r|||dz }d|vr|dzt|krt ||dzr|||dzg}|dz}|t|kr]t||rH||||dz }|t|krt||H|t||}e|||dz }|t|kd |S) zuRewrite GFM pipe tables into bold-heading + bullet groups. Tables inside fenced code blocks are left alone. r-rRFr```r"r) splitr'lstrip startswithrTABLE_SEPARATOR_REmatchrrr) rSlinesoutin_fenceirrrjs rconvert_table_to_bulletsr{s  $#T// JJt  ECH A c%jj..Qx;;==   u % % #|H JJt    FA    JJt    FA  4KKAE """((q1u66# q1u.KAAc%jj..\%(%;%;.""58,,,Qc%jj..\%(%;%;. JJ*;77 8 8 8A  4 Q; c%jj..> 99S>>r)platform_label display_labeldefaultslogger_ log_prefixrrrzlist[str] | Nonerzlogging.Logger | Nonezlist[re.Pattern]c |pt}|%|p|}|}|gSt|tr|g}t|ts,|d||t |jgSg} |D]} t| tr| s, | tj | tj `#tj $r#} |d||| | Yd} ~ d} ~ wwxYw| r%| d|t| || S|t|pg}nt|trv|} | rtj| ng} n#t"$rd} YnwxYwt| tr| nd| D}nt|tr|}n|g}g} |D]} t| } | s& | tj | tj Z#tj $r"} |d|| | Yd} ~ d} ~ wwxYw| S)aZCompile regex wake-word/mention patterns from config or env values. Two adapter families share this logic: * **Config-style** (dingtalk, telegram): pass ``platform_label`` (e.g. ``"dingtalk"``). ``raw`` is the value from ``config.extra`` after env fallback parsing; must be a list or string, anything else logs a warning and yields ``[]``. Non-string entries are skipped. A summary info log is emitted when patterns load. * **Wakeword-style** (photon, bluebubbles): pass ``defaults``. ``raw`` may be None (use defaults), a string (JSON list or comma/newline separated), a list, or a scalar (wrapped in a list). Entries are coerced via ``str()``. ``log_prefix`` is interpolated into every log message so per-adapter log output stays byte-identical to the historical inline implementations. Nz9[%s] %s mention_patterns must be a list or string; got %sz&[%s] Invalid %s mention pattern %r: %sz$[%s] Loaded %d %s mention pattern(s)cfg|].}|dD]}|/S),)rry)rrparts rrz,compile_mention_patterns..sQ< < <  3< <  JJLL< < < < rz#[%s] Invalid mention pattern %r: %s)rbrr>rwarningtyper8ryrrecompile IGNORECASEerrorinfor'rrra splitlines)rawrrrrrlogdisplaypatternscompiledpatternexcrSloadeds rcompile_mention_patternsrs4  VC!1>  I h $ $ " zH(D))  KKKX'     I%'  Ggs++ 7==??   7BM B BCCCC8    <     HH6H       {B'' C  yy{{ )-5TZ%%%2FF   FFF '55 66< < ))< < <  C  5HVV7||!!##   V OOBJtR];; < < < <x V V V KK=z4QT U U U U U U U U V OsB!2CD#DD0F FF2I  I;I66I;cjd}|dD]}|dr| }|S)zReturn True when *text* ends inside an unclosed ``` code fence. Scans line by line, toggling in/out state on lines starting with ```. An odd number of toggles means the trailing fence is unclosed. FrRr)rr)rSrrs rtext_has_unclosed_fencer sE H 4  $$ ??5 ! ! $#|H Orc|}|sdS|dd}|do|dS)z=True when the last non-empty line starts and ends with ``|``.FrRr)rstriprryrendswith)rStrimmed last_lines rtext_ends_with_table_rowr-sekkmmG u d##B'--//I    $ $ @););C)@)@@rcP|dS)z._is_table_lines6::<<""3''BH,=,=c,B,BBrNcrTd}|r|dSdS)NrR)rryrr6)atomatoms current_liness r_flush_currentz,split_markdown_atoms.._flush_currents`  "99]++Dzz|| # T"""    ! ! ! ! !  " "rrr"TrrQrl)rr>r?rrr'ry)rSrrrr"rr r!s @@rsplit_markdown_atomsr#ws JJt  EE!#MHCSCTCCCC"""""""''  '   & & &u%% !#m*<*'>u'E'Et ++>bD~   GODJJJ 4 JrrncV|pt}g}d}|D]}|s|n|||} || |kr| } |r||d}|||kr|}K||||l|||r|||S)aGreedily pack pre-split *blocks* into chunks of at most *max_length*. Blocks are joined with *sep* while they fit. A block that alone exceeds the limit is passed to *overflow(block)* (which must return a list of chunks) when provided, else emitted as-is. rQ)r'rr) blocks max_lengthrr.overflowrpackedr-block candidates rgreedy_pack_blocksr@s =SDFG!!!(FEE.F#.Fu.F.F 4 ??j ( (G    MM' " " "G 4;;* $ $G    MM((5// * * * * MM%  g MrTF)prefer_paragraphsbalance_fencesc|pt}|sgS|rt|||}nt|||}|rt|}|S)uSplit markdown text into chunks of at most *limit*, respecting fences. Two strategies, selected by ``prefer_paragraphs``: ``prefer_paragraphs=True`` (yuanbao-derived, the richest): Extract atomic blocks (code fences, tables, paragraphs), greedily merge them up to *limit*, split still-oversized non-atomic chunks at paragraph boundaries, then re-merge small neighbours. Code blocks and tables are never split in the middle; a single atom larger than *limit* is emitted oversize rather than broken. ``prefer_paragraphs=False`` (stream_consumer-derived): Newline-preferred hard splitting with headroom reserved for fence markers when the text contains ```. ``balance_fences=True`` post-processes the chunks so a split inside a code block closes the fence on the head chunk and reopens it (with the language tag) on the tail — required by callers whose chunks are delivered as independent messages that each must render standalone. )r'_chunk_markdown_paragraphs_chunk_newline_preferredr8)rSlimitrrArBrr*s rsplit_text_fence_awarerGs`8 =SD  =+D%@@)$t<<6-f55 Mrc|pt}|||kr|gSt|}gt}gd}d fd }|D]}||} rdnd} || z| z} | |krr|gd}d} s\| |krVt|st |r8|t|||| | zz }|g} tD]\} }|||kr| |*| |vr| |Dt|r| |i|}|||krQt|||\}}|s|d|||d}}|r| ||||kQ|r| |t| dkrR| dg}| ddD]<}|d}|d z|z}|||kr||d<'||=|} d | DS) zCYuanbao-derived paragraph/atom chunking pipeline (see module docs).rrNc`r*ddSdS)Nrn)rr)r* current_partssr _flush_partsz0_chunk_markdown_paragraphs.._flush_partsQs9  6 MM&++m44 5 5 5 5 5 6 6rr)rr"rrncg|]}||Srr)rcs rrz._chunk_markdown_paragraphs..s # # #! #A # # #rrl) r'r#setrraddrrrr)rSrrrr indivisible_set current_lenrKratom_lensep_len projected_lenr,idxr2 remainingheadmergedprevcombinedr*rJs @@rrDrDAs8 =SD tDzzYv  ! & &EF"%%%O!#MK6666666**4::$+!!!#g-8 9 $ $ $ LNNNMKG y(("4((),9$,?,?)   F , , , MM$    T"""w)) LNNNF''%% U 4;;) # # MM%  / ! ! MM%  "5 ) )  MM%  d9oo ))99VOD) O"+JYJ"79::9Ni $ d###d9oo ))  % MM) $ $ $ 6{{Q%aykABBZ % %E":Df}u,HtH~~**%r  e$$$$ # #v # # ##rc|||kr|gS|}d|vrt|dz |dzd}ddlm}g}|}|||krx||||}|dd|}||dzkr|}||d|||dd}|||kx|r|||S) zCStream-consumer-derived newline-preferred splitting (no balancing).rrr"r)_custom_unit_to_cprRN)maxgateway.platforms.baser]r rr) rSrFr split_limitr]r*rV _cp_budgetsplit_ats rrErEs$ vd||uv K }}%"*eqj!44 :99999FI &  k ) )'' ;GG ??4J77 jAo % %!H i  *+++hii(//55 &  k ) )! i   Mrr)r*rrr)NrnN):r;rWrloggingrr&pathlibrtypingrrutilsrr_r getLoggerr8rbr rArDOTALLrorqrrrsrtru MULTILINErvrwrxr>r{r}rrr?rrrrrrrrrrr rr#r)r/r8r@rGrDrErrrrjs  &&&&&&&&######4333333  8 $ $ @@@@@@@@LRRRRRRRRp 2:& 2 2"*\2955?KK2:?KK455"*Z((bj 55 2:/ 0 0BJy))*========F + + + + + +, RZ7 .s.t.... !3 !49 ! ! ! !.(T#Y.(3.(.(.(.(b+3+3++++j"& $#''+````$J ` : ` ! `%`````h # $    A3A4AAAA++++++ CCCCCC &2:&9::0"0"0"0"f/s/{////dcss(.!!!!HB (  (((((VR$R$R$R$jr