PmjUdZddlmZddlZddlZddlZddlZddlmZej e Z ej dZ dHd ZdId ZdJdZdKdZdLdMdZdNdOdZdHdZdJdZdPdZdJdZdIdZgdZdQdRd$ZdSd&ZdTd(Zd)ed*d+hed,d-fd.eed.hd/d0fd1eed2hd3d4ffZd5ed6<dUd8Z dVd<Z!dWd>Z"dXd?Z#dYdEZ$dZdGZ%dS)[atMessage and tool-payload sanitization helpers. Pure functions extracted from ``run_agent.py`` so the AIAgent module can stay focused on the conversation loop. These walk OpenAI-format message lists and structured payloads, repairing or stripping problematic characters that would otherwise crash ``json.dumps`` inside the OpenAI SDK or be rejected by upstream APIs. All helpers are stateless and side-effect-free except for in-place mutation of their input (where documented). Backward-compatible re-exports from ``run_agent`` remain in place so existing imports ``from run_agent import _sanitize_surrogates`` keep working. ) annotationsN)Anyz[\ud800-\udfff]textstrreturncpt|rtd|S|S)zReplace lone surrogate code points with U+FFFD (replacement character). Surrogates are invalid in UTF-8 and will crash ``json.dumps()`` inside the OpenAI SDK. This is a fast no-op when the text contains no surrogates. �) _SURROGATE_REsearchsubrs @/home/thesage/.hermes/hermes-agent/agent/message_sanitization.py_sanitize_surrogatesr s5 D!!1  4000 Kpayloadrboolc0dfd|S)uuReplace surrogate code points in nested dict/list payloads in-place. Mirror of ``_sanitize_structure_non_ascii`` but for surrogate recovery. Used to scrub nested structured fields (e.g. ``reasoning_details`` — an array of dicts with ``summary``/``text`` strings) that flat per-field checks don't reach. Returns True if any surrogates were replaced. Fct|tr|D]|\}}t|tr;t|r td|||<dUt|ttfr |}dSt|trt|D]~\}}t|tr;t|r td|||<dUt|ttfr |}dSdS)Nr T) isinstancedictitemsrr r r list enumerate)nodekeyvalueidx_walkfounds rrz-_sanitize_structure_surrogates.._walk5sQ dD ! ! !"jjll ! ! UeS))!$++E22%$1$5$5h$F$FS $d|44!E%LLL  ! !d # # !'oo ! ! UeS))!$++E22%$1$5$5h$F$FS $d|44!E%LLL ! ! ! !rrrrs @@r_sanitize_structure_surrogatesr"+s= E!!!!!!& E'NNN Lrmessagesrcpd}|D]}t|ts|d}t|tr;t|r!td||d<d}nt|tr~|D]{}t|trd|d}t|tr:t|r td||d<d}||d}t|tr:t|r td||d<d}|d}t|trs|D]o}t|ts|d} t| tr:t| r td| |d<d}|d } t| tr| d} t| tr:t| r td| | d<d}| d } t| tr:t| r td| | d <d}q|D]\} }| d vr t|tr;t|r td||| <d}Zt|ttfrt|rd}|S) aSanitize surrogate characters from all string content in a messages list. Walks message dicts in-place. Returns True if any surrogates were found and replaced, False otherwise. Covers content/text, name, tool call metadata/arguments, AND any additional string or nested structured fields (``reasoning``, ``reasoning_content``, ``reasoning_details``, etc.) so retries don't fail on a non-content field. Byte-level reasoning models (xiaomi/mimo, kimi, glm) can emit lone surrogates in reasoning output that flow through to ``api_messages["reasoning_content"]`` on the next turn and crash json.dumps inside the OpenAI SDK. Fcontentr Trname tool_callsidfunction arguments>r&roler%r') rrgetrr r r rrr")r#rmsgr%partrr&r'tctc_idfnfn_namefn_argsrrs r_sanitize_messages_surrogatesr4Lsw E3!3!#t$$  '')$$ gs # # % (<(Repaired unescaped control chars in tool_call arguments for %sz ,\s*([}\]])z\1{}[]r2Nu8Repaired malformed tool_call arguments for %s: %s → %sPuARepaired control-char-laced tool_call arguments for %s: %s → %suPUnrepairable tool_call arguments for %s — replaced with empty object (was: %s))rrstriploggerwarningjsonloadsdumpsJSONDecodeError TypeError ValueErrorrer countrangeendswithrG) rIrJ raw_strippedparsed reserialisedfixed open_curly open_bracket_escapeds r_repair_tool_call_argumentsrms(2(C'@'@H8>>###bL CYOOOtvI9UUUt  L777z&Z@@@ < ' ' NNP     )Z 8      E F>5% 0 0ES!!EKK$4$44J;;s##ekk#&6&66LA~~ z!!a |## 2YY    Ju    E#   ~~c"" u{{3'7'7%++c:J:J'J'Jcrc $$ S)9)9EKK> e   Jw    NNS<,gcrcl   N   )Z 8      NN /<$ 4sL/AB>>CC<FB'H?>H?AJJJAK55LLfinal_responsec|sdS|d}t|tr|ddkrdSt|tr|nd}|d|pddd S) u"Append a synthetic assistant turn when an interrupted tail is a tool result. A turn cut short by ``/stop`` can leave the transcript ending on a raw ``tool`` message (a tool finished, or its execution was cancelled, but the model never streamed a closing assistant turn). Persisting that tail means the next user message lands as ``… tool → user`` — a role-alternation violation that strict providers (Gemini, Claude) react to by hallucinating a continuation of the user's message and ignoring prior context, which reads to the user as "lost context" (#48879). ``finalize_turn`` closes this on the happy interrupt path, but the retry/backoff/error interrupt aborts in ``conversation_loop`` ``return`` early and never reach it — this shared helper closes the sequence on all of them. ``final_response`` is usually empty on an interrupt, so an explicit placeholder is used rather than an empty-content assistant turn. Mutates ``messages`` in place. Returns True if a closing turn was appended. FrVr+toolr= assistantzOperation interrupted.)r+r%T)rrr,rr?rX)r#rnlastrs rclose_interrupted_tool_sequencerss& u B>"D OO::<<;#; 4rcV|dddS)zRemove non-ASCII characters, replacing with closest ASCII equivalent or removing. Used as a last resort when the system encoding is ASCII and can't handle any non-ASCII characters (e.g. LANG=C on Chromebooks). asciiignoreerrors)encodedecoder s r_strip_non_asciir{;s( ;;wx; 0 0 7 7 @ @@rcRd}|D] }t|ts|d}t|trt |}||kr||d<d}nut|t r`|D]]}t|trF|d}t|trt |}||kr||d<d}^|d}t|trt |}||kr||d<d}|d}t|t r|D]} t| trq| di} t| trF| d} t| trt | }|| kr|| d<d}|D];\} } | d vr t| trt | }|| kr||| <d}<"|S) aStrip non-ASCII characters from all string content in a messages list. This is a last-resort recovery for systems with ASCII-only encoding (LANG=C, Chromebooks, minimal containers). Returns True if any non-ASCII content was found and sanitized. Fr%Trr&r'r)r*>r&r+r%r')rrr,rr{rr)r#rr-r% sanitizedr.rr&r'r/r1r3rrs r_sanitize_messages_non_asciir~Ds] E/!/!#t$$  '')$$ gs # # )(11IG##!*I  & & ) ) )dD)))88F++D!$,,)$4T$:$: $,,+4DL$(Ewwv dC  (..ID  'F WW\** j$ ' ' - - -b$''- B//B!"d++-"$&&"5"5%gs33-(8(A(AI(G332;;(,))++ ! !JC???%%% !,U33 %%(CH E ! Lrtoolsc t|S)z7Strip non-ASCII characters from tool payloads in-place.)_sanitize_structure_non_ascii)rs r_sanitize_tools_non_asciirs ( / //rc2d}g}t|D]\}}t|ts|d}t|tsFg}|D]F}t|tr|ddvrd}1||Gt |t |kr<|r||d<|ddkrd|d<||t|D]}||=|S) uBRemove image_url content parts from all messages in-place. Called when a server signals it does not support images (e.g. "Only 'text' content type is supported."). Mutates messages so the next API call sends text only. Preserves message alternation invariants: * ``tool``-role messages whose content was entirely images are replaced with a plaintext placeholder, NOT deleted — deleting them would leave the paired ``tool_call_id`` on the prior assistant message unmatched, which providers reject with HTTP 400. * Non-tool messages whose content becomes empty are dropped. In practice this only hits synthetic image-only user messages appended for attachment delivery; real user turns always include text. Returns True if any image parts were removed. Fr%type>image image_url input_imageTr+rpu:[image content removed — server does not support images])rrrr,rr?r>reversed)r#r to_deleterDr-r% new_partsr.s r_strip_images_from_messagesrsH$ EIH%%$$3#t$$  '')$$'4((    ' 'D$%% '$((6*:*:>c*c*c  &&&& y>>CLL ( ( $!*IF**"^I  ### i  QKK Lrc0dfd|S)zCStrip non-ASCII characters from nested dict/list payloads in-place.Fc&t|tru|D]^\}}t|trt |}||kr|||<d7t|tt fr |_dSt|t rnt |D]`\}}t|trt |}||kr|||<d7t|tt fr |_dSdS)NT)rrrrr{rr)rrrr}rrrs rrz,_sanitize_structure_non_ascii.._walks5 dD ! ! !"jjll ! ! UeS))! 0 7 7I E))$-S $d|44!E%LLL ! !d # # !'oo ! ! UeS))! 0 7 7I E))$-S $d|44!E%LLL ! ! ! !rr r!s @@rrrs; E!!!!!!* E'NNN Lr)r rsrr"r4rGrmr{r~rrrdeterministic_call_idcoalesce_tool_call_iduniquify_tool_call_idsreasoning_echo_familymatches_reasoning_echo_familyneeds_reasoning_echoapply_reasoning_content_policyreapply_reasoning_echor2r*indexintc|d|d|}tj|dddd}d|S)uGenerate a deterministic call_id from tool call content. Used as a fallback when the API doesn't provide a call_id. Deterministic IDs prevent cache invalidation — random UUIDs would make every API call's prefix unique, breaking OpenAI's prompt cache. rOzutf-8replacerwN call_)hashlibsha256ry hexdigest)r2r*rseeddigests rrrsd + + + +E + +D ^DKK KBB C C M M O OPSQSPS TF 6  rr/ct|tr@|ddp|ddpdSt |ddpt |ddpdS)a'Extract the effective call ID from a tool_call entry (dict or object). Single owner for the ``call_id or id`` coalescing rule: Codex Responses tool calls carry ``call_id`` (authoritative pairing key), Chat Completions ones carry ``id`` only. Returns ``""`` when neither is set. call_idr=r()rrr,rXgetattr)r/s rrr s"dIy"%%?b)9)9?RFFHHH B 2 & & E'"dB*?*? E2 L L N NNrr'c t}|pgD]`}t|tr-|dp|dpd}n$t |ddpt |ddpd}t|t r|nd}|s|ddd}|s||vr||d}|d | |vr|dz }|d | |v|  fd } t|trJ|dr||d|d<n |d<|dr |d<n7|t |dd|_ t |ddr |_ n,#t$rt d |YwxYwt|tr|d nt |d d}t|tr|d nt |d dpd}t d| |b|S)uEnsure every tool call in a single assistant turn has a distinct id. Some models/providers reuse one call id across different calls in a single batch (observed with native Kimi Responses replays, Ollama- compatible endpoints, and degraded models at long context; same bug class as openclaw/openclaw#110518 / #110956). Duplicate ids are lossy downstream: the pre-API sanitizer keeps only the first call/result pair per id (#58327), so the later call's result silently vanishes from every replayed payload, and strict providers (Anthropic tool_use, DeepSeek) reject duplicate ids outright. The first occurrence keeps its id; later collisions get a deterministic ``_d`` suffix — never a random UUID, which would break prompt-cache prefix stability across replays. Mutates the entries in place (SDK models / SimpleNamespace / dicts) and returns the same list. Blank/missing ids are left for the deterministic fallback in ``build_assistant_message``. rr(r=N|r8rr9_dc|t|tr%d|vr!d|dddSS)Nrr8)rrsplit)rnew_ids r_renamedz(uniquify_tool_call_ids.._renamedEsK%%% <#,, ;;5;;sA#6#6q#9;;;Mrz,Could not uniquify duplicate tool call id %sr)r&rHzyModel reused tool call id %s within one turn; renamed the duplicate to %s (tool=%s) to keep call/result pairing lossless.)setrrr,rrrXraddr(r ExceptionrYrZ) r'seenr/r5cidrEr_fn_fn_namers @rrrs&DB7 7  b$   P&&##9rvvd||9rCC"i..O'"dD2I2IORC'S119ciikkkr  iiQ"   d?? HHSMMM  1nn FA]]q]]Fnn       "d## (66$<<&'x411BtHH%BtH66)$$+$*ByM T4!8!8992y$//(!'BJ    NN>    H   %/r4$8$8[bffZ   gb*VZ>[>['1#t'<'<\CGGFOOO'#vW[B\B\dad fh    sBF33%GGkimiz kimi-codingzkimi-coding-cnr )z api.kimi.comz moonshot.aiz moonshot.cndeepseek)r)zapi.deepseek.commimoxiaomi)r)zapi.xiaomimimo.comzxiaomimimo.comtuple_REASONING_ECHO_RULESfamilycTtD]}|d|kr|cSt|)Nr)rKeyError)rrules r _family_rulers<% 7f  KKK  6  rprovidermodelbase_urlc ddlm t|\}}}}}|pd} |pd ||vs| |vrdSt fd|DrdSt  fd|DS)aTrue when (provider, model, base_url) matches one echo-back family. Families can overlap (e.g. a deepseek-named model pointed at a kimi host); this membership test is independent per family so per-family predicates keep their original semantics. r)base_url_host_matchesr=Tc3 K|]}|vV dSNr ).0r model_lowers r z0matches_reasoning_echo_family..s( 4 4#3+  4 4 4 4 4 4rc30K|]}|VdSrr )rhostrrs rrz0matches_reasoning_echo_family..s1GG$$Xt44GGGGGGr)utilsrrlowerany) rrrrrk raw_providerslowered_providers model_subshostsprovider_lowerrrs ` @@rrrs,+++++=I&=Q=Q:A}'Un"++--N;B%%''K=  N6G$G$Gt 4 4 4 4 4 4 444t GGGGGGGG G GGr 'str | None'c^tD]$}t|d|||r |dcS%dS)uEClassify the provider direction for the reasoning_content echo policy. Returns ``"kimi"``, ``"deepseek"``, or ``"mimo"`` (first match in table order) when the target endpoint enforces reasoning_content echo-back on assistant turns, else ``None`` (strict/indifferent side — the field must be stripped). rN)rr)rrrrs rrrsD& (a(E8 L L 7NNN  4rc(t|||duS)z>+66 NN< ( ( +S 1 1 !  (+#$&,,1E  3+?G' ( ( KK+T 2 2 2'*#$ KK#T*****r api_messagescd}|D]}|ddkr|rB|dr4t||||dr|dz }`d|vr|dd|dz }|S)u0Re-pad (or strip) assistant turns' reasoning_content for the active provider. ``api_messages`` is built once, before the retry loop, while the *primary* provider is active. A mid-conversation fallback can then switch providers, so the reasoning fields baked into ``api_messages`` are shaped for the *prior* provider and must be reconciled against the *current* one: * Switching TO a require-side provider (DeepSeek / Kimi / MiMo thinking mode): assistant turns built when the prior provider did NOT need the echo-back go out without ``reasoning_content`` and the new provider rejects them with HTTP 400 ("The reasoning_content in the thinking mode must be passed back"). Re-apply the pad. * Switching TO a strict provider that rejects the field (Mistral, Cerebras, Groq, SambaNova, …): assistant turns built under a reasoning primary carry a ``reasoning_content`` pad (often a single space ``" "``), and the strict provider rejects it with HTTP 400/422 ("Extra inputs are not permitted"). Strip the field. This is the exact cross-provider fallback bug from #45655 — a DeepSeek primary pads history with ``" "``, the request falls back to Mistral, and Mistral 422s on the stale pad. Calling this immediately before building the request kwargs reconciles the fields against the *current* provider. It is idempotent and safe to call every iteration; it covers every fallback path. Returns the number of assistant turns whose reasoning_content was added or removed. rr+rqrr8N)r,rr)rrchangedrs rrrs:G ;;v  + - -   {{.//  *7G=O P P P{{.// 1  #g-- /6661  Nr)rrrr)rrrr)r#rrr)r5rrr)rH)rIrrJrrrr)r#rrnrrr)rrrr)r)r2rr*rrrrr)r/rrr)r'rrr)rrrr) rrrrrrrrrr)rrrrrrrr)rrrrrrrr)rrrrrrrrL)rrrrrr)&__doc__ __future__rrr[loggingratypingr getLogger__name__rYcompiler rr"r4rGrmrsr{r~rrr__all__rrr frozensetr__annotations__rrrrrrr rrrs   #"""""    8 $ $  -.. BAAAAH''''T^^^^^B@AAAA8888v0000 ----`:   ^      O O O OLLLLV YY '78 9 999;;35ii 55} YY[[))XJ//-/     HHHH*    HHHH X+X+X+X+v......r