Pmj?dZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z ddl mZddlmZmZmZmZddlmZddlmZdd lmZejeZd Zd Zd Zd Z d Z!dZ"dZ#dZ$GddZ%GddZ&efdJdZ'e e!e"eddKd"Z(dLd$Z)dMdNd'Z*dMdNd(Z+ddddd)dOd0Z,ed1dPd4Z-dQd9Z.ddd:dd;d<dRdDZ/eddddEdSdIZ0dS)TuOut-of-loop shutdown and event-loop liveness backstops (#66892, #69089). When the asyncio loop freezes mid-drain, every asyncio-based recovery path is structurally unable to fire: the drain deadline, status rewrites, and forensics all need the same loop that is stuck. launchd/systemd KeepAlive only restarts a *dead* process, so a wedged-but-alive gateway sits as a zombie until manual SIGKILL. This module provides: 1. A plain OS-thread shutdown watchdog armed at ``stop()``. If shutdown has not completed within ``restart_drain_timeout + grace``, it dumps all-thread stacks via ``faulthandler`` plus a metadata snapshot, then ``os._exit`` so the service manager can revive the process. 2. An event-loop heartbeat file at ``/state/gateway.heartbeat`` so external supervision can distinguish "process alive" from "loop frozen" (``gateway_state.json`` alone can't — it only rewrites on transitions/turns). 3. A lifetime thread watchdog that can still diagnose and hard-exit when the event loop is too frozen to run its own heartbeat or timeout callbacks. 4. A self-rescheduling floor timer that keeps the loop selector's timeout finite, giving existing async recovery tasks a chance to resume. ) annotationsN)datetimetimezone)Path)AnyCallableDictOptional)!GATEWAY_SERVICE_RESTART_EXIT_CODE)get_hermes_home)atomic_json_writegN@g>@g@g$@)statezgateway.heartbeat)logszgateway-shutdown-watchdog.logc2eZdZdZd dZdd Zdd Zdd Zd S)_LoopFloorTimerHandlezBCancelable owner for the currently scheduled selector floor timer.loopasyncio.AbstractEventLoopintervalfloatcf||_||_d|_d|_|dS)NF)_loop _interval _cancelled_timer _schedule)selfrrs ?/home/thesage/.hermes/hermes-agent/gateway/shutdown_watchdog.py__init__z_LoopFloorTimerHandle.__init__;s4 !59  returnNonecZ|j|j|j|_dSN)r call_laterr_tickrrs rrz_LoopFloorTimerHandle._scheduleBs#j++DNDJGG r c@|js|dSdSr$)rrr's rr&z_LoopFloorTimerHandle._tickEs,  NN       r cXd|_|j|jdSdS)NT)rrcancelr's rr*z_LoopFloorTimerHandle.cancelIs3 ; " K    # "r N)rrrrr!r")__name__ __module__ __qualname____doc__rrr&r*r rrr8snLLHHHH!!!!!!r rc4eZdZdZddZdd Zddd ZddZd S)_LoopLivenessWatchdogHandlez6Small lifecycle handle for the daemon liveness thread. stop_eventthreading.Eventthreadthreading.Threadc"||_||_dSr$) _stop_event_thread)rr3r5s rrz$_LoopLivenessWatchdogHandle.__init__Rs% r r!r"c8|jdSr$)r8setr's rstopz _LoopLivenessWatchdogHandle.stopVs r NtimeoutOptional[float]c<|j|dS)Nr=)r9join)rr=s rrAz _LoopLivenessWatchdogHandle.joinYs! '*****r boolc4|jSr$)r9is_aliver's rrDz$_LoopLivenessWatchdogHandle.is_alive\s|$$&&&r )r3r4r5r6r+r$)r=r>r!r")r!rB)r,r-r.r/rr<rArDr0r rr2r2Oso@@+++++''''''r r2rrrrr!c t|}|dkrtn#ttf$r t}YnwxYwt ||S)zAKeep at least one timer pending so selector waits remain bounded.r)r ValueError TypeError#DEFAULT_LOOP_FLOOR_TIMER_INTERVAL_Sr)rrresolved_intervals r_arm_loop_floor_timerrJ`sk @!(OO  ! !  " z "@@@?@ '8 9 99s ::)probe_interval probe_timeout max_strikes exit_coderKrLrMintrN%Optional[_LoopLivenessWatchdogHandle]c: || | tj d fd d fd }tj|dd } |n-#t$r t d d Yd SwxYwt |S)uSStart an out-of-loop watchdog that hard-exits after missed probes. The guard is on by default; operators opt out with ``gateway.loop_watchdog: false`` in config.yaml (enforced by the caller, ``GatewayRunner._start_loop_liveness_guards`` — this module stays config-agnostic so bare-loop tests can drive it directly). probe_eventr4r!Optional[bool]c tjz} rdS|tjz }|dkr|S|t |drdSm)NTrg?r@)time monotonicis_setwaitmin)rRdeadline remainingr3r=s r_wait_for_probez5start_loop_liveness_watchdog.._wait_for_probes>##g-   "" t 4>#3#33IA~~"))+++It(<(<== t r r"c&d}svtj} |jn9#t $rYdSt $r tddYdSwxYw|}|dS|rd} rdS|dz }| kr rdS t d|n#t $rYnwxYw tj dn,#t $rtd dYnwxYw rdS dd l m}|d n#t $rYnwxYwtjdSdS) Nrr@z.Failed to schedule gateway loop liveness probeTexc_infozGateway event loop missed %d consecutive liveness probes; dumping all thread stacks and exiting with code %d so the service supervisor can restart it. all_threadsz&Loop liveness faulthandler dump failed mark_exitedloop_liveness_watchdogreason)rX threadingEventcall_soon_threadsafer; RuntimeError ExceptionloggerdebugrWcritical faulthandlerdump_tracebackgateway.lifecycle_ledgerrdos_exit) strikesrR respondedrdr\rNrrr3 strikes_limits r _watchdogz/start_loop_liveness_watchdog.._watchdogsO//(/336 #/++K ))+/::::       Dt   ( 44I    ""  qLG&&  ""  9      V+===== V V V EPT UUUUU V  ""  @@@@@@ I.FGGGGG     HY    Fm6 6 sMA B%B?BC-- C:9C:>D&D=<D=E++ E87E8Tzgateway-loop-liveness-watchdogtargetdaemonnamez.Failed to start gateway loop liveness watchdogr^N)rRr4r!rSr+)rhriThreadstartrlrmrnr2) rrKrLrMrNrxr5r\rr3rwr=s ` ` @@@@@rstart_loop_liveness_watchdogrnsHGM""J       88888888888t  -F    EPT UUUtt 'z6 : ::s A &B  B rctjdd}|rt |St S)zHHERMES_HOME for process-level identity files (ignore profile overrides). HERMES_HOME)rsenvirongetstriprr )vals r_process_hermes_homersC *.. + + 1 1 3 3C Cyy   r homeOptional[Path]cD||n t}|jtS)z1Return ``/state/gateway.heartbeat``.)rjoinpath_HEARTBEAT_RELATIVErbases rget_loop_heartbeat_pathrs'#44)=)?)?D 4=- ..r cD||n t}|jtS)zBReturn the faulthandler / metadata dump path for a fired watchdog.)rr_WATCHDOG_DUMP_RELATIVErs rget_shutdown_watchdog_dump_pathrs'#44)=)?)?D 4=1 22r )pid start_timerextrar Optional[int]rr>rOptional[Dict[str, Any]]ct|}t||ntjt jt jtj d}|t||d< ddl m }|}|r||d<n#t$rYnwxYw|r|| t!||dn,#t$rt"dd YnwxYw|S) uAtomically rewrite the loop-liveness heartbeat file. ``start_time`` is the gateway process start (``time.time()`` epoch seconds) so supervisors can detect PID reuse. Best-effort — never raises. N)r updated_atrVrr) sample_memorymem)indentz&Failed to write gateway loop heartbeatTr^)rrOrsgetpidrnowrutc isoformatrUrVrrrrrlupdater rmrn)rrrrpathpayloadrrs rwrite_loop_heartbeatrsC #4 ( (D#/33ry{{;;l8<00::<<^%%G  %j 1 1   ::::::moo  ! GEN       uN$55555 NNN = MMMMMN Ks$ B$$ B10B1 C&DD)grace_s drain_timeoutrc tt|d}n#ttf$rd}YnwxYw tt|d}n#ttf$r t}YnwxYw||zS)z=Return the wall-clock leash for the shutdown watchdog thread.)maxrrGrF!DEFAULT_SHUTDOWN_WATCHDOG_GRACE_S)rrdraingraces rresolve_shutdown_watchdog_delayrs E-((#.. z "2E'NNC(( z "22212 5=s 66AA32A3 dump_pathdelay_ssnapshotr"c |jddn#t$rYdSwxYwdtj|t jtj |pid} t|dd5}| tj |t d z| d | t!j|d n%#t$$r| d YnwxYw| d|dddn #1swxYwYn#t$$rYnwxYw t&j d|ddtjdt&jt!jddS#t$$rYdSwxYw)z:Best-effort faulthandler + metadata dump before hard-exit.T)parentsexist_okNshutdown_watchdog_fired)eventrrfired_atrazutf-8)encoding)default z(--- faulthandler dump (all threads) --- )filerbz%(faulthandler.dump_traceback failed) z--- end dump --- z&Gateway shutdown watchdog fired after z.0fzs (pid=z); dumping all thread stacks. ra)parentmkdirOSErrorrsrrrrrropenwritejsondumpsstrflushrprqrlsysstderr)rrrheaderfhs r_write_watchdog_dumpr#svtd;;;; +y{{L..88::N F  )S7 3 3 3 r HHTZ444t; < < < HH@ A A A HHJJJ C+FFFFF C C CABBBBB C HH) * * * HHJJJ                        AW D A AIKK A A A    #555555      sz --:E% AE(C?>E?D!E D!!,E E%EE% E!E%% E21E26A+G## G10G1r`zgateway-shutdown-watchdog) done_event snapshot_fnrNrr|rOptional[threading.Event]r&Optional[Callable[[], Dict[str, Any]]]r|rr4c||ntj tt|dn#tt f$r t YnwxYwdkrSd fd } tj|d|n,#t$rt d d YnwxYwS) aArm a daemon-thread hard-exit backstop for a wedged shutdown path. If ``done_event`` is set before ``delay_s`` elapses, the thread exits quietly (normal / progressing shutdown completed). Otherwise it dumps diagnostics and calls ``os._exit(exit_code)``. Never raises. Returns the ``done_event`` (creating one when omitted) so the caller can disarm on successful completion. Nrrr!r"ctj z}tj|krS|tjz } t|drdStj|kS rdSd}4 }n(#t $r}dt |i}Yd}~nd}~wwxYw  n t}t| | t d |n#t $rYnwxYwtj tj fD]'} |#t $rY$wxYw ddlm}m}||n#t $rYnwxYw ddlm}|dn#t $rYnwxYw dd lm} | d n#t $rYnwxYwt-j dS) N?r@snapshot_error)rruhShutdown watchdog fired after %.0fs — forcing process exit (asyncio drain path appears wedged; see %s)r)remove_pid_filerelease_gateway_runtime_lock)drain_log_queuercshutdown_watchdogrf)rUrVrXrYrWrlreprrrrmrorstdoutrrgateway.statusrrhermes_loggingrrrrdrsrt)rZr[rexcrzstreamrrrrddelaydonerrNrs rrxz(arm_shutdown_watchdog.._watchdogls>##e+n)) 4>#3#33IyyY!4!4y55 n)) ;;==  F-1  " 9&;== 9 9 9,d3ii8 9(39X9Z9ZVUXFFFF  OO>          D z3:.  F         T T T T T T T T O    ( ( * * * *    D   6 6 6 6 6 6 OC ( ( ( ( (    D   < < < < < < K *= > > > > >    D  sr B(( C 2CC 5D DD=E EE#F F  F F$$ F10F15G GGTryzFailed to arm shutdown watchdogr^r+) rhrirrrGrFrr}r~rlrmrn) rrrrNrr|rxrrs ``` @@rarm_shutdown_watchdogrQs$$/::Y_5F5FD2E'NNC(( z "22212 zz ::::::::::xG $TBBBHHJJJJ GGG 6 FFFFFG Ks!<AA-)B&C?C) interval_srrshould_continuerrOptional[Callable[[], bool]]c4K tt|d}n#ttf$r t}YnwxYwt || | |sdSt j|d{V| |sdSt ||H)uRewrite the loop heartbeat file on a cadence until cancelled / gated off. Runs as an asyncio task on the gateway loop — if the loop freezes, this task stops and the file mtime/updated_at goes stale for external monitors. r)rrTN)rrrGrFDEFAULT_HEARTBEAT_INTERVAL_Srasynciosleep)rrrrrs rloop_heartbeat_foreverrs0uZ((#.. z "000/0 JT::::?  &/@/@ & FmH%%%%%%%%%  &/@/@ & F >>>> ?s "==)rrrrr!r) rrrKrrLrrMrOrNrOr!rP)r!rr$)rrr!r) rrrr>rrrrr!r)rrrrr!r)rrrrrrr!r")rrrrrrrNrOrrr|rr!r4) rrrr>rrrrr!r")1r/ __future__rrrprloggingrsrrhrUrrpathlibrtypingrrr r gateway.restartr hermes_constantsr utilsr getLoggerr,rmrrrH DEFAULT_LOOP_WATCHDOG_INTERVAL_SDEFAULT_LOOP_WATCHDOG_TIMEOUT_S!DEFAULT_LOOP_WATCHDOG_MAX_STRIKESrrrr2rJrrrrrrrrrr0r rrs.#"""""   ''''''''000000000000======,,,,,,######  8 $ $%)!#&)##' "&$%!4C!!!!!!!!.''''''''&: : : : : :"=:86 c;c;c;c;c;c;L///// 33333"&&* ''''''Z7"+ + + + b-1:> $+[[[[[[@5"&48 ????????r