Pmj?6dZddlmZddlZddlZddlmZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlZddlZddlmZddlmZmZmZmZmZddlmZddlm Z m!Z!m"Z"m#Z#dd l$m%Z%e j&e'Z(dd dudZ)dZ*dZ+dgZ,hdZ-dZ.dZ/dZ0dvdZ1dwd Z2dwd!Z3dxd#Z4dydzd'Z5d{d*Z6d|d,Z7dxd-Z8d}d/Z9d0d1d~d4Z:dd6Z;Gd7d8Ze j?dCDddEZ@e jAdFe jBZCdwdGZDddIZEdyddKZFdyddLZGdwdMZHdd:ddPZIddQZJd0aKddSZLddTZMddWZNddYZOdd]ZPdd`ZQddcZRGdddeZSGdfdgZTddjZUddmZVddnZWddpZXddqZYddrZZGdsdte"Z[dS)uvCua-driver backend (macOS, Windows, Linux). Speaks MCP over stdio to `cua-driver`. The Python `mcp` SDK is async, so we run a dedicated asyncio event loop on a background thread and marshal sync calls through it. The same `cua-driver call ` surface (click, type_text, hotkey, drag, scroll, screenshot, launch_app, list_apps, list_windows, get_window_state, move_cursor, wait) works identically across macOS, Windows, and Linux — cua-driver's PARITY matrix marks the action tools VERIFIED on macOS and Windows in the cross-platform Rust port (`cua-driver-rs`). Linux is the most recent runtime (X11 today, Wayland via XWayland; pure- Wayland progress tracked upstream). It is enabled in `check_computer_use_requirements` alongside macOS and Windows. The plumbing in this file is OS-agnostic; per-host gaps (no DISPLAY, missing AT-SPI, etc.) surface as specific blocked checks via `hermes computer-use doctor` rather than failing silently. Install: - **macOS**: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" - **Windows** (PowerShell): irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex After install, `cua-driver` is on $PATH and supports `cua-driver mcp` (stdio transport) which is what we invoke. The macOS path uses private SkyLight SPIs (SLEventPostToPid, SLPSPostEventRecordTo, _AXObserverAddNotificationAndCheckRemote) that aren't Apple-public and can break on OS updates. The Windows path in cua-driver-rs uses stable Win32 APIs (SendInput + UI Automation) — not subject to the same SPI breakage class. ) annotationsN)deque)PureWindowsPath)AnyDictListOptionalTuple)windows_hide_flags) ActionResult CaptureResultComputerUseBackend UIElement)CuaTypedBrowserRouterequested_deliverynamestrokboolmessagemetaDict[str, Any] structuredr Optional[str]returnr c Zt|tr|nidfd }|d}t|tsd}|d}t|tsd}|d } t| tsd} |d } t| tsd} |d } t| tsd} |d p |d } t| tsd} t|tr|nd} t |||||| | | | |  S)aBuild an ActionResult, lifting cua-driver's structured verdict. All structured fields are additive: a driver that omits ``structuredContent`` (or any individual field) leaves the corresponding ActionResult attribute ``None``, so callers and tests see unchanged behavior on old drivers. See the action response shape in cua-driver's mcp-tool-notes and NousResearch/hermes-agent#67052. keyrrrc`|vr|S|SNget)rrscs D/home/thesage/.hermes/hermes-agent/tools/computer_use/cua_backend.py_pickz"_action_result_from.._pickXs, "9966#;; xx}}verifiedNeffect escalationpathdegradedcode reason_code) ractionrrr'r(r)r*r+ delivery_moder,)rrrr) isinstancedictrrr )rrrrrrr%r'r(r)r*r+r,r/r#s ` @r$_action_result_fromr2Es""*d33 ;B uZ  H h % % U8__F fc " "|$$J j$ ' ' 5==D dC uZ  H h % % 5== 0EE-00D dC +55G*M*MW&&SWM    #     r&HERMES_CUA_DRIVER_CMD cua-drivermcp> full screenallscreendesktop fullscreen)progmanworkerwprogram manager shell_traywndtaskbarfinderr9dock)z@!Desktopz gnome-shellz GNOME ShellCUA_DRIVER_RS_TELEMETRY_ENABLEDcv ddlm}|pidpiS#t$ricYSwxYw)zGThe ``computer_use`` config block, or ``{}`` when config is unreadable.r load_config computer_use)hermes_cli.configrFr" ExceptionrEs r$_computer_use_cfgrJsa111111 #((88>B>  s &) 88ctd}|t|StjdkrdStjdkrdSt jdsdS tdd 5}d | vr ddddS dddn #1swxYwYn#t$rYnwxYwdS) uTrue when Hermes should pass ``--no-overlay`` to cua-driver. Reads ``computer_use.no_overlay``. Default ``None`` (auto-detect): disable the overlay where idle CPU burn is a known failure mode — macOS (cursor-overlay vImage redraw loop, #28152/#47032), headless Linux / WSL2 / containers — and keep it on Windows / desktop Linux with a display. Explicit ``True`` / ``False`` overrides auto-detection. no_overlayNdarwinTlinuxFDISPLAYz /proc/versionutf-8)encoding microsoft) rJr"rsysplatformosenvironopenreadlowerrI)valfs r$_cua_no_overlayr\sL    ! !, / /C Cyy |xt |wu :>>) $ $t /G 4 4 4 affhhnn....        .                     5s<9C *C4 C CCCCC C'&C'cbttdd S)uTrue when Hermes should disable cua-driver telemetry for this user. Reads ``computer_use.cua_telemetry`` (default False → telemetry off). Unreadable config falls SAFE toward disabling telemetry. cua_telemetryF)rrJr"r&r$_cua_telemetry_disabledr`s,%''++OUCCDD DDr& Optional[int]c ttdd}n#ttf$rYdSwxYw|dkr|ndS)uLongest-edge cap for cua-driver screenshots, or None to leave unset. Reads ``computer_use.max_image_dimension`` (default 1456, matching the aux-vision downscale). ``0`` / negative / non-numeric → None. max_image_dimensionirN)intrJr" TypeError ValueError)dims r$!_computer_use_max_image_dimensionrhsg #%%))*?FFGG z "tt''33t#s/2AAbase_envOptional[Dict[str, str]]Dict[str, str]cpt||n tj}tr d|t<|S)aReturn the environment dict for spawning cua-driver. Starts from ``base_env`` (defaults to ``os.environ``) and, when telemetry is disabled (the default), injects ``CUA_DRIVER_RS_TELEMETRY_ENABLED=0``. When the user has opted in, the var is left untouched so cua-driver uses its own default. Used by every cua-driver spawn site (MCP backend, status, doctor, install) so the policy is applied consistently. N0)r1rUrVr`_CUA_TELEMETRY_ENV_VAR)rienvs r$cua_driver_child_envrps: 8/xxRZ @ @C  *&) "# Jr&windowsList[Dict[str, Any]]cD|sdStd|DdkS)zETrue when every window shares the same z_index (common on Linux/X11).Tc:h|]}|ddS)z_indexrr!.0ws r$ z)_z_index_uninformative..s&555i##555r&)len)rqs r$_z_index_uninformativer|s1 t 55W555 6 6! ;;r&stdoutc|pd}tjd|}|stjd|}|sdS t|ddS#t$rYdSwxYw)zParse ``xprop -root _NET_ACTIVE_WINDOW`` stdout into a window id. Accepts the common ``window id # 0x...`` form and falls back to the first hex token. Returns None for empty/unparseable output. zwindow id # (0x[0-9a-fA-F]+)z(0x[0-9a-fA-F]+)Nrz)researchrdgrouprf)r}textmatchs r$_parse_xprop_net_active_windowrs >2&&& tts"A A'&A'c  tjdkstjdsdS t jgddddddd }n#t$rYdSwxYw|jd krdSt|j pd S) zCBest-effort read of ``_NET_ACTIVE_WINDOW`` via xprop. Never raises.rNrON)xpropz-root_NET_ACTIVE_WINDOWTrPreplaceF)capture_outputrrQerrorstimeoutcheckrr) rSrTrUrVr" subprocessrunrI returncoderr})procs r$_linux_x11_active_window_idr(s |wbjnnY&?&?t ~ 4 4 4      tt !t )$+*; < <z&_is_real_app_window..>sh  Bu{{}}77 BBr&)r"any_NON_APP_WINDOW_TITLE_PREFIXES)rxrs @r$_is_real_app_windowr;sS EE'2  E/ r&F) exact_target app_requestedrcd|D}|}|se|sctjdkrSd|D}|r|}|rAt|r2t}|"|D]}|d|kr|cS |r|dS|dS)uSelect the best window for capture from normalized list_windows output. Callers pass windows already sorted by ``z_index`` descending (higher = frontmost). When ordering is informative, keep that frontmost contract. For unqualified default captures (no app filter and no exact pid/window_id) on Linux, desktop/shell helper windows (GNOME ``ding`` "Desktop Icons", ``@!x,y;BDHF`` backdrop helpers) are skipped first — they are targetable X11 windows but capture as empty. Then, when every remaining candidate shares the same ``z_index`` (tied or unknown, the common Linux/X11 case), prefer ``_NET_ACTIVE_WINDOW`` over list order (#58026). Exact-target captures must not pay for an ``xprop`` probe. c"g|] }|d | S) off_screenr_rvs r$ z*_select_capture_target..Vs!<<<AlO.Ys&EEE1.A!.D.DEQEEEr&N window_idr)rSrTr|rr")rqrr candidatespool real_apps active_idrxs r$_select_capture_targetrDs$=.|s((M(MtT(M(M(M(M(M(Mr&rzN) rrhermes_constantsrrIrdriverstriprYrUr*joinparts)r*rwinrs r$_wsl_windows_path_to_posixrgs 8& - - ++++++vxx K    $  C Y_" $ $S ) ) / / 1 1E  7< N(M(Msy}(M(M(M N N NNs , ;;cFeZdZdZdZddZdd Zdd ZddZddZ ddZ dS)_EmbeddedCuaDaemona}Private host-owned daemon used for an explicit unrestricted session. Cua Driver permission mode is immutable after daemon startup. Reusing the machine-wide daemon would therefore let one Hermes session's YOLO choice affect another session. A private embedded daemon gives the requesting session its own socket, process, and launch-time risk acknowledgement. .@ driver_cmdrpermission_moderNonec|dkrtd||_||_||_t t |_d|_td|_ d|_ tj j dd}tjdkr d||_dSt"jt)jd|d |_dS) N unrestrictedz7embedded permission override supports unrestricted only)maxlen win32z\\.\pipe\hermes-cua-zhc-z.sock)rfr _driver_cmd_commandlist_CUA_DRIVER_ARGS _mcp_args_processr _stderr_tail_stderr_threaduuiduuid4hexrSrT socket_pathrUr*rtempfile gettempdir)selfrrtokens r$__init__z_EmbeddedCuaDaemon.__init__s n , ,VWW W.%" $()9$:$:! (-R(8(8(8:>  "% <7 " ">u>>D   !w||#%%'9U'9'9'9  D   r&rkc6t}d|d<d|d<|S)NrCUA_DRIVER_PERMISSION_MODE1'CUA_DRIVER_DANGEROUSLY_BYPASS_APPROVALS)rp)rros r$ child_envz_EmbeddedCuaDaemon.child_envs("$$,: ()9< 56 r&processrct|dd}|dS |D]Z}t|}|r5|j|t d|[dS#t$rYdSwxYw)Nstderrzembedded cua-driver: %s)getattrrstriprappendloggerdebugrI)rrstreamliners r$ _drain_stderrz _EmbeddedCuaDaemon._drain_stderrs(D11 > F  B B4yy((B%,,T222LL!:DAAA  B B     DD sAA66 BBc |j|jdSddlm}|jst pd|_|jst tt|j\|_ |_ || }|j ddd|j ddd d g }tj|tjtjtjd | |_t#j|j|jfd d |_|jt-j|jz}t-j|kr|j.d|jpd}t d| tj|j dd|j gtjd d d|}n#t8tjf$rd}YnwxYw| |jdkrdSt-jdt-j|k| d|jpd}t d|)Nr_sanitize_subprocess_envrserve --embedded--socketz--no-permissions-gatez--permission-moderz--dangerously-bypass-approvalsT)stdinr}rrrozhermes-cua-daemon-stderr)targetargsrdaemonz; zno diagnostic outputz+embedded cua-driver exited during startup: status@)rrrrrog?zdaemon did not become readyz'embedded cua-driver startup timed out: )!rpolltools.environments.localrrresolve_cua_driver_cmd RuntimeErrorcua_driver_install_hint_resolve_mcp_invocationrrrrrPopenDEVNULLPIPE threadingThreadrrstarttime monotonic_START_TIMEOUT_SECONDSrrrOSErrorSubprocessErrorrsleepstop)rrrocommanddeadlinedetailprobes r$rz_EmbeddedCuaDaemon.starts = $););)=)=)E FEEEEEE >577=2D  :68899 9(?@P(Q(Q% t~&&t~~'7'788 M      #   ,  #( $%?     (.%-!+     !!###>##d&AAn))}!!##/4#455O9O"J&JJ "]Hj$:JK$,#' Z78     U%5%:%: JsOOO'n))* 4,--N1NMVMMNNNs22G%%H?HTuple[str, List[str]]c|j|jtd|jg|jdd|jfS)Nz)embedded cua-driver daemon is not runningrr)rrrrrrrs r$proxy_invocationz#_EmbeddedCuaDaemon.proxy_invocationsi = DM$6$6$8$8$DJKK K} ^          r&c R|j}d|_|2|ddlm} t j|jdd|jgtjtjtjd|| n#ttj f$rYnwxYw | dn#tj $rn| | d n?#tj $r-|| d YnwxYwYnwxYwt jd krQt$j|jr/ t%j|jdS#t$rYdSwxYwdSdS) Nrrrr@)rr}rrro@rrr)rrrrrrrrrrrrwaitTimeoutExpired terminatekillrSrTrUr*existsremove)rrrs r$rz_EmbeddedCuaDaemon.stops-  7<<>>#9 I I I I I I ]FJ8HI$,%-%-001A1ABB Z78     . S )))), . . .!!###.LLL----!0...LLNNNLLL-----.  . <7 " "rw~~d6F'G'G "  $*+++++     # " " "sZA!BB)(B)-C#E(C?>E?9D;8E:D;;EE9F F"!F"N)rrrrrr)rrk)rrrrrr)rr) __name__ __module__ __qualname____doc__rrrrrr rr_r&r$rrs"$    >O>O>O>O@    r&rg@rrrfloatrc6 ddlm}tj|dgdddd|tjt |t  }n5#t$r(|ttt|fcYSwxYw|j pd }|j dks|s%|ttt|fS tj|}n<#t t"f$r(|ttt|fcYSwxYwt%|t&s%|ttt|fS|d }t%|t&s%|ttt|fS|d }|d }t%|trt+d |Ds%|ttt|fSt%|t,r|s|t||fSt/|}t1|s|t||fS|t||fS)uReturn ``(command, args)`` that spawn cua-driver's stdio MCP server. Surface 8 of NousResearch/hermes-agent#47072: instead of hardcoding ``["mcp"]`` we ask the driver itself via ``cua-driver manifest`` (trycua/cua#1961). The manifest carries a stable ``mcp_invocation`` pointer with both ``command`` and ``args``, so a future cua-driver that renames or relocates the subcommand keeps working without a Hermes patch. Falls back to ``(driver_cmd, ["mcp"])`` for older drivers that don't expose ``manifest``, or any indeterminate failure — the wrapper must not refuse to start just because the discovery hop failed. When ``computer_use.no_overlay`` is enabled (or auto-detected on Linux), ``--no-overlay`` is appended to suppress the cursor overlay rendering loop that can consume CPU indefinitely when idle (#28152, #47032). Older drivers that don't recognise the flag will reject it; callers should fall back to the no-overlay invocation on spawn failure. rrmanifestTrPrrrrQrrr creationflagsro)rrmcp_invocationrrc3@K|]}t|tVdSr r0r)rwas r$rz*_resolve_mcp_invocation..Os,,N,NAZ3-?-?,N,N,N,N,N,Nr&)rrrrrr rprI_mcp_args_with_overlay_flagrrr}rrjsonloadsrfrer0r1r"r7rr_has_path_separator) rrrroutr invocationrrs r$rrs2 fEEEEEE~  $dWYX_$,..)()=)?)?@@     fff6t>& ! !DnnY''G dD ! !f,N,N,N,N,N)N)Nf6t ' --help`` once and caches the result. Older drivers (< 0.6.x) reject unknown flags, so passing ``--no-overlay`` would crash the MCP spawn. rrz--helpTrPrr rrr,F) rrrrrr rpr}rrI)rrr help_texts r$r-r-ps FEEEEE~  "dWYX[$,..(()=)?)?@@    [&B4;+<"= ** uusA&A)) A76A7z~^\s*(?:-\s+)?\[(\d+)\]\s+(\w+)(?:\s*=\s*"([^"]*)"|\s+"([^"]*)"|\s+\((?!\d+\))([^)]*)\))?(?:\s+(?:\(\d+\)\s+)?id=([^\s\[\]]+))?c"tjdkS)NrMrSrTr_r&r$ _is_macosr3s <8 ##r&valuecVtj|vptjduo tj|vSr )rUsepaltsep)r4s r$r'r's& 6U? Lry4Ke9KLr&overridec V||n$tjtd}|r|gSt g}tjd}tj dkrX| tj |dddtj |dddgnY| tj |dddtj |d ddd d g|S) aReturn candidate cua-driver commands in resolution order. ``override`` is authoritative when supplied. Otherwise a non-empty ``HERMES_CUA_DRIVER_CMD`` is authoritative; only when neither is set do we use PATH and canonical install locations. Desktop apps launched from Finder/Dock often inherit a narrow PATH that omits user-local install directories. The upstream cua-driver installer commonly places the binary under ``~/.local/bin`` on POSIX systems, so a Hermes Desktop/TUI session can otherwise filter out the `computer_use` tool even though `hermes computer-use doctor` succeeds from a login shell. Nr~rz.localbinzcua-driver.exer4z.cargoz/opt/homebrew/bin/cua-driverz/usr/local/bin/cua-driver) rUrVr"_CUA_DRIVER_CMD_ENVr_CUA_DRIVER_DEFAULT_CMDr* expanduserrSrTextendr)r8 configuredrhomes r$_candidate_cua_driver_commandsrBs'2(( GZ\^8_8_ffhhJ|)*J 7  c " "D |w GLLx0@ A A GLLx = =      GLLx = = GLLx = = * '     r&ct|D]c}tj|}t |rt j|r|cSIt j|}|r|cSddS)a%Resolve the cua-driver executable for every runtime/status surface. A supplied override (or ``HERMES_CUA_DRIVER_CMD``) is never silently replaced by another binary. Otherwise resolve PATH first, then canonical user-local installation locations used by the official installer. N)rBrUr*r>r'shutilwhich)r8 candidateexpandedresolveds r$rrs4H==  7%%i00 x ( ( |H%%  |H--H  4r&c"tduS)zDTrue if `cua-driver` resolves via env, PATH, or known install paths.N)rr_r&r$cua_driver_binary_availablerJs ! # #4 //r&Optional[float]Optional[Dict[str, Any]]c|tjdkrdnd}t}|sdS ddlm}t j|ddgd d d d |t jt|t }n#t$rYdSwxYw|j pd }|sdS tj|}n#tt f$rYdSwxYwt#|t$r|drdS|S)u^Run ``cua-driver check-update --json`` and return its parsed state. The payload mirrors the ``check_for_update`` MCP tool: ``{current_version, latest_version, update_available, ...}``. ``timeout`` defaults to 8s on POSIX and 25s on Windows — first-spawn of the exe there routinely eats several seconds in Defender/SmartScreen scanning, and a false timeout is expensive: callers treat ``None`` as indeterminate, and the ``install_cua_driver(upgrade=True)`` path used to fall through to a full multi-minute reinstall on it. Returns ``None`` (callers should stay quiet) when the result is indeterminate: the binary is missing, the driver is too old to support the verb (it predates trycua/cua#1734), the GitHub check failed (an ``error`` field is set), or the output didn't parse. Best-effort; never raises. Nrg9@g @rrz check-updatez--jsonTrPrrrerror)rSrTrrrrrrr rprIr}rr%r&rfrer0r1r")rrrrr(datas r$cua_driver_update_checkrPsV$,'11$$s'))J tEEEEEE~  2dWYX_$,..)()=)?)?@@     tt ; " # # % %C tz#  "tt dD ! !TXXg%6%6t Ks$AA<< B  B -CCCct}|r|dsdS|dpd}|dpd}d|d|dS) z~One-line "an update is available" message, or ``None`` when up to date, indeterminate, or the driver is too old to report.update_availableNlatest_version?current_version cua-driver z is available (you have z7); update with `hermes computer-use install --upgrade`.)rPr")statelatestcurrents r$cua_driver_update_nudgerZ%s $ % %E  "455t YY' ( ( /CFii)**1cG @f @ @g @ @ @r&rcvtrdSdadd}tj|dddS) zEmit an update nudge at most once per process, off-thread so the (cached, ~20h) GitHub poll never blocks the first computer_use action.NTrrc t}n#t$rYdSwxYw|rtd|dSdS)Nzcomputer_use: %s)rZrIrinfo)msgs r$_runz!_maybe_nudge_update.._run>sc )++CC    FF   1 KK*C 0 0 0 0 0 1 1s  zcua-driver-update-check)rrrr)_update_checkedrrr)r_s r$_maybe_nudge_updatera6sXO11113D egggggr&c8tjdkrd}nd}d|dS)Nrza irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iexzs /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)"zxcua-driver is not installed. Install with one of: hermes computer-use install Or run the upstream installer directly: zW Or run `hermes tools` and enable the Computer Use toolset to install it automatically.r2) installers r$rrKsG |w 8   3   a  a a ar&markdownList[UIElement]c g}t|D]}|dp@|dp+|dp|dpd}|t t |d|d|d |S) uParse UIElement list from get_window_state AX tree markdown. Last-resort fallback for cua-driver builds that don't carry the canonical ``structuredContent.elements`` array (see ``_parse_elements_from_structured`` — Surface 2 of #47072 prefers that path). Captures the label whichever form cua-driver used: ``= "value"``, ``"quoted"``, ``(parenthesised)``, or ``id=Label``. Bounds always come back ``(0, 0, 0, 0)`` because the markdown surface doesn't carry them — yet another reason to prefer the structured path; element-index clicks don't need them (the driver resolves the index to a frame internally). rrzrrrrr)indexrolelabelbounds)_ELEMENT_LINE_REfinditerrrrrd)rdelementsmrns r$_parse_elements_from_treert`sH  & &x 0 0   JaggajjJAGGAJJJ!''!**J aggajj//        Or& raw_elementsc g}|D]}t|ts|d}t|tsDt|dtr|dnd}t|dtr|dnd}t|dtr|dnd}d}|r t|dd t|d d t|d d t|d d f}n#t t f$rd}YnwxYw|d }t|tr|r|nd} |t||||| |S)uGSurface 2 of NousResearch/hermes-agent#47072: read the canonical ``structuredContent.elements`` array cua-driver-rs emits on every ``get_window_state`` response (trycua/cua#1961). Each entry has at minimum ``element_index``, ``role``, ``label``; ``frame`` (``{x, y, w, h}``) is included whenever the AT-SPI / AXFrame call returned usable bounds. Older code parsed the same information out of the markdown tree via a regex (lossy: bounds were always ``(0, 0, 0, 0)``) — this path preserves the real frame so downstream consumers (e.g. ``UIElement.center()``) work against pixel coordinates instead of just the index lookup. Unknown / malformed entries are skipped rather than failing the whole walk — the wrapper degrades to "fewer elements" rather than "no elements" on a bad row. element_indexrmrrnframeNrkxryrxh element_token)rlrmrnror|) r0r1r"rdrrerfrr) rurrrawidxrmrnrxro raw_tokenrs r$_parse_elements_from_structuredr|s"!#H  #t$$  ggo&&#s##  ",SWWV__c"B"BJswwv$.swww/?/?$E$EM   2$.swww/?/?$F$FP   D,8  & & #q))** #q))** #q))** #q))**  z* & & &% & GGO,, ' 377OIO 4          Os B FF,+F,r}bytesTuple[int, int]cF|drit|dkrVt|ddd}t|ddd}|dkr |dkr||fS|dr d}t|}|d z|kr||d kr|d z }||d z}|dz }|d vsd |cxkrdkrnn@|dz|krnt|||dzd}|dks ||z|krn{|dvri|dkrbt||dz|dzd}t||dz|dzd}|dkr |dkr||fSn||z }|d z|kdS)zCBest-effort PNG/JPEG dimension sniffing without extra dependencies.sPNG  rrbigrsr rz>> rgri)rr)rr{rd from_bytes)r}widthheightinmarker segment_lens r$_image_dimensions_from_bytesrs ~~*++!CBs2b5z511BrE E22 199!&=  ~~k""  HH!eaii1v~~QQZF FA%%)?)?)?)?4)?)?)?)?)?1uqyy..Qq1uWu==KQ!k/A"5"5!## ^^CAa!e ,fnaltcmdctrlshiftoptionrcontrolrrr)rrrcg|]<}||=Sr_)rrY)rwrs r$rz$_parse_key_combo..s7 N N N1AGGII NQWWYY__   N N Nr&z[+\-]N)rrr"r)rMODIFIER_NAMES KEY_ALIASESr modifiersrr normalizeds r$_parse_key_combors [ZZN#HHHK N N4(@(@ N N NEI C __T400  ' '   Z ( ( ( (CC  >r&c4eZdZdZd dZd dZddd Zd d Zd S) _AsyncBridgezNRuns one asyncio loop on a daemon thread; marshals coroutines from the caller.rrcRd|_d|_tj|_dSr )_loop_threadrEvent_readyr s r$rz_AsyncBridge.__init__s#:> 37 o'' r&cHjrjrdSjd fd }t j|dd_jjdstd dS) Nrrctj_tjjj j jdS#t$rYdSwxYw# jw#t$rYwwxYwxYwr ) asyncionew_event_looprset_event_looprset run_forevercloserIr sr$r_z _AsyncBridge.start.._runs /11DJ  "4: . . . KOO     &&(((J$$&&&&& DDJ$$&&&& DsB B'B BBB?B/.B?/ B<9B?;B<<B?Tzcua-driver-loop)rrrrrz)cua-driver asyncio bridge failed to startr) ris_aliverclearrrrrr)rr_s` r$rz_AsyncBridge.starts < DL1133  F       !'tDGXYYY  {,, LJKK K L Lr&>@rrKrc:ddlm}|jr |jr|js7t j|r|td|||j}|td| |S)Nr)safe_schedule_threadsafecua-driver bridge not startedr) agent.async_utilsrrrrr iscoroutinerrresult)rcororrfuts r$rz_AsyncBridge.runs>>>>>>z @ @T\5J5J5L5L @"4((  >?? ?&&tTZ88 ;>?? ?zz'z***r&c|jr=|jr$|j|jj|jr|jdd|_d|_dS)Nrr)r is_runningcall_soon_threadsaferrrr s r$rz_AsyncBridge.stop"sp : =$*//11 = J + +DJO < < < < + L  c  * * *  r&Nrr)rrKrr)rrrrrrrrr_r&r$rrssXX(((( LLLL, + + + + +r&rcLeZdZdZ d4d5d Zd6d Zd6d Zd7dZd6dZd6dZ d6dZ d6dZ d6dZ d8dZ d4d9dZd:dZd;d Zedd$Zed?d%Zd@d)ZedAd,ZedAd-Zd6d.ZdBd/Zed0d1hZdCdBd3ZdS)D_CuaDriverSessionuHolds the mcp ClientSession. Spawned lazily; re-entered on drop. Lifecycle ownership: a single long-running coroutine (`_lifecycle_coro`) opens both the stdio_client and ClientSession contexts, populates capabilities, sets `_ready_event`, and then waits on `_shutdown_event`. When shutdown is signalled the same coroutine closes the contexts — keeping anyio's cancel-scope task-identity invariant intact (the bridge schedules each `bridge.run(coro)` as a NEW task, so opening contexts in one and closing them in another raises "Attempted to exit cancel scope in a different task"). Tool calls run in their own short-lived tasks; they only touch the session object, never the surrounding contexts. Nbridgerembedded_daemonOptional[_EmbeddedCuaDaemon]rrc||_||_d|_tj|_d|_i|_i|_d|_ tj |_ d|_ d|_ d|_d|_dS)NFr)_bridge_embedded_daemon_sessionrLock_lock_started _capabilities _tool_schemas_capability_versionr _ready_event_shutdown_event_lifecycle_future _setup_error_declared_session_id)rrrs r$rz_CuaDriverSession.__init__>s  / ^%%  .0 9;(* %O--8<!%5948!!!r&c2|jstddS)Nzcua-driver session not started)rrr s r$_require_startedz"_CuaDriverSession._require_started`s(} A?@@ @ A Ar&c FKddl}ddlm}m}ddlm}ddlm}tj |_ | }d|_ t}|sttd|_ |j6|j\}} |j} n t'|\}} t)} | } ||| || } || 4d{V\} }d |_ || |4d{V}|d{V| }d |_ ||d{V||_d |_ |jt4d | |z | |z || z |j d{Vdddd{Vn#1d{VswxYwYdddd{Vn#1d{VswxYwYn3#t:$r&}||_|jd}~wwxYwd|_d |_dS#d|_d |_wxYw)u}Long-lived owner of the stdio MCP contexts. Opens, signals ready, blocks on shutdown, then cleans up. enter + exit happen in the SAME asyncio task, so anyio's cancel-scope invariant holds — fixing the "Attempted to exit cancel scope in a different task than it was entered in" warning emitted by the previous _aenter/_aexit split. rN) ClientSessionStdioServerParameters) stdio_clientrz binary-checkzmanifest-discovery)rrrozmcp-initializezcapability-discoveryreadyzBcua-driver session ready in %.1fs (manifest=%.1fs, mcp_init=%.1fs)F) rr5rrmcp.client.stdiorrrrrrr_startup_phaserrrrr rrrp initialize_populate_capabilitiesrrrrr]r BaseExceptionrr)r_timerrrr_t0rrrr _t_manifestparamsrXwritesession_t_inites r$_lifecycle_coroz!_CuaDriverSession._lifecycle_corods/ <<<<<<<<111111EEEEEE '}oo-F "/11J >"#:#<#<=== #7D $0 $ 5 F F H H  1;;== 7 C C 022 //++K**-,Y77 F$|F++ 6 6 6 6 6 6 6}e&6#(=u556666666!,,.........#oo//G+AD'55g>>>>>>>>>$+DM*1D'%))+++KK;))C/#c)+- .33555555555+666666666666666666666666666 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 60   !"D    ! ! # # #   !DM"DMMM!DM"DM ! ! ! !stC I H;7CHH; H" "H;%H" &H;)I ; II I I J I=!I88I==JJ rrcKi|_i|_d|_ |d{V}t |dgpgD] }t |dd}t |t s*t |dd}|(t |ddpi}|d}t |trd|D|j|<nt|j|<t |dd}|&t |ddpid}t |trt|ni|j|<t |d d}|(t |ddpi}|d }t |t r ||_dSdS#t$r&} t d | Yd} ~ dSd} ~ wwxYw) uSurface 4: cache per-tool capability sets + capability_version from tools/list. Soft prerequisite — discovery failure leaves the map empty and supports_capability degrades to False.rNtoolsr capabilities model_extrac<h|]}t|t|Sr_r")rwcs r$ryz;_CuaDriverSession._populate_capabilities..s7555:a+=+=5555r& inputSchemacapability_versionz5cua-driver tools/list capability discovery failed: %s)rrr list_toolsrr0rr"rrr1rIrr) rr tools_listtool tool_namecapsextraschemacvrs r$rz(_CuaDriverSession._populate_capabilitiess? #% % U&1133333333J GR88>B  #D&$77 !)S11t^T::<$D->>D"E 99^44DdD)):55#'555D&y1158EED&y1 }d;;>%dM4@@FBKK%F%/vt$<$<DDLLL""9--%94@@Bz M4@@FBYY344"c"" .+-((( . . U U U LLPRS T T T T T T T T T UsFF// G9GGc|j5|jr ddddS|j|d|_ddddS#1swxYwYdS)NT)rrrr_start_lifecycle_lockedr s r$rz_CuaDriverSession.starts Z ! !}  ! ! ! ! ! ! ! ! L    ( ( * * * DM  ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !s A4AA#&A#ctj|_d|_d|_|jj}|tdtj | ||_ |j dsI| t|dd}ddlm}td |d |d |jtd |j|jdS) z^Spawn the lifecycle owner and wait for it to reach ready. Caller must hold self._lock.Nrrrrunknownr)display_hermes_homezEcua-driver session never reached ready (timeout 30s; stuck in phase: z.). Run `hermes computer-use doctor` and check z&/logs/agent.log for the phase timings.z!cua-driver session setup failed: )rrrrrrrrrrun_coroutine_threadsaferrr_signal_shutdown_lockedrrr)rloopphasers r$rz)_CuaDriverSession._start_lifecycle_lockeds@&O-- #|! <>?? ?!(!A  " "D" "  %%d%33   ( ( * * *D"2I>>E < < < < < <Q#(QQ'&((QQQ    (GD4EGG$ % ) (r&c|j5|js ddddSd|_|ddddS#1swxYwYdS)NF)rr_stop_lifecycle_lockedr s r$rz_CuaDriverSession.stops Z * *=  * * * * * * * *"DM  ' ' ) ) )  * * * * * * * * * * * * * * * * * *s AAA  A c\||j}|dS |dna#tjj$rt dYn1t$r%}t d|Yd}~nd}~wwxYwd|_dS#d|_wxYw)zbSignal shutdown + wait for the lifecycle coroutine to unwind. Caller must hold self._lock.Nrrz*cua-driver session shutdown timed out (5s)zcua-driver shutdown error: %s) rrr concurrentfutures TimeoutErrorrwarningrI)rrrs r$rz(_CuaDriverSession._stop_lifecycle_locked$s $$&&&$ ; F * JJsJ # # # #!. I I I NNG H H H H H ? ? ? NN:A > > > > > > > > ? &*D " " "TD " ) ) ) )s88B".B&B"( B1B B"BB"" B+c|jj}|j}|D|D|r2 ||jdS#t $rYdSwxYwdSdSdS)z8Set the asyncio shutdown event from the caller's thread.N)rrrrrrr)rrevents r$rz)_CuaDriverSession._signal_shutdown_locked7s|!$   1doo6G6G 1 ))%)44444       1 1 1 1sA AArrrrcfK|j||d{V}t|Sr )r call_tool_extract_tool_result)rrrrs r$_call_tool_asyncz"_CuaDriverSession._call_tool_asyncBs=}..tT::::::::#F+++r& capabilityr rrc|)|j|tvStfd|jDS)uReturn True when the connected cua-driver advertises the given capability token (trycua/cua#1961 capability vocabulary). When ``tool`` is given, scope the check to that specific tool's advertised capability set. When omitted, return True if ANY tool advertises the capability — useful for "is this feature available anywhere on the driver" probes. Always returns False before the session is started (so consumers on a dead/uninitialised wrapper degrade rather than crash). Nc3 K|]}|vV dSr r_)rwrr(s r$rz8_CuaDriverSession.supports_capability..Us(NN$:%NNNNNNr&)rr"rrvalues)rr(r s ` r$supports_capabilityz%_CuaDriverSession.supports_capabilityGs_  !3!7!7cee!D!DD DNNNN$2D2K2K2M2MNNNNNNr&c||jvS)umReturn True when ``tools/list`` advertised a tool by this name. Used to route capture(): cua-driver dropped the standalone ``screenshot`` tool and folded full-window PNG capture into ``get_window_state`` (whose own description notes it "Also captures a PNG screenshot of the specified window"). Older drivers that still expose ``screenshot`` keep using it; newer ones fall through to ``get_window_state``. Returns False when discovery hasn't populated the map yet — callers treat that as "unknown" and probe defensively rather than trusting it. )r)rrs r$ _has_toolz_CuaDriverSession._has_toolWst)))r& property_namect|di|i}t|tr|dnd}t|to||vS)aReturn whether a live action schema accepts ``property_name``. This deliberately inspects tools/list rather than guessing from the package version or requiring a capability token the driver never shipped. A missing/invalid schema fails closed. r propertiesN)rr"r0r1)rr r/rr1s r$supports_input_propertyz)_CuaDriverSession.supports_input_propertyfsd3377bAA1;FD1I1ISVZZ ---t *d++K 0KKr&c*t|jS)zTrue once ``tools/list`` populated the per-tool map. When False, ``_has_tool`` answers are not trustworthy (discovery failed or the session hasn't started) and capture() should probe defensively.)rrr s r$capabilities_discoveredz)_CuaDriverSession.capabilities_discoveredqs D&'''r&c|jS)uDriver-advertised capability vocabulary version (empty string when the driver predates the field — older builds had no version).)rr s r$r z$_CuaDriverSession.capability_versionxs ''r&rcg}|d|dfD]}t|tr||-|d |t j|dZ#t tf$r%|t|YwxYwd|S)z@Flatten a logical MCP error into text for narrow classification.rOstructuredContentNT) sort_keysr) r"r0rrr%dumpsrerfr)rchunksr4s r$_logical_error_textz%_CuaDriverSession._logical_error_text~sjj((&**5H*I*IJ . .E%%% . e$$$$".MM$*Ud"C"C"CDDDD!:....MM#e**-----.# yy   s)B3B=<B=ct|tr|ddurdS||}d|vo d|vpd|vod|vS)zARecognise cua-driver's explicit recoverable ended-session result.isErrorTFrz has endedz session ended start_session)r0r1r"r;rY)clsrrs r$_is_ended_session_resultz*_CuaDriverSession._is_ended_session_results&$'' 6::i+@+@+L+L5))&117799   +'E?g+E +7* r& first_resultrrc|j}|r ||jvr|Std|||j|dd|i|}|ddur1td||||S|j||||S)zARevive the stable session and replay one rejected tool call once.zAcua-driver session %s ended during %s; reviving and retrying oncer>rrr=Tz.cua-driver session %s could not be revived: %s) r_LIFECYCLE_CALLSrr!rrr'r"r;)rrrrAr session_id revive_results r$_revive_declared_session_oncez/_CuaDriverSession._revive_declared_session_onces.  TT%:::  O      ((  ! !/Iz3J K K)     Y ' '4 / / NN@((77     |  ! !$ - -    r&excrIc|jj}t|jdd}|dvp4|dod|vpt |t t fS)zHReturn True for MCP/stdio failures that are recoverable by reconnecting.rr> EndOfStreamBrokenResourceErrorClosedResourceErroranyioResource) __class__rrrr0BrokenPipeErrorEOFError)rGrmodules r$_is_closed_session_errorz*_CuaDriverSession._is_closed_session_errorsf}% b99 Q Q <!!'**AzT/A <#:;; r&c@t|}d|vp d|vpd|vpd|vS)uReturn True for the cua-driver daemon-proxy EAGAIN congestion error. On macOS the ``cua-driver mcp`` bridge forwards calls to the CuaDriver daemon over a non-blocking unix socket. Heavier ops (notably ``get_window_state``, which walks the AX tree and captures a PNG) can come back as an ``McpError`` carrying ``Resource temporarily unavailable (os error 35)`` — POSIX EAGAIN — when the socket buffer is momentarily full. This is transient by definition: the same call succeeds when retried after a short pause (which is why spaced-out single calls work while rapid/large ones intermittently fail). Detect it by message so we can retry with backoff rather than surfacing an empty 0x0 capture to the model. See the EAGAIN diagnosis in references/catalog-add-troubleshooting (apple-music skill) and the cua-driver daemon-proxy note. z Resource temporarily unavailablez os error 35zdaemon transport errorz daemon proxyr)rGr^s r$_is_transient_daemon_errorz,_CuaDriverSession._is_transient_daemon_errorsE"#hh .# 5 %# %'3. %$  r&c|jrH |n2#t$r%}td|Yd}~nd}~wwxYwd|_i|_d|_|d|_dS)zRecreate the MCP session after the daemon/stdin transport was closed. Caller must hold self._lock (the reconnect-once retry path holds it).z6cua-driver session cleanup before reconnect failed: %sNFrT)rrrIrrrrrrrs r$_restart_session_lockedz)_CuaDriverSession._restart_session_lockeds = Z Z++---- Z Z Z UWXYYYYYYYY Z #%  $$&&& s A AA c  ddl}ddl}ddl}ddlm}t |}d} |dkr7d|vr3|dd\} } tj| | |d<t} | sttt} g} t|d d}|7|d} |} d |jg} | d |t#j|g| }d }d }d}d} t'|D]k} ||ddddt+d|t-|| }n(#t.$r}td|d||d}~wwxYw|jpd}|ddp|jpddd}t7d|d|dfDd}|dkr; t#j||d}n#t"j$rd}YnwxYw||}nF||dz kr;t> d||dz|||!||dz}m|td|d|d |g}d}tE|t r|nd}d!}tE|t rj|#d"dup|#d#du}|#d$}|s|#d%p| }|rtj$%|r tM|d&5} tOj(| )*d'}dddn #1swxYwYn3#t.$r&}t>+d(||Yd}~nd}~wwxYw|r|,||#d)}!|!)|#d*}"|"|"d+nd}#|#r|#d,|!n|!}||||d-| rEtj$%| r' tj-| S#t\$rYSwxYwSS#| rEtj$%| r' tj-| w#t\$rYwwxYwwwxYw).uFallback transport: invoke ``cua-driver call `` as a subprocess instead of going through the stdio MCP bridge. The ``cua-driver mcp`` stdio bridge can persistently fail to forward heavier calls (notably ``get_window_state``) to the daemon with POSIX EAGAIN, while the plain ``cua-driver call`` path — which talks to the daemon over its own socket — keeps working. When the MCP path gives up, we retry over the CLI and remap the JSON into the same dict shape that ``_extract_tool_result`` produces, so callers (capture(), _action(), list_windows parsing) are transport-agnostic. For ``get_window_state`` we route the screenshot to a temp file via ``screenshot_out_file`` so the daemon returns a tiny JSON body (a path) instead of a multi-megabyte base64 blob — the large payload is what congests the daemon socket and triggers EAGAIN in the first place. We read the PNG back from disk and base64-encode it ourselves. The CLI call is itself retried a few times with backoff, since the underlying daemon socket can still be momentarily busy. rNrget_window_statescreenshot_out_file cua_shot_z.png)prefixsuffixrrcallrhg?rTrPrr)rrrQrrrrozcua-driver CLI fallback for z failed to spawn: c3&K|] }|dk|V dS)Nr_)rwrs r$rz7_CuaDriverSession._call_tool_via_cli..1s&JJ1!r''Q''''JJr&{[ra)defaultrzzMcua-driver CLI fallback for %s got no JSON (attempt %d/%d); retrying in %.1fsrz returned no JSON after z attempts: Fr=is_errorscreenshot_png_b64screenshot_file_pathrbasciiz.cua-driver CLI fallback: failed reading %s: %s tree_markdown element_countz elementsr)rOimagesr7r=)/rrrrrr1mkstemprUrrrrrprr rrr%r9rangermaxr rIr}rrminfindr&JSONDecodeErrorrr!rr0r"r*rrWbase64 b64encoderXdecoderrrr)$rrrr _subprocess _tempfilerr call_args shot_filefddriver_commandr socket_argsrrattemptsbackoffparsedlast_errattemptrrr(rrFrlrOrreshotfpathfhrecrs$ r$_call_tool_via_cliz$_CuaDriverSession._call_tool_via_clisu( )((($$$$EEEEEEJJ #' % % %*?y*P*P%--[-PPMB HRLLL/8I+ ,/11 :68899 9(** !# !$(:DAA  &,==??BN'1133I%'BCK    Jy ! !    O  ?? ! !j&??DtgV_ilmqszi{i{&8&:&:44Y??+DD !jjj&'ad'a'a^_'a'abbhiij{(b//11tt9A):DSD(AJJ# >JJJB;;)$(Js566{$;$; /)))$( ) ,!*X\))NN=gk8W KK(((qLG~"747777,477 !#FD3=fd3K3K)UQUJH&$'' E"::i00D8ZFJJz>K)Ee!6!6ee!%eT!2!2Sb'-'7 'B'B'I'I''R'RSSSSSSSSSSSSSSS(eee"LL)Y[`bcdddddddde(MM$'''zz/22#O44B24.....bG3:Dg/////D %/#  RW^^I66 Ii((((D  y RW^^I66 Ii((((D  s R AER  F'E==FBR H"!R "H63R 5H66DR N4":N( N4(N,,N4/N,0N43R 4 O$>OR O$$A R &Q;; RR "S0SS SSSSr> end_sessionrc~|js8||jvr/td||| |j||||}n#t$r}| |r8td||| |||cYd}~S| |std||j 5|dddn #1swxYwY|j||||}Yd}~nd}~wwxYw|dkrJ|ddur3|d}t!|t"r |r||_||r|||||}|d kr<|ddur%|d|jkrd|_|S) Nz=cua-driver session not active on %s; (re)starting before callrzIcua-driver MCP transport failed on %s (%s); falling back to CLI transportz:cua-driver MCP session closed during %s; reconnecting oncer>r=Trr)rrCrr!rrrrr'rIrTrrRrrWr"r0rrr@rF)rrrrrr declared_ids r$r%z_CuaDriverSession.call_toolys} T-B!B!B NNOQU    JJLLL  \%%%%dD11&FF   ..q11 D4591..tT7CCCCCCCC0033  NNWY] ^ ^ ^ / /,,... / / / / / / / / / / / / / / /\%%%%dD11&FFFFFF & ? " "vzz)'<'3E66E;r )rrrrrrr)rrrrrrrrrr)r(rr rrr)rrrr)r rr/rrrrrrr)rrrr)rrrr) rrrrrArrrrr)rGrIrr)rrrrrrrrr) rrrrrrrrrrrrrr'r,r.r2propertyr4r  staticmethodr; classmethodr@rFrRrTrWr frozensetrCr%r_r&r$rr/s|  "9= 8 8 8 8 8DAAAA\"\"\"\"|,U,U,U,U\!!!!#%#%#%#%J********&    ,,,, OOOOO * * * * L L L L(((X( (((X(  ! ! !\ !    [  ! ! ! ! F   \    \ 0    DDDDR!y/=!ABB0000000r&r mcp_resultrcd}g}g}t|dddu}t|ddpd}g}t|dgpgD]}t|dd}|dkr'|t|dd pd @|d krPt|d d} | r=|| t|d dpd } || |rud d|D} | drt j| n| }n#t j$r| }YnwxYw|||||dS)uConvert an mcp CallToolResult into a plain dict. cua-driver returns a mix of text parts, image parts, and structuredContent. We flatten into: { "data": , "images": [b64, ...], "image_mime_types": [mime, ...], # parallel to `images`, "" when absent "structuredContent": , "isError": bool, } structuredContent is populated from the MCP result's structuredContent field (MCP spec §2024-11-05+) and takes precedence for structured data like list_windows window arrays. `image_mime_types` is the explicit `mimeType` cua-driver emits on every image part as of trycua/cua#1961 (Surface 7 of NousResearch/hermes-agent#47072). Each entry corresponds index-for-index with `images`; an empty string entry signals the part carried no mimeType (older cua-driver build), and the caller should fall back to base64-prefix sniffing. Nr=FTr7contenttyperrimagerOmimeTyperc3K|]}||V dSr r_)rwts r$rz'_extract_tool_result..s'77Q71777777r&)rbrc)rOrlimage_mime_typesr7r=)rrrrrr%r&rr) rrOrlrrer text_chunksrptypeb64mimejoineds r$r&r&s.DF"$z9e44B ? ? ? ? g  $--C . c"""tZ66<" ''---77k77777 )/)B)B:)N)NZ4:f%%%TZDD#   DDD ,'   s;=D99E  E r(#tuple[Optional[str], Optional[str]]c|dpg}|r=|dr5|dpg}|r|dr|dnd}|d|fS|dpi}|dp|d}|r0|dp|d pd}||fSd S) uPull a (png_b64, mime_type) pair out of a flattened tool result. cua-driver delivers window screenshots in two shapes depending on tool + transport: * As an MCP ``image`` content part — surfaced by ``_extract_tool_result`` in ``out["images"]`` with a parallel ``image_mime_types`` entry. This is what ``get_window_state`` emits over the stdio MCP transport. * As a base64 field inside ``structuredContent`` — ``screenshot_png_b64`` (+ ``screenshot_mime_type``). This is what ``get_window_state`` returns when its structured payload carries the image instead of a content part (newer driver builds; also the shape seen via the ``cua-driver call`` CLI surface). Checking both makes capture() robust to either delivery shape, so the image never silently drops just because the driver moved it between the content list and structuredContent. Returns ``(None, None)`` when neither location carries an image. rlrrNr7rfpng_b64screenshot_mime_type mime_type)NNr!)r(rlmimesrrrs r$_image_from_tool_resultrs(WWX   $"F &)*++1r 7U1X7uQxx4ay$,--3J ..- . . K*..2K2KC  NN1 2 2 ~~k**  Dy :r&ct|tst|ttfsdS t|}n#t$rYdSwxYw|dkr|ndS)zCReturn a positive integer, rejecting booleans and malformed values.Nr)r0rrdrrf)r4rs r$ _positive_intr st%jc &C&CtU ttaZZ66T)sA AA raw_windowsc g}|D]7}t|tst|d}t|d}||b|d}t|tt frt|t s|nd}|dd}|dd}|t|tr|nd|||d d ut|tr|nd|d 9|S) u Normalise cua-driver ``list_windows`` entries, dropping unusable ones. Every downstream operation needs both an integer ``pid`` (for get_window_state / action tools) and ``window_id`` (for screenshot / element clicks), so a window missing either is uncapturable. Crucially, on X11 a window's PID comes from the *optional* ``_NET_WM_PID`` property — the desktop root, panels, and override-redirect popups routinely omit it, so the driver reports ``pid: null`` for them. Coercing every entry unconditionally (``int(w["pid"])``) let one such window abort enumeration of the real, targetable windows. We skip the unusable entries instead so capture() and focus_app() still find the windows that matter. ``z_index`` follows CUA Driver semantics: higher = closer to front. Wayland may return ``z_index: null`` (undefined stacking order); we treat null as the lowest priority so real windows still sort above desktop/root windows, and the backmost never ends up selected as the capture target. pidrNrurapp_namerr is_on_screenFrrrrrru) r0r1rr"rdrrrr) rrqrxpid_int window_id_intz_rawrurrs r$_ingest_windowsrsG*%'G   !T""  e --%aeeK&8&899 ?m3 i  %ec5\::a:eUYCZCZa%%`a55R((gr""$.x$=$=E2&%%//58(44Zd?dddddd-d@ddCZdd-dDddFZdd-dDddHZdddJZddKZddLZdddNZddddd-dOddTZddUZddVddWZddZZdd[Zddd\dd`ZddaZ ddcZ!dddZ"dedfdgdhddoZ#ddpddsZ$dddddtddyZ%dddddzdd~Z&ddpddZ'd-dddZ(ddZ)ddZ*d-dedddZ+ddZ,ddZ-ddZ.ddZ/ddZ0dddddZ1ddZ2ddddZ3dS)CuaDriverBackendz@Default computer-use backend. Cross-platform via cua-driver MCP.standardrrrrc|dvrtd|||_|dkrttpd|nd|_t |_t|j|j|_d|_ d|_ d|_ d|_ i|_ dtjjdd|_t%|j|jj|jj|_dS)N>rrz(unsupported cua-driver permission mode: rrzhermes-rrDr%has_tool)rfrrrrrrrr _active_pid_active_window_id _last_app _last_target_snapshot_tokensrrr _session_idrr%r._typed_browser)rrs r$rzCuaDriverBackend.__init__vs  "> > >YYYZZ Z..00 577=2 O O O  $~~ )$,8MNN *.04(,AE13$!B$*,,*:3B3*? A A2'm-],   r&rct|dd}|2t|j|jj|jj}||_|S)zIReturn the per-backend typed route, including test-constructed instances.rNr)rrrrr%r.r)rroutes r$_browser_routezCuaDriverBackend._browser_routesR.55 =(+-10E #(D  r&ctddlm}|ddddl}| |j|j|jn/#t$r"|j|j wxYw |j dd|j in2#t$r%}t d|Yd}~nd}~wwxYw|jjrt}|rJ || n2#t$r%}t d |Yd}~nd}~wwxYwt#rT |d|j dS#t$r&}t d |Yd}~dSd}~wwxYwdSdS) Nr)ensureztool.computer_useF)promptr>rz:cua-driver start_session failed (continuing anonymous): %s)rcz5cua-driver set_config(max_image_dimension) failed: %s cursor_idz.cua-driver set_agent_cursor_enabled failed: %s)ratools.lazy_depsr importlibinvalidate_cachesrrrrIrr%rrrrrh set_configr\set_agent_cursor_enabled)r _lazy_ensurerrmax_dims r$rzCuaDriverBackend.startsf ;::::: (7777 ##%%% $0%++--- M   ! ! ! !   $0%**,,,   Z M # #OiAQ5R S S S S Z Z Z LLUWX Y Y Y Y Y Y Y Y Z = ! V899G ]]OOO@@@@ ]]]LL!XZ[\\\\\\\\]    VV11%4CS1TTTTT VVVLL!QSTUUUUUUUUUV V V V VsS9A55,B!%"C C7C22C7D.. E8EE/F F=F88F=c|jjrV |jdd|jin2#t$r%}t d|Yd}~nd}~wwxYw |j |j|j |j dSdS#|j |j wwxYw# |j|j |j ww#|j |j wwxYwxYw)Nrrz7cua-driver end_session failed (continuing teardown): %s) rrr%rrIrrrrrrVs r$rzCuaDriverBackend.stopsv = ! [ [ '' 4CS7TUUUU [ [ [ VXYZZZZZZZZ [ 1 M    1 !!###(4)..00000544(4)..00005 1 !!###(4)..000054(4)..00005sD"1 A AA $C!>B;;#C!E#D<"E#EErc>tjdvrdStS)N)rMrrNF)rSrTrJr s r$ is_availablezCuaDriverBackend.is_availables# <; ; ;5*,,,r&cLd|_d|_d|_d|_i|_dS)zGForget a capture/focus target so a failed lookup cannot misroute input.N)rrrrrr s r$_clear_active_targetz%CuaDriverBackend._clear_active_targets/!%  "r&rmoderr c X|t|dddgd|dS)zAReturn an empty capture after disarming any prior target context.rNr)rrrrrrapp window_title png_bytes_len)rr )rrrs r$_failed_capturez CuaDriverBackend._failed_capturesC !!###     r&rrrcj |j||}n##t$r|wxYw|ddur[|d}|t d|dt |tr|rd|ndz|S)zKCall a capture-stage tool and disarm state on transport or logical failure.r=TrOrVz failedz: r)rr%rIrr"rr0r)rrrr(rs r$_call_capture_toolz#CuaDriverBackend._call_capture_tool$s -))$55CC     % % ' ' '   779   % %ggfooG  % % ' ' '+d+++%/%=%=Q'Q>>>>rS  s >rrc|dd|jd}tt|}|dd|r|St d |jdd|jdd}n4#t$r'}t d|gcYd }~Sd }~wwxYw| d dur0t d | gStt|}|d d|S) uLoad normalized visible windows, with the shared CLI recovery path. Windows are sorted by ``z_index`` **descending**: CUA Driver defines higher values as closer to the front, so the frontmost window ends up at index 0 — which is what ``capture()`` and ``focus_app()`` pick as the default target. ``_ingest_windows`` already normalised null ``z_index`` (Wayland) to 0, so those windows sort to the back. list_windowsT)on_screen_onlyrc|dSNrur_rxs r$z0CuaDriverBackend._load_windows..C 1Y<r&)rreversezScua-driver list_windows returned no windows over MCP; re-fetching via CLI transportg4@z3cua-driver CLI re-fetch for list_windows failed: %sNr=z:cua-driver CLI re-fetch for list_windows returned an errorc|dSrr_rs r$rz0CuaDriverBackend._load_windows..Yrr&) rrrrsortrr!rrrIrNr"r)rr(rqcli_outrGs r$ _load_windowszCuaDriverBackend._load_windows4st%% #0@ A A  "";C"@"@AA // >>>  N ,    m66#'D4DEEGG     LLNPS T T TIIIIII  ;;y ! !T ) ) LLU V V V  % % ' ' 'I!";G"D"DEE // >>>s2$B C!C=CCrqrc | sgS fd|D}|r|S |}n5#t$r(}td||g}Yd}~nd}~wwxYwt t |D]ttr ddur/ d}t|tst|ttfsv t|}n#t$rYwxYw|dkrfdd D} |vr |t fd |Dr| fd |D} | r| S fd |D} | r| Sfd |D} | r| S fd|DS)aResolve ``app=`` through exact names before convenience substrings. Linux ``list_windows`` can omit an app name while ``list_apps`` retains name/bundle-ID metadata. Exact direct names and exact metadata aliases must win over substring matches: querying ``Code`` must not silently select ``Visual Studio Code`` merely because it is frontmost. c g|]O}t|ddkM|PSrrrr"rrYrwrx app_lowers r$rz;CuaDriverBackend._match_windows_for_app..js\   Cj" 5 566<<>>DDFFFF FFFr&z1computer_use list_apps fallback failed for %r: %sNrunningFrrch|]f}t|xt,@gSr_)r0r"rrrY)rwrraw_appr4s r$ryz:CuaDriverBackend._match_windows_for_app..sr C(8(88u3??EJKKMM ##%%r&) bundle_idbundleIdrr display_namec3 K|]}|vV dSr r_)rwaliasrs r$rz:CuaDriverBackend._match_windows_for_app..s(==EY%'======r&cBg|]}|dv|Srr!)rwrx exact_pidss r$rz;CuaDriverBackend._match_windows_for_app..s-KKKe 0J0J!0J0J0Jr&c g|];}t|ddv9|.sP   Cj" 5 566<<>>>> >>>r&cBg|]}|dv|Srr!)rwrx partial_pidss r$rz;CuaDriverBackend._match_windows_for_app..s-OOO!!%%,,,2N2NA2N2N2Nr&c g|]p}t|dds9t|ddvn|qS)rrrrrs r$rz;CuaDriverBackend._match_windows_for_app..s|   quuZ,,--3355 Sw!3!344::<<<< <<>++LL    LLLcSV W W WLLLLLL   #uu !$ # & &Ggt,,  I0F0F%0O0Okk%((G'4((  7S#J0O0O  'll    axxXG G##s####====W===== &  %%%KKKKWKKK  "! !         "! !OOOOwOOO  $# #         s*A B "BB "D22 D?>D?somNrrrarc d "||_||||dSt|}t|}||||dS|pd||ddddg}nP |}n##t$r|wxYw|s||S|r|p|rn|tvrAd5d ""fd |D}|s||d|dSt|d}n:|8|6|r4| ||} | s||d|dS| }t|t||dup|du} | d|_ | d|_ i|_| d} |s|js | p|pd|_|j |j d|_d} d} g}dx}}d}|dkr|jdp |jj }d}|r;|d|j dd|jd}t-|\} } | sd}||d|j |j |jd}t-|\} } t/|d t0r|d nd}t3|\}}t5jd!|}|r|d"}| st:d#|j  |jd|j |j |jdd$}| d%d&ur|n%| d'r|d'd} d(} n#t$r&}t:!d)|Yd}~nd}~wwxYwn|d|j |j |jd}d6d+}||rt:d,|j |j  |jd|j |j |jdd$}| d%d&ur|n ||s|}n2#t$r%}t:!d-|Yd}~nd}~wwxYwt/|d t0r|d nd}t3|\}}| d.pi d/}t/|tDr|rtG|}n|rtI|ng}d0|D|_t-|\} } t5jd!|}|r|d"}d}| rf tKj&| d1}tO|}tQ|\} }!| r|!r| }|!}n%#t$rtO| d2zd3z}YnwxYwtS|||| || ||| 4 S)7uCapture the frontmost on-screen window or an exact known target. Maps hermes `capture(mode, app)` → cua-driver `list_windows` + `get_window_state` (ax/som) or `screenshot` (vision). Nz3z?rFrrrxrrrc|ddd|ddtfdtDS)Nrr rc3 K|]}|vV dSr r_)rwrhaystacks r$rzGCuaDriverBackend.capture.._is_desktop_window..s(NN48+NNNNNNr&)r"rYr_DESKTOP_WINDOW_NAMES)rxrs @r$_is_desktop_windowz4CuaDriverBackend.capture.._is_desktop_windowsbeeJ33JJaeeGR6H6HJJPPRRNNNN8MNNNNNNr&c*g|]}| |Sr_r_)rwrxrs r$rz,CuaDriverBackend.capture..s*CCCQ-?-?-B-BCqCCCr&z'cBtfddDrdndS)Nc3K|]G}|dddddvVHdS)rrrrN)r"rY)rwrrxs r$rz=CuaDriverBackend.capture.... sr''AEE*b11HHAEE'24F4FHHNNPPP''''''r&)r;r<r=r@r9rrz)rrs`r$rz*CuaDriverBackend.capture.. sH3''''['''$$aar&)rz!)rrrrrrrvision screenshotjpegU)rformatqualityrrY)rrrrOzAXWindow\s+"([^"]+)"rzzbcua-driver vision capture returned no image over MCP (window_id=%s); re-fetching via CLI transportrr=Trlz image/pngz8cua-driver CLI re-fetch for vision screenshot failed: %sr(c|drdS|dpi}|ds|drdSt|dtr|dnd}t|pd\}}|o| S)NrlFr7rrrfrOr)r"r0rrr)r(sc_txt_trs r$ _gws_is_emptyz/CuaDriverBackend.capture.._gws_is_empty s778$$! 5gg1228b77:&&!#''2F*G*G! 5)3CGGFOOS)I)IQcggfooor(332-288::..r&zrcua-driver get_window_state returned an empty result over MCP (pid=%s window_id=%s); re-fetching via CLI transportz7cua-driver CLI re-fetch for get_window_state failed: %sr7rrc6i|]}|j |j|jSr_)r|rl)rwrs r$ z,CuaDriverBackend.capture.. s6%%%?%%%%r&)validatergrh) rrrrrrrrrimage_mime_typerxrrr)r(rrr)*rrrrIrrrY_SCREEN_CAPTURE_SENTINELSsortedrrrrrrrrrr.r4rrrr0rrrrrrr!rr"rNrrrtrs b64decoder{rr )#rrrrr target_pidtarget_window_idrqr9filteredrrrr+rrrrruse_screenshotsc_outgws_outrr%rwtrcli_excr'r sc_elementsrr}detected_widthdetected_heightrs# @r$capturezCuaDriverBackend.captures, ?i3{i/++O's++J,Y77 !%5%=++[ I2!-# GG ,,..   ))+++  2++D111 ;9,,9J9J9L9LPi9i9i O O O ODCCC'CCCG ++J#JJJ   GG[Y.3.227C@@H ++>*#'#3)-)?'+'7 G{{9--55113333 X..6")("3A"6*5 LLRT[1 <--"+!%!7#/G" / / / /}W%% K$d&< "m>>*#'#3)-)?'+'7 G{{9--55113333*]733*") LLQSZ '1#&F&FN76??BD,T22MGT#;;':;;ArFFzRRK+t,, K K:;GG>BJ4T:::%%!%%%D !(?w'G'G $G_2D99B +!xx{{   6 6&w??? #C 2Ns2S2S/!-o-*E,F 6 6 6 #G q 0A 5  6%'+    sP&A;; BBN O!OO+A(R SR>>S5?W55XXr.r/Optional[ActionResult]c|r|dkrdS|dkrtd|dd|dS|j|d std|d dd Sd|d <dS) a3Attach delivery_mode to an input-action args dict. Background is the default and never needs a flag. Foreground is only sent when the live action schema accepts it; on an older driver that lacks the property we refuse with a structured ``foreground_unsupported`` result instead of silently downgrading to background (which would land the input somewhere the model didn't expect). Returns an ActionResult to short-circuit on refusal, or None to proceed. See NousResearch/hermes-agent#67052 phase B. backgroundN foregroundFbad_delivery_modezunknown delivery_mode u — use background|foreground.rr.r,rr/foreground_unsupportedzThe connected cua-driver action schema does not accept delivery_mode, so foreground delivery is unavailable. Use another verified rung without assuming the reported package version describes the live schema.rr.r,r/r)r rr2)rr.rr/s r$_apply_deliveryz CuaDriverBackend._apply_delivery s    = =4 L ( (.Aaaaa  }44V_MM .F*A    !-_tr&bring_to_frontr c||||}||S|r|dkrtd|ddS|jdstd|ddd S|j|jtd|d dd S||j|j }|js|S|||}|r ddd|j d<|S)a!Apply one delivery rung, optionally focusing via its own tool. ``bring_to_front`` is never an input-action property. When explicitly requested, the separately approved standalone focus action runs first, then the original foreground input runs unchanged. Nr?F"bring_to_front_requires_foregroundz3bring_to_front requires delivery_mode='foreground'.rArEbring_to_front_unsupportedOThe connected cua-driver does not advertise the standalone bring_to_front tool.rCbring_to_front_target_requiredzFCapture an exact target before requesting persistent foreground focus.rT)invokedr foreground_focus) rDr rr.rrrEr_actionr)rr.rr/rErefusalfocusedrs r$_run_input_actionz"CuaDriverBackend._run_input_action" sK&&vt]CC  N   ,,#!=Q  =**+;<< #!5".m '4+A+I#!9".d ))$0*G: fd++  (//FK* + r&leftrzF)elementryrzbutton click_countrr/rErRryrzrSrTrdrOptional[List[str]]c|j} | tdddS|pd} | dvrtddd|dS|d krd nd} | | d } |)|jtd| d S|| d <|j| d<nD|0|.|jtd| dS|| d<|| d<|j| d<ntd| dS|r|| d<|| | ||S)NFclick*No active window — call capture() first.rr.rrQ>rQrightmiddlezunknown button u" — expected left, right, middle.r double_click)rrSz,No active window_id for element_index click.rwrz)No active window_id for coordinate click.ryrzzclick requires element= or x/y.modifier)rr rYrrP) rrRryrzrSrTrr/rEr button_normr rs r$rWzCuaDriverBackend.clickX s ;5(TVVV V'..00 9 9 95(f&(f(f(fhhh h!,!1!1~~w'*kBB  %-#uT,Z\\\\$+D ! $ 6D   ]q}%-#uT,WYYYYDIDI $ 6D  5(IKKK K  )(D %%dD-PPPr&) from_element to_elementfrom_xyto_xyrSrr/rEr_r`raOptional[Tuple[int, int]]rbc |j} | tdddSd| i} |0|.|jtdddS|| d<|| d<|j| d <n|||jtddd St|d t|d c| d <| d<t|d t|d c| d<| d<|j| d <ntdddS|d| ||S)NFdragrXrYrz+No active window_id for element-based drag.r_r`rz(No active window_id for coordinate drag.rrzfrom_xfrom_yto_xto_yzGdrag requires from_element/to_element or from_coordinate/to_coordinate.)rr rrdrP) rr_r`rarbrSrr/rErrs r$rezCuaDriverBackend.drag sh ;5(TVVV V %s|  # (>%-#uV,Y[[[[#/D !+D  $ 6D    U%6%-#uV,VXXXX-0__c'!*oo *DNDN),U1XE!H &DL$v, $ 6D  5(qsss s%%fdM>RRRr&rg)amountrRryrzrr/rE directionrjc x|j} | tdddS| |tdtd|d} ||j|| d<|j| d <nM|K|I|jtddd S|jd d r || d <|| d<|j| d <|d| ||S)NFscrollrXrYrz2)rrkrjrwrz*No active window_id for coordinate scroll.zinput.scroll.coordinatesr ryrz)rr rorprrr,rP) rrkrjrRryrzrr/rErrs r$rmzCuaDriverBackend.scroll s ;5(TVVV V"!SV__--    4#9#E$+D ! $ 6D   ]q}%-#uX,XZZZZ}00*1 S S $ 6D %%hm^TTTr&)r/rErc|j}|j}||tdddS|||d}|d|||S)NF type_textrXrY)rrr)rrr rP)rrr/rErrrs r$rqzCuaDriverBackend.type_text sg* ;)+5(TVVV V'*DQQ%%k4WWWr&rc"|j}|j}||tdddSt|\}}|stddd|dS|r"||||gzd}|d|||S|||d }|d |||S) NFrrXrYzCould not parse key from ''.)rrrhotkey)rrr press_key)rrr rrP) rrr/rErrkey_namerrs r$rzCuaDriverBackend.key s* ;)+5(TVVV V/t44) O5(MT(M(M(MOOO O  \+.Y,5 ,B$D$DD))(D-XX XYxHHD))+t]N[[ [r&r4c|j}|j}||tdddS|tdddS||||d}|d|S)zBSet a value on an element. Handles AXPopUpButton selects natively.NF set_valuerXrYz,set_value requires element= (element index).)rrrwr4)rrr rM)rr4rRrrrs r$rxzCuaDriverBackend.set_value s* ;)+5(TVVV V ?5(VXXX X"$    ||K...r&c |jdd|ji}|d}|d}t |t r.|d}t |t r|r|St |t r|r|St |t r.|d}t |t r|r|S|d}t |t r|r|Stt|}|r|St |trg}| D]v}tj d|}|r]| |dt!|dd w|SgS) Nrrr7rOrz(.+?)\s+\(pid\s+(\d+)\)rzrr)rr%rr"r0r1rrrr splitlinesrrrrrrd) rr(rrOrderived parsed_appsrrss r$rzCuaDriverBackend.list_apps sm%%kIt?O3PQQWW011 wwv j$ ' ' >>&))D$%% $  dD ! ! d K dD ! ! 88F##D$%% $  wwv dD ! ! d K$%>s%C%CDD  N dC  K)) ] ]I8$??]&& 0@0@0B0B3qwwWXzz??'['[\\\  r&c*|Sr )rr s r$rzCuaDriverBackend.list_windows4 s!!###r& raise_windowc  |}n##t$r|wxYw|||}|r|dnd}|r|d|_|d|_i|_|dp||_|j|jd|_|rj|j dstdd d d S| |j|j}|j s|Sd |_d |jd<|Std d d|dd|jd|jdS|tdd d|dS)uTarget an app, optionally invoking standalone foreground focus. cua-driver background-automation never needs to bring a window to the front: capture(app=...) already selects the right window via list_windows. We implement focus_app as a pure window-selector — enumerate on-screen windows, find the best match for *app*, and store its pid/window_id so that subsequent click/type calls hit the right process. The default remains non-disruptive. ``raise_window=True`` is explicit, separately approved by the Hermes adapter, and uses cua-driver's standalone ``bring_to_front`` tool rather than an action property. rNrrrrrEF focus_apprHrIrATtarget_selectedz Targeted z (pid z , window z) without raising window.rYz#No on-screen window found for app 'rs)rrIrrrrrrrrr.r rErr.r)rrr~rqmatchedrrOs r$rzCuaDriverBackend.focus_app7 s ((**GG     % % ' ' '  --gs;; '0D  %e}D %+K%8D "$&D !#J/63DN'!3!!D  }../?@@' *9 q  --("4.z#"N!,26 ./ TF:$6TTd>NTT"&"8TTT  !!###u[$Q#$Q$Q$QSSS Ss 7)rrurlsadditional_arguments creates_new_application_instancerrrrc|s|stdd|ji}|r||d<|r||d<|rt||d<|rt||d<|rd|d<|jd |}|d p d |d iS) uIdempotent launch. Returns ``{pid, bundle_id, name, windows[]}`` so callers can skip an extra ``list_windows`` round-trip before ``get_window_state``. ``creates_new_application_instance=True`` forces a new instance even if the app is already running — use it when concurrent runs may touch the same app so each session gets its own isolated window.z,launch_app requires either bundle_id or namerrrrrTr launch_appr7rO)rfrrrr%)rrrrrrrr(s r$rzCuaDriverBackend.launch_appz s" M MKLL L )4+;<  * )D   DL  &::DL  F+/0D+E+ED' ( + <7;D3 4m%%lD99&'@FCK+@@r&cL|ddt|iS)zYTerminate by pid. Equivalent to ``kill -9`` on POSIX, ``taskkill /F`` on Windows.kill_apprrMrd)rrs r$rzCuaDriverBackend.kill_app s#||JC(9:::r&)rc|dt|i}|t||d<|d|dS)zActivate a window so subsequent foreground-dispatched input lands on it. cua-driver's docstring notes this is the cheaper path than per-call SetForegroundWindow flashes.rNrrEFinject_session)rdrM)rrrrs r$rEzCuaDriverBackend.bring_to_front sE !&s3xx0  #ID ||,d5|IIIr&kwargsrc @|jdi|S)z@Exact-bind a native browser window or read fresh semantic state.r_)robserverrs r$typed_browser_statez$CuaDriverBackend.typed_browser_state ',t""$$,66v666r&c @|jdi|S)zRun one namespaced typed-browser mutation in this exact route.r)rmutate)rrrrs r$typed_browser_actionz%CuaDriverBackend.typed_browser_action s+""$$++KT+RRRr&ch|dt|t|dS)uHMove the agent-cursor *overlay* to a screen point. This is a visual hint — it does NOT move the real OS pointer (cua-driver explicitly avoids stealing pointer focus). The overlay glides smoothly to the target, so consumers use it before a click to give a visible "where the agent is going" cue. move_cursor)ryrzr)rryrzs r$rzCuaDriverBackend.move_cursor s, ||MQc!ff+E+EFFFr&rc|jdd|ji}|dpi}t |ddt |ddfS)zPReturn the *real* OS cursor position in screen points (origin top-left).get_cursor_positionrr7ryrrz)rr%rr"rd)rr(r#s r$rz$CuaDriverBackend.get_cursor_position stm%% !It/?#@  WW( ) ) /R266#q>>""CsA$7$777r&ct|jdd|ji}|dpiS)zReturn the logical size of the main display in points plus its backing scale factor. Shape: ``{width, height, backing_scale_factor}``.get_screen_sizerr7rr%rr"rr(s r$rz CuaDriverBackend.get_screen_size sBm%%  4+;<  ww*++1r1r&g?rr)factorr r!rrxr{rr r!c  |jdt|t|t|t|t|t||t||jd S)zReturn a JPEG / PNG of a sub-region of a window, optionally scaled. cua-driver supports zoom-to-rect for callers that need a higher-resolution view of a specific element.zoom) rryrzrxr{rr r!r)rr%rdrr) rrryrzrxr{rr r!s r$rzCuaDriverBackend.zoom sn }&&vYqauQxxeAhhFmmW' 0 0    r&renabledrc^dt|i}|r||d<|d|S)z:Toggle the agent cursor overlay's visibility for this run.rrr)rrM)rrrrs r$rz)CuaDriverBackend.set_agent_cursor_enabled s;!*4==9  * )D ||6===r&)glide_msdwell_ms idle_hide_msrrrKrrci}|t||d<|t||d<|t||d<|r||d<|d|S)uTune the overlay's motion timings — glide duration, post-click dwell, idle-hide delay. Each None means "leave at current value".Nrrrrset_agent_cursor_motion)rrM)rrrrrrs r$rz(CuaDriverBackend.set_agent_cursor_motion st "  $XD   $XD   ##(#6#6D  * )D ||5t<<EEww*++1r1r&) record_video output_dirrc|jd|t||jd}|dpiS)a;Enable trajectory recording (per-turn screenshots + action JSON) to ``output_dir``. ``record_video=True`` ALSO captures the main display to ``/recording.mp4`` (H.264). Recording ownership is keyed by this run's session id so concurrent runs don't fight over the recorder.start_recording)rrrr7)rr%rrr")rrrr(s r$rz CuaDriverBackend.start_recording, sVm%%&7$ ..': :    ww*++1r1r&ct|jdd|ji}|dpiS)zDisable recording and finalise the mp4 (if video was on). Returns the recorder's final state including ``last_video_path``.stop_recordingrr7rrs r$rzCuaDriverBackend.stop_recording: sEm%%&6 t'9   ww*++1r1r&ct|jdd|ji}|dpiS)zReturn the current recorder state without changing it. Shape: ``{recording, enabled, output_dir, next_turn, last_video_path, last_error, owner, video_active}``.get_recording_staterr7rrs r$rz$CuaDriverBackend.get_recording_stateB sBm%% !It/?#@  ww*++1r1r&)dry_run speed_factortrajectory_dirrrc|jd|t|t||jdS)zReplay a prior recording's turn stream by re-invoking each turn's tool call in lexical order. ``dry_run=True`` logs without actually firing the tools.replay_trajectory)rrrr)rr%rrr)rrrrs r$rz"CuaDriverBackend.replay_trajectoryK sH }&&':,G}}!,//' = =    r&cF|jdd|jiS)zBootstrap ffmpeg for ``start_recording(record_video=True)`` on Linux / Windows. macOS records natively via ScreenCaptureKit and doesn't need ffmpeg.install_ffmpegr)rr%rr s r$rzCuaDriverBackend.install_ffmpegX s+}&& y$*:;   r&ct|jdd|ji}|dpiS)z-Return the current cua-driver runtime config. get_configrr7rrs r$rzCuaDriverBackend.get_configb s@m%% 9d&67  ww*++1r1r&c H|dt|S)uSet cua-driver config keys. Common keys include ``max_image_dimension`` (image-output resizing), recording flags, etc. Unknown keys are passed through verbatim — cua-driver validates against its own schema.r)rMr1)rconfigs r$rzCuaDriverBackend.set_configi s ||L$v,,777r&c|jdd|ji}|dp d|diS)a.Return a lightweight snapshot of running regular apps + on-screen visible windows with bounds, z-order, owner pid. Roughly the data ``list_windows`` exposes, in one call. Most callers should prefer ``capture()`` / ``focus_app()`` which already use this shape internally.get_accessibility_treerr7rOrrs r$rz'CuaDriverBackend.get_accessibility_treer sK m%% $y$2B&C  ww*++DF /DDr& page_argsc t|||jd}|||jd|S)azInteract with a browser page loaded in a running app (Chrome, Safari, Edge, ...). cua-driver routes through CDP / Apple Events / AX tree depending on the target. ``action`` + ``page_args`` shape depends on the requested operation (e.g. ``action="eval"`` takes ``js: str``); see cua-driver's ``page`` tool description for the full grammar.)rr.rpage)rdrupdaterr%)rrr.rrs r$rzCuaDriverBackend.page sNs88'   I}&&vt444r&rrrc|rt|ni}|d|j|j|||S)uCall any cua-driver MCP tool by name with arbitrary args. ``session`` is injected (preserves the caller's explicit one via setdefault). For tools the wrapper doesn't already type- wrap, this is the supported escape hatch — preferred over reaching for ``self._session.call_tool`` directly because it keeps the session-id contract consistent with everything else.rr)r1 setdefaultrrr%)rrrrpayloads r$r%zCuaDriverBackend.call_tool sN!%,$t***"9d&6777}&&tWg&FFFr&r c|d}t|tsdS|j|}|sdS|jd|sdS||d<dS)aSurface 6: when the wrapper is about to call a token-capable tool with `element_index`, look up the matching `element_token` from the last snapshot and attach it. cua-driver-rs's contract for combined args is documented in trycua/cua#1961: "element_token takes precedence over element_index when both supplied. Returns an explicit 'stale' error if the snapshot has been superseded." Gated on the per-tool capability claim so we don't send the field to drivers that predate the surface (which would reject the schema with `additionalProperties: false`). rwNzaccessibility.element_tokensror|)r"r0rdrrr,)rr rr~rs r$_maybe_attach_element_tokenz,CuaDriverBackend._maybe_attach_element_token shh''#s##  F%))#..  F}00 *1    F %_r&Trrc ||||r|d|j |j||}nG#t $r:}t d|td|d|cYd}~Sd}~wwxYw|d }|d}| dpi}d } t|tr$t| d d } nt|tr|} | s8t|tr#t| d d } i} t|tr| |t|tr| |t||| | || d  S) Nrzcua-driver %s call failedFzcua-driver error: rYr=rOr7rrr/r)rrrrr%rIr exceptionr r"r0r1rrr2) rrrrr(rrrOrrrs r$rMzCuaDriverBackend._action s ((t444  9 OOIt'7 8 8 8 Y-))$55CC Y Y Y   8$ ? ? ?5?WTU?W?WXXX X X X X X X YY 6{WW0117R  dD ! ! $((9b1122GG c " " G 9:j$77 9*..B7788G " dD ! !  KK    j$ ' ' $ KK # # #"4WdJ6:hh6O6OQQQ QsA B/B BB)r)rrrr)rrrr)r)rrrrrr r)rrr)rqrrrrrrr)rNNN) rrrrrrarrarr )r.rrrr/rrr<) r.rrrr/rrErrr )rRraryrarzrarSrrTrdrrUr/rrErrr )r_rar`rararcrbrcrSrrrUr/rrErrr )rkrrjrdrRraryrarzrarrUr/rrErrr )rrr/rrErrr )rrr/rrErrr r )r4rrRrarr )F)rrr~rrr ) rrrrrrUrrUrrrr)rrdrr )rrdrrarr )rrrr)rrrrrrLrr)ryrdrzrdrr )rrrr)rrdryrrzrrxrr{rrrr rr!rdrr)rrrrrr ) rrKrrKrrKrrrr ) rrUrrrrrrrr )rrrr)rrrrrr)rrrrrrrr)rr )rrdr.rrrrr)rrrrLrrrr)r rrrrr)rrrrrrrr )4rrrrrrrrrrrrrrr;rDrPrWrermrqrrxrrrrrrErrrrrrrrrrrrrrrrrrrrr%rrMr_r&r$rrssJJ0 0 0 0 0 d    :V:V:V:Vx1111(----####       &&&&PM M M M d!!#' M M M M M ` $$$$L4444r"&)-'+$3Q3Q3Q3Q3Q3Qp'+$(-1+/)-'+$"S"S"S"S"S"SP!%)-'+$(U(U(U(U(U(UVFJ).XXXXXX@D#(\\\\\\./////&""""H$$$$;S;S;S;S;SL$("$(4816AAAAAAB;;;; 37 J J J J J J77777777!%)- SSSSSSGGGG88882222!       .=A>>>>>>=A<@@D;? ======&GK<@;?:> <<<<<<*;?222222.3 2 2 2 2 2 222222222+003           22228888 E E E E5555$ G&* G G G G G G&&&&> $ 'Q'Q'Q'Q'Q'Q'Q'Qr&r)rrrrrrrrrrrrrr rr)rrar )rirjrrk)rqrrrr)r}rrrar,)rqrrrrrrrr)r*rrr)rrrrrr)rr*rrrr*)rrrr)r4rrr)r8rrr*)r8rrr)rrKrrL)rrrr)rdrrre)rurrrre)r}rrr)rrrr)rrrr)rrrr)r(rrr)r4rrra)rrrrrr)r(rrrr)rqrrrrr)\r __future__rrrs collectionsrconcurrent.futuresr functoolsr%loggingrUrrDrrSrrrrpathlibrtypingrrrr r hermes_cli._subprocess_compatr tools.computer_use.backendr r rr tools.computer_use.browser_router getLoggerrrr2r<r=rr-rrrnrJr\r`rhrpr|rrrrrrrr$ lru_cacher-compile MULTILINErpr3r'rBrrJrPrZr`rarrtrrrrrrr&rrrrrrr_r&r$rs!!F#"""""     ######33333333333333<<<<<< BAAAAA  8 $ $)-<<<<<<\.&7VUU";<EEEE $ $ $ $     <<<<$====&      FOOOO0XXXXXXXX|IJIJIJIJIJIJ\.Q V2:.L   $$$$MMMM!!!!!H&0000 ;?222222j    **82222j$$$$N2////////lz z z z z z z z z5555p$$$$N****----`4    &lQlQlQlQlQ)lQlQlQlQlQr&