PmjCUdZddlmZddlmZmZdddiddgd d d d dgd dd ddddddddddddddddddddddddd d!did"d"d#d$d%dgd&d'd d(d dgd)d*d+d,d-dd.dd/dd0dd1d d!did"d"d2d$d3d d!did"d"d4d$d5dgd6d7d d8dd9dd:dd;diddd?dd@dAdBddCdDdEddFddGdHgdId dJdDdKddLddMddNddOddPddQddRddSddTddUdVgdWd dXddYdZgd[d d\dd]dd^dd_dd`ddaddbdgdcddd deddfddDdgddgdhdid dgdjdkd ddldmgdnd ddodddpdd d!didqd,ddrddAdsddAdtddAduddAdvddAdwddAdxddDdyddzdgd{d|Zd}ed~<ddZdS)uNSchema for the generic `computer_use` tool. Model-agnostic. Any tool-calling model can drive this. Vision-capable models should prefer `capture(mode='som')` then `click(element=N)` — much more reliable than pixel coordinates. Pixel coordinates remain supported for models that were trained on them (e.g. Claude's computer-use RL). ) annotations)AnyDict computer_useuDrive the desktop in the background via cua-driver — screenshots, mouse, keyboard, scroll, drag — without stealing the user's cursor or keyboard focus. Supported on macOS, Windows, and Linux. Preferred workflow: call with action='capture' (mode='som' gives numbered element overlays), then click by `element` index for reliability. Pixel coordinates are supported for models trained on them. Works on any window — hidden, minimized, or behind another app. Requires cua-driver to be installed.objectactionstring)captureclick double_click right_click middle_clickdragscrolltypekey set_valuewait list_apps list_windows focus_appcua_browser_statecua_browser_preparecua_browser_navigatecua_browser_clickcua_browser_typecua_browser_pointercua_browser_dialogcua_browser_set_input_filescua_browser_downloadu Which action to perform. `capture` is free (no side effects). All other actions require approval unless auto-approved. Use `set_value` for select/popup elements and sliders — it selects the matching option directly without opening the native menu (no focus steal).)renum descriptionmode)somvisionaxuCapture mode. `som` (default) is a screenshot with numbered overlays on every interactable element plus the AX tree — best for vision models, lets you click by element index. `vision` is a plain screenshot. `ax` is the accessibility tree only (no image; useful for text-only models).appuOptional. Limit capture/action to a specific app (by name, e.g. 'Safari', or bundle ID, 'com.apple.Safari'). If omitted, operates on the frontmost app's window. Pass app='screen' (or 'desktop') to capture the OS desktop/shell surface — e.g. to see the wallpaper or click the taskbar. Note: capture is per-window; a single image cannot span multiple monitors, so on a multi-screen setup capture one window or display at a time.)rr"pidintegerzqOptional exact process target for action='capture'. Pair with window_id when discovery cannot resolve an X11 app. window_idzOptional exact native window target for action='capture'. Pair with pid when an external cua-driver list_windows lookup has already identified the window. max_elementsuOptional cap on the AX `elements` array returned by `action='capture'`. Default 100, hard maximum 1000. Dense UIs (Electron apps such as Obsidian or VS Code, JetBrains IDEs) can publish 500+ AX nodes — capping prevents a single capture from blowing session context. When the cap trims the response, `total_elements` and `truncated_elements` are surfaced in the result so you can re-call with `app=` to narrow scope or raise `max_elements` when the full tree is required. Has no effect on `mode='som'` / `mode='vision'` when a screenshot is included in the response; only the rare image-missing fallback returns an `elements` array and is subject to the cap.di)rr"defaultminimummaximumelementzoThe 1-based SOM index returned by the last `capture(mode='som')` call. Strongly preferred over raw coordinates. coordinatearrayrzPixel coordinates [x, y] relative to the captured window screenshot (top-left origin). Only use this if no element index is available.)ritemsminItemsmaxItemsr"button)leftrightmiddlezMouse button. Defaults to left. modifiers) cmdshiftoptionaltctrlfnwinwindowssupermeta)rr!z%Modifier keys held during the action.)rr5r" from_elementzSource element index (drag). to_elementzTarget element index (drag).from_coordinatez3Source [x,y] (drag; use when no element available). to_coordinatez3Target [x,y] (drag; use when no element available). direction)updownr9r:zScroll direction.amountzScroll wheel ticks. Default 3.valuezFor action='set_value': the value to set on the element. For AXPopUpButton / select dropdowns, pass the option's display label (e.g. 'Blue'). For sliders and other AXValue-settable elements, pass the numeric or string value.textz+Text to type (respects the current layout).keyszUKey combo, e.g. 'cmd+s', 'ctrl+alt+t', 'return', 'escape', 'tab'. Use '+' to combine.secondsnumberzSeconds to wait. Max 30. raise_windowbooleanuOnly for action='focus_app'. If true, brings the window to front (DISRUPTS the user). Default false — input is routed to the app without raising, matching the background co-work model. delivery_mode background foregroundudHow input is delivered, for the input actions (click, double_click, right_click, drag, scroll, type, key). `background` (DEFAULT) routes input to the target without raising it or stealing focus — the co-work model. `foreground` briefly fronts the window, acts, then restores the prior frontmost app. A `confirmed` effect is done. For `unverifiable`, inspect fresh state before any retry even if escalation is recommended. Escalate only after `suspected_noop` or a structured refusal. Do not predict the rung from the app being Electron/Chromium. Foreground is a visible focus change and needs its own approval.bring_to_frontzOptional and only valid with delivery_mode='foreground'. Explicitly invokes cua-driver's standalone bring_to_front tool before the input; it is never passed as an input property. This persistent focus change has a separate approval scope. Default false.tab_idz4Opaque tab capability returned by cua_browser_state.refz@Current semantic ref from the latest cua_browser_state snapshot.destination_refz3Current destination ref for a typed pointer action.urlzURL for cua_browser_navigate. input_routetrusted dom_eventzrTyped-browser trust class. Defaults to trusted. dom_event is an explicit downgrade and is never selected silently.snapshot_format semantic_v2 dom_refs_v1z:Typed-browser snapshot format; semantic_v2 is the default.queryzOptional browser-state query. scope_refz)Optional current ref to scope a snapshot. continuationz,Continuation minted by the current snapshot. profile_mode) isolated_newisolated_namedexisting_profilezBrowser preparation mode. existing_profile is decided by cua-driver's immutable permission mode: standard requires a certified protected host; explicit Hermes YOLO uses a private unrestricted daemon. profile_namezName for isolated_named setup.z;Explicitly allow launch of a driver-owned isolated browser.)hoverr r rrz"Operation for cua_browser_pointer.)inspectacceptdismisszHPage JavaScript dialog action; native prompts stay on the native ladder. insert_text keystrokeszrsT#"""""  z  4H9""z J  ///-  Kz b  7  cz ~ !Oz L !@Mz \ ! *)]z J !'Kz Z  ),*  [z p  333@qz z $ G  {z R Y-KMMSz V 9+IKKWz Z  ),1T   [z f  ),1T gz t  7772uz ~ !?z H  SIz z \  L]z d  ;ez r  9sz | !=}z P  %|4  Qz t !5  uz J  UKz R  aSz Z  T  [z b H5TUUcz d  "K0O ez t  & 6[  uz ~ h7VWWz @ (;fggAz B X>lmmCz D  NNN+  Ez X X>^__Yz z \"\ !RRRC'' !888i&& !& 5]"" #+;[\\$,=fgg (+P !T!!!)9Z[[ (9XYY"3YZZ"3YZZ%6YZZ%6YZZ"Be z z z v J{ ~~L'L'LLLL^ r