epjRdZddlZddlZddlZddlZddlZddlZddlmZm Z m Z m Z m Z m Z mZddlmZddlmZmZmZmZmZmZmZejdZGddZdS) uFull-text / trigram / CJK message search and FTS maintenance for SessionDB. Mixin contract: this is a plain mixin class consumed by ``hermes_state.SessionDB``. It defines no ``__init__`` and no state of its own; methods access the host's attributes (``self._conn``, ``self.db_path``, ``self._execute_write`` and other SessionDB methods) established by ``SessionDB.__init__``. It must never import hermes_state (cycle) — shared module-level constants live in hermes_state_common. N)AnyCallable CollectionDictListOptionalTuple)describe_skill_invocation)FTS_CJK_STALE_KEYFTS_SQLFTS_STORAGE_VERSIONFTS_TRIGRAM_SQLMAX_FTS5_QUERY_CHARSSCHEMA_VERSION_FTS_CJK_TRIGGERS hermes_stateceZdZdZdZedeeedee edffdZ dTdZ dee ee ffd ZdTd Zdefd Zdefd Zdee ee ffd ZdefdZdTdZdTdZdefdZdefdZdTdZdddedefdZdTdZdefdZdefdZddddeee ee fgdfdede ee ffdZ dUd#ed$ed%ed&ed'ee edfde ee ff d(Z! dVd#ed*ed+ede"e ee ffd,Z#e$d-edefd.Z%e$d/edefd0Z&e$d1edefd2Z'ed1edefd3Z(ed-edefd4Z)e$d-edefd5Z*d6dddd)d7d8d9ed:ed;ed+ede"ed*ed?edee"e ee ffd@Z+ dWd-ede"ed*ed?edAed+edeeede"e ee ffdBZ,d-edefdCZ- dWd-ede"ed*ed?edAed+edeeede"e ee ffdDZ.dddddEdFed*ed+edee"ede"e ee ffdGZ/ dXd-ed*edHedIedJe"ed=e"ede"e ee ffdKZ0dLedefdMZ1defdNZ2defdOZ3ddPdQedReedefdSZ4dS)YSessionSearchMixinu>See module docstring — mixin for SessionDB (Search cluster).) id session_idrolesnippet timestamp tool_namesourcemodelsession_startedcontextfieldsreturn.cN|dSt|trtdt||j}|r2t ddt|tfd|jDS)z=Validate and canonically order an optional result projection.Nz?search fields must be a collection of field names, not a stringz unknown search result field(s): , c3$K|] }|v|V dSN).0field requesteds 9/home/thesage/.hermes/hermes-agent/hermes_state_search.py z.@s6  EYDVDVEDVDVDVDV  ) isinstancestr TypeErrorset difference_SEARCH_MESSAGE_RESULT_FIELDS ValueErrorjoinsortedtuple)clsrunknownr(s @r)_search_message_fieldsz)SessionSearchMixin._search_message_fields3s >4 fc " " _]^^ ^KK &&s'HII  ^\ &QX//@Z@Z\\]] ]    "@      r+Nc|jsdS ||jdS#tj$r&}t d|Yd}~dSd}~wwxYw)zDRun one bounded FTS5 merge pass without failing the completed write.N) max_pagesz FTS incremental merge failed: %s) _fts_enabled_merge_fts_incrementally_FTS_MERGE_MAX_PAGES_PER_INDEXsqlite3Errorloggerwarning)selfexcs r)_try_incremental_merge_ftsz-SessionSearchMixin._try_incremental_merge_ftsDs   F D  ) )= *     } D D D NN=s C C C C C C C C C  Ds(AAAc|5}|dd}dddn #1swxYwYd|D}|d}|dSt |dpd}t |}|dkrdSt dt d|z|z }d |||d S) a!Return deferred-rebuild progress, or None when no rebuild pending. Shape: {"pending": True, "total": , "indexed": , "percent": <0-100 int>}. Consumed by search_messages() notes and by status surfaces (dashboard/desktop can poll this to render a progress indicator). Reads state_meta directly via _read_ctx instead of calling get_meta() (which takes self._lock) so search_messages doesn't block on the writer lock when checking rebuild status. 5SELECT key, value FROM state_meta WHERE key IN (?, ?))fts_rebuild_high_waterfts_rebuild_progressNc,i|]}|d|dSkeyvaluer%r&rs r) z9SessionSearchMixin.fts_rebuild_status..c"222%!G*222r+rGrHrdTpendingtotalindexedpercent _read_ctxexecutefetchallgetintminrBconnrowmeta high_waterprogressrTpcts r)fts_rebuild_statusz%SessionSearchMixin.fts_rebuild_statusRs*^^   ,,GBhjj                 32c222XX677  4txx 677<1==J A::4#s3>E12233%HQTUUU)A  AAc~|jfd}||tddS)u4Finalize the deferred rebuild: boundary sweep + clear markers. The sweep is cheap insurance against any write that slipped through the migration-boundary instant (between high_water capture and trigger activation): re-index any row near the boundary that the index is missing. docsize has one row per indexed doc, so the anti-join is exact and runs on a narrow id range. The trigram half of the sweep is gated on ``self._trigram_available`` for the same reason ``fts_rebuild_step()`` gates its backfill INSERT: when the SQLite build has no trigram tokenizer (or the table was never created), an unconditional INSERT raises ``no such table`` and aborts the whole rebuild — taking ``optimize_fts_storage()`` down with it. c&|d}|Qt|d}|dz |dz}}|d||fr|d||f|ddS)NASELECT value FROM state_meta WHERE key = 'fts_rebuild_high_water'rzINSERT INTO messages_fts(rowid, content, tool_name, tool_calls) SELECT m.id, m.content, m.tool_name, m.tool_calls FROM messages m WHERE m.id > ? AND m.id <= ? AND NOT EXISTS (SELECT 1 FROM messages_fts_docsize d WHERE d.id = m.id)a INSERT INTO messages_fts_trigram(rowid, content, tool_name, tool_calls) SELECT m.id, m.content, m.tool_name, m.tool_calls FROM messages m WHERE m.id > ? AND m.id <= ? AND m.role <> 'tool' AND NOT EXISTS (SELECT 1 FROM messages_fts_trigram_docsize d WHERE d.id = m.id)zVDELETE FROM state_meta WHERE key IN ('fts_rebuild_high_water', 'fts_rebuild_progress')rYfetchoner\)r_hw_rowhwlohiinclude_trigrams r)_doz3SessionSearchMixin._fts_rebuild_finish.._dos\\Shjj !^^dBIB ^ H #LLj R  LLE     r+u7Deferred FTS rebuild complete — all messages indexed.N)_trigram_available_execute_writer@info)rBrrrqs @r)_fts_rebuild_finishz&SessionSearchMixin._fts_rebuild_finishnsU 1     : C    MNNNNNr+c j5djdjdddzfD}dddn #1swxYwY|sdS|dfd } t |S#tj $r&}t d |Yd}~d Sd}~wwxYw) a"Tear down one chunk of a demoted v22 FTS shadow table. The trash tables are PLAIN tables (their vtable parent was demoted away during the migration), so chunked DELETE + final DROP involve no FTS5 machinery at all. Returns True while teardown work remains. cg|] }|d Srr%rMs r) z?SessionSearchMixin._fts_teardown_trash_step..s)!r+zNSELECT name FROM sqlite_master WHERE type = 'table' AND name LIKE ? ESCAPE '\'_\_%NFrc Pd|ddD}|rd|nd}|dd|d|d d jd }|jd kr3|d td dS)Nc6g|]}|ddk|dS)rr%rMs r)rzzLSessionSearchMixin._fts_teardown_trash_step.._do..s1Q4!88!88r+zPRAGMA table_info()r"rowidz DELETE FROM z WHERE (z ) IN (SELECT z FROM z LIMIT rzDROP TABLE IF EXISTS z"Old FTS shadow table %s torn down.T)rYr3_FTS_REBUILD_CHUNK_ROWSrowcountr@ru)r_pk_colsrKcurrBtbls r)rrz8SessionSearchMixin._fts_teardown_trash_step.._dos"ll+F+F+F+FGGG)0<$))G$$$WC,,RsRRCRRRR&)RR262NRRRC|q   :S::;;; @#FFF4r+z0FTS trash teardown chunk failed (will retry): %sT) _lock_connrY_FTS_TRASH_PREFIXreplacerZboolrtr>OperationalErrorr@debug)rBtrashrrrCrs` @r)_fts_teardown_trash_stepz+SessionSearchMixin._fts_teardown_trash_steps\Z  "j002+33C??#EG(** E                5Ah        ++C0011 1'    LLKS Q Q Q44444 s*AA++A/2A/ !B++C :CC c|jsdS|d}|dSt||j|jfd} ||}n8#t j$r&}t d|Yd}~dSd}~wwxYw|dur>| }|&|d|dkr| dSt|S) a\Backfill one chunk of the deferred FTS rebuild. Returns True when more work remains, False when the rebuild is complete (or none is pending). Safe to call from any process at any time; chunks are claimed atomically inside the write transaction, so concurrent callers interleave instead of duplicating rows. FrGNcr|d}|dSt|d}|krdSt|z}|d||fr|d||f|dt |f|kS)N?SELECT value FROM state_meta WHERE key = 'fts_rebuild_progress'FrzINSERT INTO messages_fts(rowid, content, tool_name, tool_calls) SELECT id, content, tool_name, tool_calls FROM messages WHERE id > ? AND id <= ?zINSERT INTO messages_fts_trigram(rowid, content, tool_name, tool_calls) SELECT id, content, tool_name, tool_calls FROM messages WHERE id > ? AND id <= ? AND role <> 'tool'zBUPDATE state_meta SET value = ? WHERE key = 'fts_rebuild_progress'rYrlr\r]r-)r_r`rcupperchunkrbrqs r)rrz0SessionSearchMixin.fts_rebuild_step.._dos,,Qhjj {u3q6{{H:%%u5(*55E LL+5!       Bu%  LL5U     :% %r+z)FTS rebuild chunk failed (will retry): %sTrUrT) r;get_metar\rsrrtr>rr@rrervr) rBhigh_water_rawrrmorerCstatusrrbrqs @@@r)fts_rebuild_stepz#SessionSearchMixin.fts_rebuild_steps+  5'?@@  !5(( 1,% &% &% &% &% &% &% &N &&s++DD'    LLDc J J J44444  5==,,..F!fY&76'?&J&J((***5Dzzs A!!B0BBc|5}|dd}dddn #1swxYwYd|D}|d}|dSt |dpd}t |}|dkrdSt dt d|z|z }d |||d S) z:CJK-index backfill progress, or None when none is pending.rF)fts_cjk_rebuild_high_waterfts_cjk_rebuild_progressNc,i|]}|d|dSrJr%rMs r)rOz=SessionSearchMixin.fts_cjk_rebuild_status..rPr+rrrrQTrRrWr^s r)fts_cjk_rebuild_statusz)SessionSearchMixin.fts_cjk_rebuild_statuss( ^^   ,,GJhjj                 32c222XX:;;  4txx :;;@qAAJ A::4#s3>E12233%HQTUUUrfc|jr|jsdS|d}|dSt||jfd} ||}n8#t j$r&}t d|Yd}~dSd}~wwxYw|dur>| }|&|d|dkr| dSt|S) z=Backfill one chunk of the CJK index. True while work remains.FrNc>|d}|dSt|d}|krdSt|z}|d||f|dt |f|kS)NzCSELECT value FROM state_meta WHERE key = 'fts_cjk_rebuild_progress'FrzINSERT INTO messages_fts_cjk(rowid, content, tool_name, tool_calls) SELECT id, content, tool_name, tool_calls FROM messages WHERE id > ? AND id <= ? AND role <> 'tool'zFUPDATE state_meta SET value = ? WHERE key = 'fts_cjk_rebuild_progress'r)r_r`rcrrrbs r)rrz4SessionSearchMixin.fts_cjk_rebuild_step.._do*s,,9hjj {u3q6{{H:%%u5(*55E LL>5!     LL9U     :% %r+z-CJK FTS rebuild chunk failed (will retry): %sTrUrT) r;_fts_cjk_loadedrr\rrtr>rr@rr_fts_cjk_rebuild_finishr)rBrrrrrCrrrbs @@r)fts_cjk_rebuild_stepz'SessionSearchMixin.fts_cjk_rebuild_step s$  (< 5'CDD  !5(( , & & & & & &0 &&s++DD'    LLH# N N N44444  5==0022F!fY&76'?&J&J,,...5Dzzs AB.BBcxd}||d|_tddS)z?Boundary sweep + clear the cjk markers; index becomes servable.c|d}|7t|d}|dz |dz}}|d||f|ddS)NzESELECT value FROM state_meta WHERE key = 'fts_cjk_rebuild_high_water'rrjaINSERT INTO messages_fts_cjk(rowid, content, tool_name, tool_calls) SELECT m.id, m.content, m.tool_name, m.tool_calls FROM messages m WHERE m.id > ? AND m.id <= ? AND m.role <> 'tool' AND NOT EXISTS (SELECT 1 FROM messages_fts_cjk_docsize d WHERE d.id = m.id)z^DELETE FROM state_meta WHERE key IN ('fts_cjk_rebuild_high_water', 'fts_cjk_rebuild_progress')rk)r_rmrnrorps r)rrz7SessionSearchMixin._fts_cjk_rebuild_finish.._doPs\\;hjj !^^dBIB b H  LLM     r+Tu7CJK FTS index backfill complete — serving CJK search.N)rt_fts_cjk_availabler@rurBrrs r)rz*SessionSearchMixin._fts_cjk_rebuild_finishNsH   ( C   "& MNNNNNr+c|jsdSd}||}|rU|j5||j|jddddS#1swxYwYdSdS)aRebuild path for a stale cjk index (triggers were dropped). The gap's extent is unknown, so the only safe recovery is a from- scratch rebuild: drop the table + triggers, clear the breadcrumb, recreate via ``_ensure_fts_cjk_schema`` (which sets fresh backfill markers on a populated DB). Called from ``optimize_fts_storage`` on a tokenizer-capable host; no-op when not stale. Nc>|dtf}|sdStD]}|d||d|d|dtddS) Nz&SELECT 1 FROM state_meta WHERE key = ?FzDROP TRIGGER IF EXISTS z%DROP TABLE IF EXISTS messages_fts_cjkz(DROP VIEW IF EXISTS messages_fts_cjk_srcz&DELETE FROM state_meta WHERE key IN ('z<', 'fts_cjk_rebuild_high_water', 'fts_cjk_rebuild_progress')T)rYr rlr)r_staletrigs r)rrz7SessionSearchMixin._fts_cjk_reset_if_stale.._dotsLL8"$hjj  u) ? ? =t==>>>> LL@ A A A LLC D D D LL.&...    4r+)rrtr_ensure_fts_cjk_schemarcommit)rBrr was_stales r)_fts_cjk_reset_if_stalez*SessionSearchMixin._fts_cjk_reset_if_stalehs#  F   "'',,  $  $ $++DJ777 !!### $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $  $ $s4A,,A03A0c |dd}|sdS|dd}| S#tj$rYdSwxYw)aTrue when the base FTS table exists but indexes nothing while ``messages`` has rows. Caller must hold ``self._lock``. This is the post-demote empty-index shape: external-content FTS with zero ``messages_fts_docsize`` rows against a non-empty messages table. Healthy installs (and mid-backfill installs that still hold markers) never match. z%SELECT EXISTS(SELECT 1 FROM messages)rFz1SELECT EXISTS(SELECT 1 FROM messages_fts_docsize))rYrlr>r)rBr_has_msghas_ftss r)'_fts_external_index_empty_with_messagesz:SessionSearchMixin._fts_external_index_empty_with_messagess ll7hjjG ullChjjG; '   55 s/A#/A##A65A6c |dd}t|dkS#tj$rYdSwxYw)zTrue when the base external-content index holds no rows. A missing table counts as empty: the schema ensure that follows creates it fresh. z)SELECT COUNT(*) FROM messages_fts_docsizerT)rYrlr\r>r)rBr_ns r)_fts_index_known_emptyz)SessionSearchMixin._fts_index_known_emptysd   ;hjjAq66Q; '   44 s?AAActdD]4} |d|d|d #tj$rY1wxYwdS)u&Delete every indexed row from the v23 external-content tables. Uses the FTS5 ``'delete-all'`` special command — the documented O(1) truncate for external-content tables. A plain no-WHERE ``DELETE`` is O(rows) on external-content FTS5 (each row's delete tokens are regenerated from the content table; measured ~12µs/row, minutes on a large index, while holding the write lock) and corrupts the index if indexed rows have diverged from ``messages`` — precisely the broken- bookkeeping shape this repair path handles. The backfill chunk worker replays its whole selected id range with no anti-join, so a replay from zero is only safe once the index is known empty — this is how a partially indexed DB gets there. ) messages_ftsmessages_fts_trigram INSERT INTO (z) VALUES('delete-all')N)rYr>r)rBr_rs r)_reset_fts_index_to_emptyz,SessionSearchMixin._reset_fts_index_to_emptysn<  C  MCMM#MMMNNNN+      s #55Fforcerc0|d}||st|d}|d}|?||s|||d|S|dd}dt |fdfD]\}}|d ||ft|S) ahWrite ``fts_rebuild_high_water`` / ``fts_rebuild_progress`` for a full backfill. Returns the high-water id. When ``force`` is False and high_water is already set, only repairs a missing progress key (stuck no-op when high_water exists alone), and only after the index is known empty: the chunk worker replays its whole selected id range without an anti-join, so a partially indexed DB is first reset to a known-empty surface rather than rebuilt from zero on top of surviving rows. Caller must hold the write transaction / lock as appropriate. riNrrz~INSERT INTO state_meta (key, value) VALUES ('fts_rebuild_progress', '0') ON CONFLICT(key) DO UPDATE SET value = excluded.valuez)SELECT COALESCE(MAX(id), 0) FROM messagesrG)rH0zgINSERT INTO state_meta (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value)rYrlr\rrr-)rBr_r existing_hwrnrckvs r)_seed_fts_rebuild_markersz,SessionSearchMixin._seed_fts_rebuild_markerssBll O  (**   "5 "[^$$B||Qhjj 224889224888 L I \\ 7  (**Q&s2ww / )   DAq LLHA     2wwr+c<fd}|dS)uHeal interrupted demote/backfill bookkeeping before optimize runs. Covers two post-#65798 failure classes: 1. Empty external-content index with messages present and no rebuild markers (demote crash window after empty v23 tables landed but before markers, or settle that stamped without backfill). Seed a full backfill. 2. ``fts_rebuild_high_water`` present without ``fts_rebuild_progress`` (partial meta) — seed progress so the chunk loop is not a no-op, resetting a partially populated index to a known-empty surface first so the anti-join-free chunk replay cannot duplicate rows. Must not invent markers on a still-legacy inline DB: that would make ``optimize_fts_storage`` skip demote (``legacy and not pending``) and attempt v23-shaped INSERTs against the inline table forever. c|d}|j|d}|?|s||ddS|rdS|r.|d|ddSdS)Nriz;SELECT 1 FROM state_meta WHERE key = 'fts_rebuild_progress'zsINSERT INTO state_meta (key, value) VALUES ('fts_rebuild_progress', '0') ON CONFLICT(key) DO UPDATE SET value = '0'z8DELETE FROM state_meta WHERE key = 'fts_storage_version'Tr)rYrlrr_db_has_legacy_inline_ftsrr)r_rrcrBs r)rrz._dos*,,7hjj  & <<9(**#66t<<=66t<<<LLE --d33 ;;DAA A N..t4.@@@@@  A Ar+N)rtrs` r)_repair_optimize_bookkeepingz/SessionSearchMixin._repair_optimize_bookkeepings>$! A! A! A! A! AD C     r+c@|jr|jrdS|j5||jr ddddS|jdr ddddS|jrC|jdtdr ddddS| |jr ddddS| |jcdddS#1swxYwYdS)azTrue when `optimize_fts_storage()` has work to do: either this DB is a legacy inline-FTS install that can be optimized to the v23 external-content schema, or a previous optimize run was interrupted (legacy vtables already demoted, but backfill markers and/or trash tables remain) and re-running would resume it, or the CJK-bigram index needs a backfill/rebuild on this tokenizer-capable host, or a prior demote left an empty external-content index without markers (healable on re-run). False for fresh and fully-optimized installs (and when FTS5 is unavailable).FNTESELECT 1 FROM state_meta WHERE key = 'fts_rebuild_high_water' LIMIT 1zFSELECT 1 FROM state_meta WHERE key IN ('fts_cjk_rebuild_high_water', 'z ') LIMIT 1) r; read_onlyrrrrYrlrr _has_fts_trashr)rBs r)fts_optimize_availablez)SessionSearchMixin.fts_optimize_available3s,  DN 5 Z L L--dj99  L L L L L L L Lz!!?hjj  L L L L L L L L #  (:(:Q3DQQQ))hjj ) L L L L L L L L*""4:.. - L L L L L L L L4?? KK5 L L L L L L L L L L L L L L L L L Ls)D-D;=DD-DDDcfd}t|}j5jdt }jdt }t|_|stj dj dddn #1swxYwY|S)uDemote the legacy inline FTS vtables and stage their shadow tables for chunked teardown. Returns MAX(messages.id) as the rebuild high water. O(1) schema surgery — the heavy delete is deferred to the chunked teardown, exactly as the validated auto path did. Markers are written in the same BEGIN IMMEDIATE as the demote, *before* the empty v23 schema is created. Schema creation uses ``executescript`` and therefore cannot run inside that transaction (it issues an implicit COMMIT — see the CJK recreate path). Creating the empty schema only after markers are durable closes the crash window where trash + empty v23 tables exist with no backfill claim. c@||dt|d}|r|d|d|dd|dD}|D]}|d|d ||d }|d |S) Nz,DROP VIEW IF EXISTS messages_fts_trigram_srczSELECT 1 FROM sqlite_master WHERE type = 'table' AND name IN ('messages_fts', 'messages_fts_trigram') AND sql LIKE 'CREATE VIRTUAL TABLE%' LIMIT 1zPRAGMA writable_schema=ONzDELETE FROM sqlite_master WHERE type = 'table' AND name IN ('messages_fts', 'messages_fts_trigram') AND sql LIKE 'CREATE VIRTUAL TABLE%'zPRAGMA writable_schema=RESETcg|] }|d Sryr%rMs r)rzzRSessionSearchMixin._demote_legacy_fts_to_trash.._stage..ys)AaDr+zSELECT name FROM sqlite_master WHERE type = 'table' AND (name LIKE 'messages_fts_%' ESCAPE '\' OR name LIKE 'messages_fts_trigram_%' ESCAPE '\')z ALTER TABLE z RENAME TO fts_v22_trash_Tr;DELETE FROM state_meta WHERE key = 'fts_optimize_available')_drop_fts_triggersrYrrlrZr)r_hadshadowsshrnrBs r)_stagez>SessionSearchMixin._demote_legacy_fts_to_trash.._stageisW  # #D ) ) ) LLG H H Ht||?hjj C  S 8999 ;  ;<<<"&,,M##hjj "SSBLL!Q!Q!QR!Q!QRRRR //D/AAB LLM   Ir+rrz@failed to create v23 messages_fts during optimize-storage demoteN) r\rtr_ensure_fts_schemarr rrrsr>rr)rBrrnbase_ok trigram_oks` r)_demote_legacy_fts_to_trashz.SessionSearchMixin._demote_legacy_fts_to_trash\s ! ! ! ! ! F$$V,, - -Z --dj.'RRG00 2OJ'+:&6&6D # .V J       sBCC CT) progress_cbvacuumrrc jsdddSjrdddSj5j}dddn #1swxYwYddu}|r|sn|r|sj5jdt}jdt}t|_ |stjd jdddn #1swxYwYjrRj5jjdddn #1swxYwYd t&d dffd }d t(d dffd }|d t+j} sn+|d|t+j| z S|d t+j} sn+|d|t+j| z S|d t+j} sn+|d|t+j| z Sj5jddu} j} j} dddn #1swxYwY| s| s| r,| s| rdnd} t<d| | | | d| ddSd}|r|d j5jddddn #1swxYwYd}n9#tj$r'}t<d|d}Yd}~nd}~wwxYw j5jddddn #1swxYwYn2#t@$r%}t<!d|Yd}~nd}~wwxYwfd}"|}|!t<d|d||dS|dt<#dtHd|d S)!uMigrate a legacy v22 inline-FTS DB to the v23 external-content schema, foreground and to completion. Safe to re-run: if a previous attempt was interrupted it resumes from the progress marker. ``progress_cb`` receives {"phase", "percent", "indexed", "total"} dicts for a CLI progress bar. Returns a summary dict. The trigram tokenizer being unavailable is not fatal — the base index is still rebuilt (CJK falls back to LIKE), mirroring normal startup. Ffts5_unavailable)okreasonrNrGrrz?failed to re-create v23 messages_fts on optimize-storage resumephaser cdS}|}||r|dnd|r|dnd|r|dndddS)NrVrQrUrrT)rrVrUrT)rer)rstrrBs r)_emitz6SessionSearchMixin.optimize_fts_storage.._emits"((**Bz0022 K,.72i==C,.52i==A(*1G      r+ chunk_secondscftjtj|jzdS)aInter-chunk throttle (see the chunk-engine note above). The chunk methods themselves never sleep, so this loop is the single place the duty cycle is enforced: without it, back-to-back BEGIN IMMEDIATE chunks starve any live gateway/CLI process sharing the DB out of its lock retries (the measured ~85% write-lock ownership that froze concurrent sessions). N)timesleepmax_FTS_REBUILD_MIN_PAUSE_FTS_REBUILD_DUTY_FACTOR)rrBs r)_pausez7SessionSearchMixin.optimize_fts_storage.._pauses@ Js+ ==     r+backfillTteardownrbackfill_incompleteteardown_incompletezPFTS storage optimization did not settle (%s): pending=%s trash=%s empty_index=%s)rrvacuumedrVACUUMz$VACUUM after FTS optimize failed: %szPRAGMA wal_checkpoint(TRUNCATE)z:WAL checkpoint (TRUNCATE) after optimize VACUUM failed: %scv|ddS|rdS|rdS|dt t f|d|dt t fdS)Nrrrz{INSERT INTO state_meta (key, value) VALUES ('fts_storage_version', ?) ON CONFLICT(key) DO UPDATE SET value = excluded.valuerz7UPDATE schema_version SET version = ? WHERE version < ?)rYrlrrr-r r)r_rBs r)_settlez8SessionSearchMixin.optimize_fts_storage.._settleWs||?hjj%-,""4(( -,,;;DAA -,, LLH())+    LLV W W W LLI0   4r+z,FTS storage optimization settle refused (%s)donez/FTS storage optimization complete (layout v%d).)rr)%r;rrrrrrrrr rrrsr>rrrrrr-floatr monotonicrrrrYrlrrr@rA Exceptionrrtrur )rBrrlegacyrSrrrr_t0 still_pending still_trash empty_indexr vacuum_okrCrrefusals`` r)optimize_fts_storagez'SessionSearchMixin.optimize_fts_storages   ?+=>> > > 8;77 7 ))+++ Z @ @33DJ??F @ @ @ @ @ @ @ @ @ @ @ @ @ @ @-- 899E  $' $  , , . . . .  $V $ $ $11J"44J 6 +/z*:*:'"25 !!### $ $ $ $ $ $ $ $ $ $ $ $ $ $ $( $$&&&   $ $ $++DJ777 !!### $ $ $ $ $ $ $ $ $ $ $ $ $ $ $          % D       j +.""C((**  E*    F4>##c) * * *  + j +.""C,,..  E*    F4>##c) * * *  + j +.""C0022  E*    F4>##c) * * *  +Z S S J..?hjj%M--dj99KFFtzRRK  S S S S S S S S S S S S S S S  EK E; E)6+++%%*  NN5 {K     6tDD D   E(OOO "Z11J&&x000111111111111111 + " " "EsKKK!  " ZJJJ&&'HIIIJJJJJJJJJJJJJJJ    P      0%%g..   NN>    7 JJ J f  =?R    222sAA!$A!#BD55D9<D9"4F""F&)F&0A#MM#&M#,O(3O O(OO(!O"O((P7PP"Q)Q QQQQQ R &RR ruser assistantraround_message_idwindowbookend keep_rolescf|dkrd}||}|d}|sgddggdS|t|fd|D}n|}|dd} |dd} g} g} |dkr5} d }g}|4d d |D}d |d }t |}| d|d|| g||R} | d|d|| g||R} t t| } dddn #1swxYwYdtttfffd ||d|dfd| Dfd| DdS)u)Return an anchored window plus session bookends. Built on top of ``get_messages_around``. Three slices: - ``window``: messages immediately surrounding the anchor. Filtered to ``keep_roles`` (tool-response noise dropped by default), EXCEPT the anchor itself is always preserved regardless of role. - ``bookend_start``: first ``bookend`` user/assistant messages of the session — but only those whose id is strictly before the window's first message id. Empty when the window already overlaps the session head. Empty-content messages (tool-call-only assistant turns) are skipped so they don't crowd out actual prose openings. - ``bookend_end``: last ``bookend`` user/assistant messages of the session, same non-overlap rule at the tail. Bookends let an FTS5 hit anywhere in a long session yield the goal (opening) and the resolution (closing) on a single call — without loading the whole transcript. Returns ``{"window": [], "messages_before": 0, "messages_after": 0, "bookend_start": [], "bookend_end": []}`` when the anchor isn't in the session. ``keep_roles=None`` disables role filtering (raw window + raw bookends). r)r r )r messages_beforemessages_after bookend_start bookend_endNctg|]4}|dks|dv2|5S)rr)r[)r&mrkeep_sets r)rzz8SessionSearchMixin.get_anchored_view..sM55;;"333quuV}}7P7P7P7P7Pr+r,c3K|]}dVdS?Nr%r&r{s r)r*z7SessionSearchMixin.get_anchored_view..s"0I0I0I0I0I0I0I0Ir+z AND role IN (rz6SELECT * FROM messages WHERE session_id = ? AND id < ?z0 AND length(content) > 0 ORDER BY id ASC LIMIT ?z6SELECT * FROM messages WHERE session_id = ? AND id > ?z1 AND length(content) > 0 ORDER BY id DESC LIMIT ?r ct|}d|vr|d|d<|drZ tj|d|d<n;#tjt f$r"tdg|d<YnwxYw|d |d|d<|S)Ncontent tool_callszIFailed to deserialize tool_calls in get_anchored_view, falling back to []display_metadata) dict_decode_contentr[jsonloadsJSONDecodeErrorr.r@rA_decode_display_metadata)r`msgrBs r)_hydratez6SessionSearchMixin.get_anchored_view.._hydratess))CC!%!5!5c)n!E!EIww|$$ ++(, 3|3D(E(EC %%,i8+++NNc)+C %%% + ww)**6*.*G*GL^H_*`*`&'Js A''5BBr rc&g|] }|Sr%r%r&rNr&s r)rzz8SessionSearchMixin.get_anchored_view..s!FFFahhqkkFFFr+c&g|] }|Sr%r%r(s r)rzz8SessionSearchMixin.get_anchored_view..s!BBBAHHQKKBBBr+) get_messages_aroundr/rXr3listrYrZreversedrr-r)rBrrr r r  primitive window_rowsfiltered_window window_min_id window_max_idbookend_start_rowsbookend_end_rowsr_ role_clause role_paramsrole_placeholdersr&rs` ` @@r)get_anchored_viewz$SessionSearchMixin.get_anchored_viewsD Q;;G,, )&-   )  #$"#!#!    !:H&OO *O#At, #B- )+&( Q;;!! DT $& )(+0I0Ij0I0I0I(I(I%"G3D"G"G"GK"&z"2"2K%)\\/6A/// FFgFF && (** #$(<<06A000 FFgFF $$ (** !$(1A(B(B#C#C 1 D D D D D D D D D D D D D D D4 T#s(^      "&():;'(89FFFF3EFFFBBBB1ABBB    sB=EE!$E!limitinclude_inactivec|rdnd}d}|j5|jd||d|t|f}|}dddn #1swxYwYg}|D] } || d} t | trBd| D} d d | D } | sd } n)t | trt| p| } nd} d | } t| d kr | dd dz} || d| d| d |S)uReturn the *limit* most-recent user messages, newest first. Each entry is a dict with keys ``id``, ``timestamp``, ``preview``. ``preview`` is the first 80 characters of the message content (with line breaks collapsed to spaces). Used by the /rewind slash command picker, CLI/TUI/gateway ``/undo [N]``, and any other caller that needs real user-turn targets. Bookkeeping timeline rows (``display_kind`` set — e.g. model_switch, async_delegation_complete, auto_continue, hidden) are excluded. They are durable ``role='user'`` rows for the API transcript, but no client counts them as user turns (desktop demotes them to system / drops them; the CLI already uses ``not m.get("display_kind")``). Including them here made ``/undo`` soft-delete from a marker instead of the last real turn — same class of index skew as the prompt.submit ordinal bug. By default only active messages are returned. rz AND active = 1z0 AND (display_kind IS NULL OR display_kind = '')zRSELECT id, timestamp, content FROM messages WHERE session_id = ? AND role = 'user'z ORDER BY id DESC LIMIT ?Nrcg|]F}t|tr/|ddk0|ddGStypetextrr,rr[r&ps r)rzz@SessionSearchMixin.list_recent_user_messages..#sW*+!!T**/0uuV}}/F/FEE&"%%/F/F/Fr+ c3K|]}||V dSr$r%r&ts r)r*z?SessionSearchMixin.list_recent_user_messages..'s'">">A">1">">">">">">r+[multimodal content]PMz...rr)rrpreview)rrrYr\rZr r,r+r3stripr-r splitlenappend) rBrr9r: active_clausedisplay_clausecursorrowsresultr`decoded text_partsrJs r)list_recent_user_messagesz,SessionSearchMixin.list_recent_user_messagess 0/E4E K Z % %Z''+ +"0+++SZZ( F??$$D % % % % % % % % % % % % % % %(*  C**3y>::G'4(( /6 ((">">j">">">>>DDFF54GGS)) 4G<<Ghhw}}//G7||b  !#2#,. MMd)!$[!1&     sAA!!A%(A%queryc|dt}g}g}d}|t|kr||}|dkr|||dz }<|d|dz}|dkr|d|dz }v||||dz|dt|dz d|dz}|t|kd |}t jd d|}t jd d |}t jd d|}t jdd |}t jdd |}t jdd|}t|D]\}}| d|d|} |S)aSanitize user input for safe use in FTS5 MATCH queries. FTS5 has its own query syntax where characters like ``"``, ``(``, ``)``, ``+``, ``*``, ``{``, ``}``, the column-filter operator ``:`` and bare boolean operators (``AND``, ``OR``, ``NOT``) have special meaning. Passing raw user input directly to MATCH can cause ``sqlite3.OperationalError``. Strategy: - Preserve properly paired quoted phrases (``"exact phrase"``) - Strip unmatched FTS5-special characters that would cause errors - Wrap unquoted hyphenated and dotted terms in quotes so FTS5 matches them as exact phrases instead of splitting on the hyphen/dot (e.g. ``chat-send``, ``P2.2``, ``my-app.config.ts``) Nr"rrrCzQrz [+{}():\"^]z\*+*z(^|\s)\*z\1z(?i)^(AND|OR|NOT)\b\s*z(?i)\s+(AND|OR|NOT)\s*$z\b(\w+(?:[._-]\w+)+)\bz"\1") rrMrNfindr3resubrK enumerater)rW _quoted_partspiecesichend sanitizedquoteds r)_sanitize_fts5_queryz'SessionSearchMixin._sanitize_fts5_query<s(+++, !  #e**nnqBSyy b!!!Q**S!a%((Cbyy c"""Q  qqy!1 2 2 2 MM>#m"4"4q"8>>> ? ? ?aA#e**nn"GGFOO F>3 :: F63 22 F;y99 F4b)//:K:KLL F5r9??;L;LMM F4gyII #=11 C CIAv!))/!///6BBII   r+cpcd|cxkodkncp_d|cxkodkncpOd|cxkodkncp?d|cxkodkncp/d |cxkod kncpd |cxkod kncpd |cxkodkncS)NN4Mߦ0?0@0000r%)rhs r)_is_cjk_codepointz$SessionSearchMixin._is_cjk_codepoints"&&&&&&&&'"&&&&&&&&'2(((((((('"&&&&&&&&'"&&&&&&&& ' "&&&&&&&& ' "&&&&&&&&  (r+r?c|D]~}t|}d|cxkrdks]nd|cxkrdksNnd|cxkrdks?nd|cxkrdks0nd |cxkrd ks!nd |cxkrd ksnd |cxkrdkrn{dSdS)zBCheck if text contains CJK (Chinese, Japanese, Korean) characters.rjrkrlrmrnrorprqrrrsrtrurvrwTF)ord)r?rcrhs r) _contains_cjkz SessionSearchMixin._contains_cjks   BRB"&&&&&&&&"&&&&&&&&2(((((((("&&&&&&&&"&&&&&&&&"&&&&&&&&"&&&&&&&&&tt'ur+c:tfd|DS)zCount CJK characters in text.c3`K|](}t|$dV)dS)rNrxrz)r&rcr6s r)r*z0SessionSearchMixin._count_cjk..s<FFs'<'=2 chars and unigrams only for isolated chars, so a 1-char CJK term can't match inside longer runs there — those queries keep the LIKE substring route. rrTr~)r6rWrunrcs r)_has_lone_cjk_runz$SessionSearchMixin._has_lone_cjk_runs_  B$$SWW-- q!8844axr+cd|dD}t|otd|DS)aTrue when every non-operator token is long enough for the trigram tokenizer to match (>=3 chars). The trigram tokenizer indexes overlapping 3-character sequences, so a token shorter than 3 chars produces no trigrams and can never match. With FTS5's implicit-AND between tokens, a single short token makes the whole MATCH return nothing, so the trigram path is only worth taking when every searchable token qualifies. c>g|]}|dv|S>ORANDNOTrrEs r)rzz?SessionSearchMixin._trigram_eligible_tokens..s6   wwyy 444 444r+rYc3<K|]}t|dkVdSrN)rMrEs r)r*z>SessionSearchMixin._trigram_eligible_tokens..s,#@#@ACFFaK#@#@#@#@#@#@r+)rKrLrall)rWtokenss r)_trigram_eligible_tokensz+SessionSearchMixin._trigram_eligible_tokenssn  {{3''--//5577   F||@#@#@#@#@#@ @ @@r+rr)table source_filterexclude_sources role_filterr9offset raw_queryr order_by_sqlrrrrc |} g} | D]]} | dvr| | .| d| ddzdz^d| } |dg}| g}|s|d|K|dd d |Dd |||K|d d d |Dd |||rK|dd d|Dd ||d|d|d|dd|d|d }||| g|5} |||}n!#tj $rYddddSwxYwd| DcdddS#1swxYwYdS)uRun a search against a substring-capable FTS index. ``table`` is ``messages_fts_trigram`` (default) or ``messages_fts_cjk``. The trigram tokenizer indexes overlapping 3-byte sequences, so it matches substrings regardless of word boundaries — both CJK phrases the unicode61 tokenizer splits into single characters and Latin runs the unicode61 tokenizer fuses onto adjacent CJK (e.g. ``修改youer服务端``). The cjk-bigram tokenizer splits Latin runs off adjacent CJK, giving the same recovery as an exact ranked token match. Each non-operator token is quoted to neutralise FTS5 special characters while boolean operators (AND/OR/NOT) are preserved. Returns the matching rows, or ``None`` when the query cannot be executed (e.g. the tokenizer is unavailable at runtime) so the caller can fall back to another strategy. >rrrrY""rCz MATCH ?!(m.active = 1 OR m.compacted = 1)N s.source IN (rc3K|]}dVdSrr%rs r)r*z9SessionSearchMixin._run_trigram_search..s"5Q5Qac5Q5Q5Q5Q5Q5Qr+rs.source NOT IN (c3K|]}dVdSrr%rs r)r*z9SessionSearchMixin._run_trigram_search..s"9W9W!#9W9W9W9W9W9Wr+ m.role IN (c3K|]}dVdSrr%rs r)r*z9SessionSearchMixin._run_trigram_search..s"3M3MAC3M3M3M3M3M3Mr+zx SELECT m.id, m.session_id, m.role, snippet(z, -1, '>>>', '<<<', '...', 40) AS snippet, m.content, m.timestamp, m.tool_name, s.source, s.model, s.started_at AS session_started FROM z' JOIN messages m ON m.id = zL.rowid JOIN sessions s ON s.id = m.session_id WHERE  AND & LIMIT ? OFFSET ? c,g|]}t|Sr%rr&r`s r)rzz:SessionSearchMixin._run_trigram_search..s???#DII???r+) rLrrNrr3extendrXrYr>rrZ)rBrrrr:rrrr9rrpartstok trigram_query tri_where tri_paramstri_sqlr_ tri_cursors r)_run_trigram_searchz&SessionSearchMixin._run_trigram_searchsQ<"" A ACyy{{222 S!!!! S3;;sD#9#99C?@@@@ '''( )?  B   @ A A A  $   TSXX5Q5Q=5Q5Q5Q-Q-QTTT U U U   m , , ,  &   Z9W9W9W9W9W1W1WZZZ [ [ [   o . . .  +   P3883M3M3M3M3M+M+MPPP Q Q Q   k * * *      (-  << **  !   & 5&/*** ^^   @ !\\':>> +     @ @ @ @ @ @ @ @ @?)<)<)>)>???  @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @s6#I%G<;I<H IH II I sortc tj} d} ||||||||||  } | ttjdd} n#t t f$rd} YnwxYwtj| z dz} | | krKtd| || | t| nd|ddSS# ttjdd} n#t t f$rd} YnwxYwtj| z dz} | | krKtd| || | t| nd|ddwwxYw) aInstrumented wrapper around :meth:`_search_messages_impl`. Logs one line per slow search with the routing path taken, so production latency stays attributable per query shape (the 2026-07 session_search investigation needed trace archaeology to discover the LIKE full scans; this makes the next regression a grep). Threshold: HERMES_SEARCH_SLOW_MS (default 1000; 0 logs every call). N)rrrr9rrr:rHERMES_SEARCH_SLOW_MS1000g@@zBest-effort name of the routing path a query takes (log-only).remptyfts5rYfts_cjkcjg|]/}|dv|-|0Srrr{r&rFrBs r)rzz.XsK7799$888T=O=OPQ=R=R8888r+c3JK|]}|dkVdSrrrs r)r*z;SessionSearchMixin._describe_search_path..\s4??1**Q.??????r+rtrigram like_scanr7) rgr{rKrrrLanyrrsr)rBrWrerawrshorts` r)rz(SessionSearchMixin._describe_search_pathMs- 11%+2>>I w%%i00 v//#&&,,..C& !t/E/Ec/J/J ! y99;;F?????????Es##q(((4;R( y;   99 s#CCAC:AC C)(C)c pAB| AjsgS|r|sgS|}|sgSt |t r-|} | dvrd} nd} | dkrd} n | dkrd} nd} dg} |g} |s| d |Md d |D}| d |d | ||Md d|D}| d|d | ||rMd d|D}| d|d | |d| }| ||gd|d| d}g} |}| rV|d} |}fd| D}tfd|D}d}t|od|v}jrK|sH|s2| }g}|D]]}|dvr||.|d|ddzdz^d|}dg}|g} |s|d |K|d d d|Dd | ||K|dd d |Dd | ||rK|dd d!|Dd | |d"d|d#| d$}!| ||g 5}"|"|!| }#d%|#D}d&}dddn #1swxYwYn #t,j$rt0d'd&(Ynt,j$r}$|$r 5}"|"|!| }#d)|#D}d&}dddn #1swxYwYnJ#t,j$rt0d*YnwxYwt0d+|$Yd}$~$nd}$~$wwxYw|s*|d,kr#|s jr|s| }g}|D]]}|dvr||.|d|ddzdz^d|}%d-g}&|%g}'|s|&d |K|&d d d.|Dd |' ||K|&dd d/|Dd |' ||rK|&dd d0|Dd |' |d1d|&d#| d$}(|' ||g 5}"|"|(|'})d2|)D}d&}dddn #1swxYwYn#t,j$rYnt,j$r}$|$r 5}"|"|(|'})d3|)D}d&}dddn #1swxYwYnJ#t,j$rt0d4YnwxYwt0d5|$Yd}$~$nd}$~$wwxYw|sEd6| Dp|g}*g}+g},|*D]i}|d7d8d9d:d;d<}-|+d=|,d9|-d9d9|-d9d9|-d9gz },jd>d?|+d g}.|s|.d |K|.d d d@|Dd |, ||K|.dd dA|Dd |, ||rK|.dd dB|Dd |, |dCd|.dD}/|, ||g|*dEg|,z},5}"|"|/|,}0dF|0D}dddn #1swxYwYn 5}"|"|| }1dG|1D}dddn #1swxYwYn#t,j$rgcYSt,j$r}$|$s5}"|"|| }1dH|1D}dddn #1swxYwYYd}$~$nd}$~$wwxYw}2|2t?||kr ||t?|z ||||I}3dJ|DB| BfdK|3Dn7#t,j$r%}$t0dL|$Yd}$~$nd}$~$wwxYw|s|st|rd|vs|d}4jr"!|4dM| ||||||N }5|5r|5}|s=jr6"|r!!|4| ||||||O}6|6r|6}AdPAvr|ndQ}7|7D]=}8 5}"|"dR|8dS|8dSf}9g}:|9D]};|;dT}<#|<}=t |=tHrBdU|=D}>ddV|>D}?|?pdW}@nt |=t r|=}@ndX}@|:|;dY|@ddZd[ dddn #1swxYwY|:|8dP<)#tJ$r g|8dP<Y;wxYw|D]}8|8&dTdAAfd\|D}|S)]u Full-text search across session messages using FTS5. Supports FTS5 query syntax: - Simple keywords: "docker deployment" - Phrases: '"exact phrase"' - Boolean: "docker OR kubernetes", "python NOT java" - Prefix: "deploy*" Returns matching messages with session metadata, content snippet, and surrounding context (1 message before and after the match). ``fields`` selects a result projection; omitting it preserves the complete legacy result. Context is only loaded when that projection consumes it. ``sort`` controls temporal ordering: - ``None`` (default): FTS5 BM25 relevance only. Time-neutral. - ``"newest"``: order by message timestamp DESC, then by rank. - ``"oldest"``: order by message timestamp ASC, then by rank. The short-CJK LIKE fallback already orders by timestamp DESC and ignores ``sort``. The trigram CJK path honours ``sort`` like the main FTS5 path. Rewound (``active=0``, ``compacted=0``) rows are excluded by default — the user took those back. Compaction-archived rows (``active=0``, ``compacted=1``) ARE included by default: they were summarized away from the live context but remain part of the conversation's record, so the pre-compaction transcript stays discoverable after in-place compaction (#38763). Pass ``include_inactive=True`` to search every row regardless. )newestoldestNrzORDER BY m.timestamp DESC, rankrzORDER BY m.timestamp ASC, rankz ORDER BY rankzmessages_fts MATCH ?rrc3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl..s"*F*F13*F*F*F*F*F*Fr+rrc3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl..s"+I+IAC+I+I+I+I+I+Ir+rc3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl..s"(B(B(B(B(B(B(B(Br+rra SELECT m.id, m.session_id, m.role, snippet(messages_fts, -1, '>>>', '<<<', '...', 40) AS snippet, m.content, m.timestamp, m.tool_name, s.source, s.model, s.started_at AS session_started FROM messages_fts JOIN messages m ON m.id = messages_fts.rowid JOIN sessions s ON s.id = m.session_id WHERE rrrYcjg|]/}|dv|-|0Srrrs r)rzz.sK!!!7799$888T=O=OPQ=R=R8888r+c3JK|]}|dkVdSrrrs r)r*z;SessionSearchMixin._search_messages_impl..sE!!+,""Q&!!!!!!r+Ftool>rrrrrCzmessages_fts_cjk MATCH ?c3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl.."=Y=Yac=Y=Y=Y=Y=Y=Yr+c3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl.."A_A_!#A_A_A_A_A_A_r+c3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl.. ";U;UAC;U;U;U;U;U;Ur+a SELECT m.id, m.session_id, m.role, snippet(messages_fts_cjk, -1, '>>>', '<<<', '...', 40) AS snippet, m.content, m.timestamp, m.tool_name, s.source, s.model, s.started_at AS session_started FROM messages_fts_cjk JOIN messages m ON m.id = messages_fts_cjk.rowid JOIN sessions s ON s.id = m.session_id WHERE z z6 LIMIT ? OFFSET ? c,g|]}t|Sr%rrs r)rzz.9"N"N"N499"N"N"Nr+Tz;messages_fts_cjk query failed; falling back to trigram/LIKE)exc_infoc,g|]}t|Sr%rrs r)rzz.K++"+"+"25DII+"+"+"r+zYCJK-bigram FTS search still failing after in-place rebuild; falling back to trigram/LIKE.zuCJK-bigram FTS search hit a corruption error (%s) and no in-place rebuild was possible; falling back to trigram/LIKE.rzmessages_fts_trigram MATCH ?c3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl..srr+c3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl..vrr+c3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl..yrr+a SELECT m.id, m.session_id, m.role, snippet(messages_fts_trigram, -1, '>>>', '<<<', '...', 40) AS snippet, m.content, m.timestamp, m.tool_name, s.source, s.model, s.started_at AS session_started FROM messages_fts_trigram JOIN messages m ON m.id = messages_fts_trigram.rowid JOIN sessions s ON s.id = m.session_id WHERE c,g|]}t|Sr%rrs r)rzz.rr+c,g|]}t|Sr%rrs r)rzz.rr+zNTrigram FTS search still failing after in-place rebuild; falling back to LIKE.zjTrigram FTS search hit a corruption error (%s) and no in-place rebuild was possible; falling back to LIKE.c>g|]}|dv|SrrrEs r)rzz.s6!!!wwyy(<<<<<.s">Z>Zqs>Z>Z>Z>Z>Z>Zr+c3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl..s"B`B`13B`B`B`B`B`B`r+c3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_messages_impl..s".sKKKStCyyKKKr+c,g|]}t|Sr%rrs r)rzz.FFFStCyyFFFr+c,g|]}t|Sr%rrs r)rzz.rr+r:rrrch|] }|d S)rr%)r&rs r) z;SessionSearchMixin._search_messages_impl..s555AdG555r+c30K|]}|dv |VdS)rNr%)r&rseen_idss r)r*z;SessionSearchMixin._search_messages_impl..s1QQQ49P9Pq9P9P9P9PQQr+z$Unindexed-gap supplement skipped: %smessages_fts_cjk)rrr:rrrr9r)rr:rrrr9rrr%aWITH target AS ( SELECT session_id, timestamp, id FROM messages WHERE id = ? ) SELECT role, content FROM ( SELECT m.id, m.timestamp, m.role, m.content FROM messages m JOIN target t ON t.session_id = m.session_id WHERE (m.timestamp < t.timestamp) OR (m.timestamp = t.timestamp AND m.id < t.id) ORDER BY m.timestamp DESC, m.id DESC LIMIT 1 ) UNION ALL SELECT role, content FROM messages WHERE id = ? UNION ALL SELECT role, content FROM ( SELECT m.id, m.timestamp, m.role, m.content FROM messages m JOIN target t ON t.session_id = m.session_id WHERE (m.timestamp > t.timestamp) OR (m.timestamp = t.timestamp AND m.id > t.id) ORDER BY m.timestamp ASC, m.id ASC LIMIT 1 )rrcg|]F}t|tr/|ddk0|ddGSr=r@rAs r)rzz.sW***67#-a#6#6*;<55==F;R;R!"fb 1 1;R;R;Rr+c3K|]}||V dSr$r%rEs r)r*z;SessionSearchMixin._search_messages_impl..s'+G+G!Q+GA+G+G+G+G+G+Gr+rGrrr)rrc.g|]fdDS)c*i|]}|v||Sr%r%)r&r'matchs r)rOzGSessionSearchMixin._search_messages_impl...s$RRR5E>>e >>>r+r%)r&r result_fieldss @r)rzz.s?SRRR-RRRr+)'r8r;rKrgr,r-lowerrNr3rr{rrLrrrrrrrXrYrZr>rr@r DatabaseError_try_runtime_fts_rebuildrArsrerM_search_unindexed_gaprrr r+rpop)CrBrWrrrr9rrr:r sort_normr where_clausesparamssource_placeholdersexclude_placeholdersr6 where_sqlsqlmatchesis_cjkr cjk_count_tokens_for_check_any_short_cjk_trigram_succeeded_wants_tool_rowsrrr cjk_query cjk_where cjk_paramscjk_sqlr_ cjk_cursorrCrrrrr non_op_tokens token_clauses like_paramsesc like_wherelike_sql like_cursorrQrebuild_status gap_matches _fb_querycjk_fb tri_matchescontext_matchesr ctx_cursor context_msgsrNrrTrUr?rJrrsC` @@r)rz(SessionSearchMixin._search_messages_implcs)V33F;;   I EKKMM I))%00 I dC   **,,I 444 I  Z>ZM>Z>Z>Z6Z6Z&]&]&]^^^&&}555".%%&c#((B`B`P_B`B`B`:`:`&c&c&cddd&&7774%%&YCHH ^^%%L"&,,x"E"EKKKK4H4H4J4JKKKGLLLLLLLLLLLLLLL G^^%%G!\\#v66FFFFOO4E4EFFFGGGGGGGGGGGGGGGG+    ( G G G44S99^^%%G!\\#v66FFFFOO4E4EFFFGGGGGGGGGGGGGGGG G,0022  %#g,,*>*> J"88CLL(%5"/$3 + 9 65W555QQQQ+QQQQQQQ+ J J J CSIIIIIIII J*$ *$ *+&&$ *,2[+@+@ C((..00I& %11,!-%5"/$3 +!2  %$G *+ *11%88 * #66!-%5"/$3 +!7   *)G%, ]0J0JGGPR %: &: &E9 &^^%%5!% <teDk2? " "JB$&L'0022 l"&"6"6s";";>44 )**;B***J$'88+G+Gz+G+G+G#G#G#M#M#O#OD&*&D.DGG'55)&-GG&(G$++%&vY74C4=II!G555555555555555l$0i   & & &#%i    & ' 'E IIi & & & &  $$G s4S7S ? S SSSS+W W W)V =7V4 V V V V V  W )V85W7V88WW >`!7` `!``!``!!d 2d db9*7b-! b9-b1 1b94b1 5b98d9)c%"d$c%%dd  5m  mmn</5n0$ n<0n44n<7n48n<<qq*q5q< qq qq qq As##t2tt{5$C2{# {5#{' '{5*{' +{55||r fts_querycP|}||dkrgS|d|d} }g} tjd|D]j} | dd} | r| dvrU| | k| sgSd g} || g}| D]b}|d d d d dd}| d|d |d gdzz }c|s| d|K| ddd|Dd|||K| ddd|Dd|||rK| ddd|Dd||dd| d}| dg|z|gz}| 5}| || }dddn #1swxYwYd|DS)uLIKE-scan the rows the deferred rebuild hasn't indexed yet. Only touches ids in (fts_rebuild_progress, fts_rebuild_high_water] — a range that shrinks to nothing as the backfill advances. The FTS query is degraded to per-token substring terms (AND-joined; quoted phrases kept whole), which is deliberately recall-over-precision: temporary results beat silently missing ones mid-rebuild. NrrUrTz "[^"]+"|\S+rYr[>rrrNEARzm.id > ? AND m.id <= ?rrr}rr{r|rrrrrc3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_unindexed_gap..s"1M1M!#1M1M1M1M1M1Mr+rrc3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_unindexed_gap..s"5S5Sac5S5S5S5S5S5Sr+rc3K|]}dVdSrr%rs r)r*z;SessionSearchMixin._search_unindexed_gap..s"/I/I/I/I/I/I/I/Ir+a SELECT m.id, m.session_id, m.role, substr(m.content, max(1, instr(m.content, ?) - 40), 120) AS snippet, m.content, m.timestamp, m.tool_name, s.source, s.model, s.started_at AS session_started FROM messages m JOIN sessions s ON s.id = m.session_id WHERE rzC ORDER BY m.timestamp DESC LIMIT ? c,g|]}t|Sr%rrMs r)rzz.s&&&AQ&&&r+) rer]findallrKrrNrr3rrXrYrZ)rBr#r9r:rrrrrcrbtermsraw_tokrwherertermrrr_rRs r)rz(SessionSearchMixin._search_unindexed_gaps7$((** >UaZZI%i0&/*z.)<<  G--$$**3//5577C #))++)EEE LL     I)* *- ' 'D,,tV,,44S%@@HHeTTC LL6    z3zzzlQ& &FF > LL< = = =  $ LLP1M1M}1M1M1M)M)MPPP Q Q Q MM- ( ( (  & LLVSXX5S5S?5S5S5S-S-SVVV W W W MM/ * * *  ' LLLsxx/I/I[/I/I/I'I'ILLL M M M MM+ & & &  <<&&    (f$w. ^^   8<<V,,5577D 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8&&&&&&s)JJJinclude_archivedrsourcesc | |pd r|dkrgS||||t|dz|d|d }dtt t fdtf fd tt| fd  }d |d |DS) aSearch surfaced sessions by exact/prefix/substring session id. Desktop search uses this alongside FTS message search so users can paste a session id from logs, CLI output, or another Hermes surface and jump straight to that conversation. Matching also checks ``_lineage_root_id`` for projected compression-chain tips, so an old root id still resolves to the live continuation row. rrT)rr0rr9rr/order_by_last_activeid_queryr`r c$t|dpdt|dpdg}d|D}tfd|DrdStfd|DrdSd S) Nrr_lineage_root_idc:g|]}||Sr%)r)r&rLs r)rzzKSessionSearchMixin.search_sessions_by_id..score..s%BBBEEB%++--BBBr+c3$K|] }|kV dSr$r%r&rLneedles r)r*zJSessionSearchMixin.search_sessions_by_id..score..s';;u5F?;;;;;;r+rc3BK|]}|VdSr$) startswithr9s r)r*zJSessionSearchMixin.search_sessions_by_id..score..s1DD5##F++DDDDDDr+r)r-r[r)r`ids normalizedr:s r)scorez7SessionSearchMixin.search_sessions_by_id..scoresswwt}}*++S9K1L1L1RPR-S-STCBBSBBBJ;;;; ;;;;; qDDDDDDDDD q1r+c6|d|dfS)Nrrr%)itemr@s r)z:SessionSearchMixin.search_sessions_by_id..seeDGnnd1g6r+)rKcg|]\}}|Sr%r%)r&r{r`s r)rzz.s1113111r+N) rKrlist_sessions_richrrr-rr\r4r_) rBrWr9r/rr0r candidatesrankedr:r@s @@r)search_sessions_by_idz(SessionSearchMixin.search_sessions_by_ids"+2$$&&,,.. !I,,+eai''-!%-    tCH~ #       j ! !6666   21&%.1111r+namecp |jd|ddS#tj$rYdSwxYw)z6True if an FTS5 virtual table is queryable in this DB.zSELECT 1 FROM z LIMIT 0TF)rrYr>r)rBrIs r)_fts_table_existsz$SessionSearchMixin._fts_table_existssU  J  >>>> ? ? ?4$    55  s "55c Hd}|j5|jD]w}||s |jd|d|d|dz }@#t j$r&}td||Yd}~pd}~wwxYw dddn #1swxYwY|S)uMerge fragmented FTS5 b-tree segments into one per index. FTS5 indexes grow as a series of incremental segments — one per ``INSERT`` batch driven by the message triggers. Over tens of thousands of messages these segments accumulate, which both bloats the ``*_data`` shadow tables and slows ``MATCH`` queries that must scan every segment. The special ``'optimize'`` command rewrites each index as a single merged segment. This is purely a maintenance operation — it changes neither search results nor ``snippet()`` output, only on-disk layout and query speed. It is complementary to VACUUM: ``optimize`` compacts the FTS index internally, then VACUUM returns the freed pages to the OS. Skips any FTS table that does not exist (e.g. the trigram index when disabled via ``HERMES_DISABLE_FTS_TRIGRAM`` or not yet created), so it is safe to call unconditionally. Returns the number of FTS indexes that were optimized. rrrz) VALUES('optimize')rzFTS optimize failed for %s: %sN) r _FTS_TABLESrKrrYr>rr@rA)rB optimizedrrCs r) optimize_ftszSessionSearchMixin.optimize_fts*s:* Z  '  --c22 J&&FsFFSFFFNII/NN8#s                s: B&ABB!B=BBBBBc d}|j5|jD]}||s |jd|d|d|j|dz }Y#t j$r?}|jt d||Yd}~d}~wwxYw dddn #1swxYwY|S)aNRebuild FTS5 indexes from the canonical ``messages`` table. Uses the FTS5 ``'rebuild'`` command, which rewrites the internal b-tree segments from the content rows. This is the documented recovery for a corrupt FTS index that rejects message writes while reads still succeed (issue #50502). Unlike ``optimize_fts`` (which merges existing segments), ``rebuild`` discards and recreates the index data entirely. Safe to call when FTS tables don't exist (skips them). Returns the number of FTS indexes that were rebuilt. rrrz) VALUES('rebuild')rzFTS rebuild failed for %s: %sN) rrMrKrrYrr>rrollbackr@rA)rBrebuiltrrCs r) rebuild_ftszSessionSearchMixin.rebuild_ftsQs_ Z  '  --c22 J&&EsEESEEEJ%%'''qLGG/J'')))NN7c                s: C ?A+*C +B9:5B4/C 4B99C  C C ) max_commandsr:rTc t|tst|tstd|dkrt d||j}t|tst|tstd|dkrt dd}|j5|jD]}||st|dds!|j d |d |d t|D]K}|j j }|j d |d |d |f|d z }|j j |z dkrnLd|_dddn #1swxYwY|S)uRun bounded FTS5 ``'merge'`` commands against each present index. A positive merge rank tells SQLite to stop after approximately that many output pages, so each command holds the write lock for milliseconds regardless of index size — unlike ``'optimize'``, which rewrites the whole index in one transaction (measured 9-18 s per index on a 10 GB production DB, long enough to exhaust a competing writer's entire lock-retry patience). Protocol (SQLite FTS5 §6.8-6.9): - ``usermerge`` is lowered to its minimum of 2 (persisted in the ``%_config`` shadow table, applied once per instance) so a positive merge acts on ANY level holding >= 2 segments. With the default of 4, levels below that threshold are never merged by a positive-rank command and a fragmented index cannot converge. - Up to *max_commands* merge commands run per index, stopping early on the documented no-progress signal: the delta in ``total_changes`` is < 2 (the command's own INSERT accounts for 1 change; >= 2 means real merge work happened). Each command is its own implicit transaction (the connection runs with ``isolation_level=None``), so the SQLite write lock is released between commands and competing processes can interleave writes mid-pass. Missing tables are valid schema variants (FTS variants are optional, and ``optimize_fts_storage`` legitimately drops + backfills these tables while writers keep running) and are skipped, mirroring ``optimize_fts``. Other SQLite errors propagate to the caller. Returns the number of merge commands executed. zmax_pages must be an integerrz#max_pages must be greater than zeroNzmax_commands must be an integerz&max_commands must be greater than zero_fts_usermerge_floor_appliedFrrz, rank) VALUES('usermerge', 2)z, rank) VALUES('merge', ?)rr=T)r,rr\r.r2_FTS_MERGE_COMMANDS_PER_PASSrrMrKgetattrrrYrange total_changesrV)rBr:rTexecutedrr{befores r)r<z+SessionSearchMixin._merge_fts_incrementallypsD i & & >BCC C  > > 1  EFF F Z 5 5'  --c22 t%CUKKJ&&1s11S111|,,A!Z5FJ&&LsLLSLLL" MHz/&81<<=04D -- 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5.s/B5E00E47E4)r N)rrr)r8F)NNNr8rNFN)r8TNNN)5__name__ __module__ __qualname____doc__r1 classmethodrrr-r r8rDrrrervrrrrrrrrrrr\rrrrrrr7rrV staticmethodrgrxr{rrrrrrrrrHrKrOrSr<r%r+r)rr#s HH %! jo. %S/ "   [ D D D DVHT#s(^$<VVVV80O0O0O0Od($((((TB$BBBBHVc3h(@VVVV$,d,,,,\OOOO4%$%$%$%$Nt: d    (@E--------^4!4!4!4!l'L'L'L'L'LRBSBBBBNCG ]3]3]3hS#X'7'=>?]3 ]3 c3h ]3]3]3]3F0E x x x x  x  x U38_- x  c3hx x x x z!& AAAA A d38n  AAAAFL!CL!CL!L!L!\L!\(c(d(((\( C D   \ GcGcGGG[Gcd["AAAAA\A(,$(%)!%M@M@M@M@ M@  M@  M@CyM@cM@#YM@M@M@ $tCH~& 'M@M@M@M@d$(%)!%!&,0000Cy0c 0 #Y 0  0000C)0 d38n 0000d332$(%)!%!&,0{{{Cy{c { #Y {  {{{{C){ d38n {{{{D"'-1/3+/G'G'G'G'G'  G'  S * G'"$s),G'd3i(G' d38n G'G'G'G'X!%!%)32323232 32  32 c 32c32 d38n 32323232j c d    %c%%%%NS@@DEEEE/7}E EEEEEEr+r)r`loggingr!rr]r>rtypingrrrrrrr agent.skill_commandsr hermes_state_commonr r r rrrr getLoggerr@rr%r+r)rhs9  IIIIIIIIIIIIIIIIII::::::  > * *R"R"R"R"R"R"R"R"R"R"r+