Skip to content

How Index Search Works

This section is for those interested in technical details. It is not required for understanding operations or settings.

Libraries and Versions

Library Version Purpose
Lucene.Net 4.8.0-beta00017 Full-text search engine core
Lucene.Net.Analysis.Kuromoji 4.8.0-beta00017 Japanese morphological analysis (tokenization)
Lucene.Net.QueryParser 4.8.0-beta00017 Query string parsing
sqlite-net-pcl 1.9.172 SQLite async database access
SQLitePCLRaw.bundle_green 2.1.11 SQLite native bindings
.NET 8 / WPF -- Application framework
CommunityToolkit.Mvvm 8.2.2 MVVM framework

Index Physical Structure

Indexes are stored in the index/ directory inside the data folder in the app execution folder. Uses Lucene's FSDirectory (file system-based) and opens in CREATE_OR_APPEND mode.

Directory file structure:

File Description
segments_N Lucene segment manifest (N is the generation number)
_N.cfs Compound File Storage (segment data body)
write.lock Lucene writer exclusive lock file
indexed_roots.json Persisted data for indexed roots
last_full_rebuild.txt ISO8601 timestamp of the last full rebuild
zenith.db SQLite database (history, search history, usage statistics, etc.)

indexed_roots.json format:

{
  "C:\\Users\\example\\Documents": "2026-03-07T10:30:00|locked|5428"
}

The value is pipe-delimited with 3 elements: ISO8601 timestamp (last index date), locked flag (only when archive-locked), and document count snapshot.

Document Schema (Lucene Field Definitions)

Each file/folder is registered as a single Lucene document. The field structure is as follows.

Field Name Type Tokenized Stored Purpose
path StringField No Yes File/folder full path. Used as the UpdateDocument Term for unique identification. Also used for PrefixQuery-based path scope filtering
name TextField Yes (JapaneseAnalyzer) Yes File/folder name. Main search target. Tokenized via Kuromoji morphological analysis
name_raw StringField No Yes Lowercased copy of the file name (not tokenized). Fallback for WildcardQuery partial matching
size Int64Field -- Yes File size in bytes. 0 for folders. Used in NumericRangeQuery for size filtering
modified StringField No Yes Last modified date. Converted to SECOND precision string via DateTools.DateToString. Used in TermRangeQuery for date filtering
is_dir Int32Field -- Yes Directory flag (0=file, 1=folder)

Japanese Tokenization

The name field uses JapaneseAnalyzer (Kuromoji morphological analysis engine) for searching. This splits Japanese text into meaningful word units, enabling natural searches like finding "3rd Meeting Minutes.docx" when searching for "meeting minutes."

Tokenization flow:

  1. Call JapaneseAnalyzer's GetTokenStream("name", keyword)
  2. Get ICharTermAttribute and read tokens sequentially with IncrementToken()
  3. Build a PhraseQuery (for multiple tokens) or TermQuery (for a single token) from the resulting token sequence

This processing is executed in the BuildQueryFromTokenized() method as a fallback when QueryParser.Parse() throws a ParseException. Even when queries containing wildcard characters (*, ?) or boolean operators cause parse errors, the search automatically falls back to tokenization-based search.

Query Construction Pipeline

This explains the full process from search keyword to Lucene query construction.

Step 1: Query Parse

Parses the keyword with QueryParser (field: name, analyzer: JapaneseAnalyzer). Initial settings:

  • DefaultOperator = Operator.AND (only results containing all keywords)
  • AllowLeadingWildcard = true (allows leading wildcards like *report)

Step 2: Parse Error Fallback

If a ParseException occurs, BuildQueryFromTokenized() morphologically analyzes the keyword and builds a token-based query (see 4-8-4).

Step 3: Hybrid Query Construction

In addition to the parse result, a WildcardQuery (*keyword*) on the name_raw field is combined as a SHOULD condition.

BooleanQuery {
  SHOULD: parsedQuery        <- name field (morphologically analyzed)
  SHOULD: WildcardQuery      <- name_raw field (partial match)
}

This ensures partial matches for alphanumeric characters and katakana that morphological analysis cannot split.

Step 4: Path Scope Application

  • Normal search: The current folder path is added as PrefixQuery("path", "C:\\current\\path\\") with MUST condition, limiting to that folder's descendants
  • Index search: Multiple index roots from GetScopePathsForSearch() are combined as PrefixQuery SHOULD clauses in a BooleanQuery with MUST condition

Step 5: Filter Application (ApplySearchFilter)

When search filters are set, the following queries are added with MUST condition.

  • Size filter: NumericRangeQuery.NewInt64Range("size", min, max, true, true)
  • Date filter: TermRangeQuery("modified", minStr, maxStr, true, true) -- dates are converted to SECOND precision strings via DateTools.DateToString

Step 6: AND to OR Fallback

When AND search returns 0 results and the keyword contains spaces, DefaultOperator is switched to Operator.OR for automatic re-search. Even if "meeting report" yields no AND hits, files containing "meeting" or "report" will be shown.

Result limits and sorting:

  • Normal search: Maximum 500 results
  • Verification (no keyword): Maximum 1,000 results
  • CSV export: Limit specifiable via maxResultsOverride parameter
  • Sort order: Date modified descending (SortField("modified", SortFieldType.STRING, true))

Examples:

Input Constructed Query Description
minutes name:minutes OR name_raw:*minutes* Morphological analysis + partial match OR
meeting materials (name:meeting AND name:materials) OR name_raw:*meeting materials* AND search, falls back to OR if 0 results
*.xlsx (in folder C:\Work) name_raw:*.xlsx* AND path:C:\Work\* Wildcard + path scope
Size >= 1 MB + this month Main query AND size:[1048576 TO *] AND modified:[20260301... TO 20260307...] Lucene-level filter application
Item Normal Search Index Search
Scope Single current folder (PrefixQuery limited) All registered roots (multiple PrefixQuery OR combination)
Data Source Lucene index (same engine) Lucene index (same engine)
Filter Method Size/Date: Lucene query level Size/Date: Lucene query level
Extension Filter UI level (ICollectionView filter, 11 categories) UI level (ICollectionView filter, 11 categories)
Max Results 500 500

Both normal and index search use the same Lucene engine, but differ in scope breadth. Normal search is limited to a single folder's descendants with rootPath = CurrentPath, while index search spans all registered roots.

Indexing Pipeline

The processing flow from folder registration to index completion.

  1. Registration: Add folder to _pendingRoots queue
  2. Exclusive control: SemaphoreSlim(1,1) limits to 1 concurrent task. After acquisition, move to _inProgressRoots
  3. Box Drive warm-up: For Box Drive paths, WarmUpBoxDirectoryAsync() recursively enumerates all directories beforehand, promoting stub file materialization and preventing "access denied" errors during the main scan
  4. Tree traversal: Stack-based non-recursive depth-first traversal of the folder tree
  5. Enumeration options: EnumerationOptions with IgnoreInaccessible = true, AttributesToSkip = FileAttributes.System | FileAttributes.Temporary
  6. Document registration: UpdateDocument (upsert) adds/updates files/folders as Lucene documents
  7. Commit: IndexWriter.Commit() at regular intervals. Commit frequency varies by drive type:
  8. Local drive: Every 500 items
  9. Network drive: Every 100 items
  10. Box Drive: Every 50 items
  11. Throttling: Wait times inserted after batch processing to control CPU, disk, and network load:
  12. Local: 15ms (50ms in power-saving mode)
  13. Network: 50ms (100ms in power-saving mode)
  14. Box Drive: 200ms (+150ms in network low-priority mode, +50ms in power-saving mode, max 400ms)
  15. Thread priority: During scanning, Thread.CurrentThread.Priority = ThreadPriority.BelowNormal is set and restored on completion
  16. Completion: Added to _indexedRoots and persisted to indexed_roots.json. Timestamp and document count are recorded

Exclusions (automatically skipped):

The following folders and files are automatically excluded from indexing.

  • Folders: .git, .svn, .hg, node_modules, bower_components, .vs, obj, bin, .nuget, __pycache__, .mypy_cache, venv, .venv, .next, .nuxt, .gradle, $Recycle.Bin, System Volume Information, Recovery, PerfLogs, all folders starting with $
  • Files: desktop.ini, Thumbs.db, .DS_Store, NTUSER.DAT, Office temporary files starting with ~$, extensions .tmp/.temp/.bak/.swp/.swo, files without extensions

Concurrency Control and Thread Management

  • SemaphoreSlim(1, 1): Limits index creation tasks to 1 concurrent task. Ensures stable status bar progress display and reduces server load
  • lock(_lockObj): Protects concurrent access to IndexWriter and collections (_indexedRoots, _inProgressRoots, etc.)
  • Thread.Priority = BelowNormal: Lowers thread priority during scanning to maintain UI responsiveness
  • CancellationToken chaining: CancellationTokenSource.CreateLinkedTokenSource combines the global cancellation token with user-operation cancellation tokens, allowing cancellation from either source
  • CpuIdleService integration: When IdleOnlyExecution is enabled in Interval mode, index execution waits until CPU usage drops below the threshold (default 20%)
  • Fire-and-forget exception handling: Auto mode incremental update tasks use ContinueWith(OnlyOnFaulted) to observe exceptions and prevent UnobservedTaskException

SQLite Database (zenith.db)

Various app history and statistical data is stored in data/index/zenith.db (SQLite).

Table Name Purpose Key Columns
HistoryRecord Folder browse history Path (PK), LastAccessed, SourceType (Local/Server/Box/SPO), AccessCount
SearchHistoryRecord Search history Key (PK), Keyword, IsIndexSearch, IsGrepSearch, LastSearched, PresetName, MinSizeText, MaxSizeText, StartDateText, EndDateText
RenameHistory Rename history Name, LastUsed
CustomRenameButton Custom rename button definitions User-defined rename templates
UsageRecord License usage records Id (PK, AutoIncrement), FeatureKey (indexed), UsedAt (ISO8601)
ActionStat Usage statistics ActionKey, Count, LastUsedAt

The SearchHistoryRecord Key is a composite key of keyword + "\u0001" + (isIndexSearch ? "1" : "0"), distinguishing between normal and index search for the same keyword. Filter condition columns (PresetName, size/date text) were added in V2 migration.

FileSystemWatcher Integration (Folder Monitoring)

Each tab monitors the current folder with FileSystemWatcher, reflecting external file changes in real time.

Monitoring settings:

  • NotifyFilter: FileName, DirectoryName, LastWrite, Size
  • InternalBufferSize: 64KB (65,536 bytes)
  • Monitored events: Created, Deleted, Changed, Renamed

Index integration in Auto mode:

When the index update mode is Auto, indexes are immediately updated in response to file change events.

  • Created / Changed -> AddFileToIndex(fullPath)
  • Deleted -> RemoveFileFromIndex(fullPath)
  • Renamed -> RemoveFileFromIndex(oldPath) + AddFileToIndex(newPath)

In Interval / Manual modes, indexes are updated only by scheduled intervals or manual triggers. The Interval schedule records the last run in index/last_interval_update.txt and, at startup, counts the first wait from it (if the interval has passed, at least 2 minutes).

Reflecting the app's own moves and copies:

In every mode, when a move, copy, delete or rename finishes, IndexSync puts the update on a single queue and IndexService deletes the documents for places that are gone, including everything under them, and adds the new places, including everything under them (only places under a registered folder that is not locked). The queue keeps a batch rename's "A → temporary name → B" in order. After a transfer, folder sizes in the list are fetched again. A folder with no documents in the index is treated as unknown (blank), not 0.

Changes seen by the watcher on the open folder are reflected in Auto and Interval modes (not in Manual; a folder's Changed is skipped). For watcher changes and renames, commits are batched over 1 second.

Automatic reconnection on error:

When monitoring errors occur, automatic reconnection is attempted with exponential backoff.

  • Wait times: 1s -> 2s -> 4s -> 8s -> ... -> max 30s
  • Maximum retries: 10
  • After all retries fail, monitoring stops and is supplemented by refresh when the tab becomes active

Idle Execution (CPU load avoidance)

To prevent indexing from interfering with user operations or other apps, load control runs only when the system is idle.

  • Setting: Control Deck → Index → "Update index in power-saving mode" (WindowSettings.IdleOnlyExecution) + "CPU usage threshold" (IdleCpuThreshold, default 30%).
  • CPU monitoring: CpuIdleService polls PerformanceCounter("Processor", "% Processor Time", "_Total") at regular intervals and holds a moving average.
  • Decision flow:
  • Before each batch, the indexer checks CpuIdleService.IsIdle.
  • If CpuIdleService.IsIdle == false (above threshold), it backs off exponentially and re-evaluates.
  • Even during back-off waits, user-initiated search requests are not blocked (search uses a separate queue).
  • Network drive light processing: When NetworkDriveSlowIndex is ON, additional throttling is applied to UNC / mapped drives to protect LAN / cloud bandwidth.
  • Full rebuild minimum interval: FullRebuildMinInterval (6h / 12h / 24h) causes manual full-rebuild requests within the interval to be converted to incremental updates.

This mechanism also evaluates idleness immediately after startup, so it does not contribute to startup freezes (startup-time indexing is not part of StartupInitTask; it runs on a deferred queue after IndexService itself initializes).