FTS5 Query Normalizer and Builder (store/fts-query.ts)

Parses, normalizes, and builds SQLite FTS5 full-text search queries, removing stop words and handling operators. Reuse when implementing robust search features.

// Source: https://github.com/chandra447/pi-hermes-memory | License: MIT | Copyright 2025 Chandra Teja
// Provided via justclaw.it.com; keep this notice with the code.
const FTS5_OPERATOR_PATTERN = /\b(OR|AND|NOT|NEAR)\b/;
const FTS5_TOKEN_PATTERN = /"([^"]*)"|(\S+)/g;
const NATURAL_LANGUAGE_CONNECTORS = new Set(['and', 'or', 'not', 'near']);

// Common English stop words that add noise to FTS5 searches. They are silently
// dropped from natural-language queries so common filler words cannot dominate
// the FTS5 ranking or produce misleading "AND"-style misses. The list is
// deliberately conservative: only high-frequency function words are dropped.
// Words that frequently carry the actual search intent (can, need, off, like,
// get) stay, so "can the app start" still searches for "can".


... (truncated -- full source via MCP)