FTS5 xPhraseFirst/Next Token Offset Ordering: Guarantees, Risks, and Resolutions
Uncertainty in FTS5 xPhraseFirst/Next Token Offset Ordering The core issue revolves around whether the token offsets returned by SQLite’s FTS5 auxiliary functions xPhraseFirst and xPhraseNext are guaranteed to be sorted in ascending order. Developers implementing ranking functions or phrase proximity analysis often rely on the order of token offsets to compute metrics like phrase density,…