Handling FTS5 xTokenize Byte Offset Validation and Error Scenarios in Custom Tokenizers

Understanding FTS5 xTokenize Byte Offset Vulnerabilities and Undefined Behavior

1. Core Issue: Unvalidated Byte Offsets in FTS5 Tokenizers Leading to Memory Safety Risks

The xTokenize function in SQLite’s FTS5 extension is responsible for breaking input text into tokens and reporting their byte offsets within the original text. These offsets are critical for features like snippet, highlight, and offsets, which rely on accurate positional data to generate contextually relevant results. However, the FTS5 engine does not enforce validation of the byte offsets returned by custom tokenizers. This creates scenarios where invalid offsets (e.g., negative values, offsets exceeding text length, or non-monotonic ranges) can propagate through FTS5’s internal logic, leading to undefined behavior such as out-of-bounds memory access, segmentation faults, or data corruption.

For example, the fts5SentenceFinderCb function (used by the snippet feature) directly uses the reported offsets to extract text fragments. If a tokenizer erroneously reports an end offset larger than the input text’s length, the function will attempt to read memory beyond the text buffer. Similarly, overlapping or regressive offsets (where a token’s start is before the previous token’s end) can disrupt the logic of phrase matching or sentence boundary detection. The absence of explicit documentation on these edge cases leaves developers unaware of the need to implement safeguards in custom tokenizers.

The root of the problem lies in the lack of contractual guarantees between FTS5 and tokenizer implementations. Tokenizers are treated as trusted components, and FTS5 assumes they will return valid offsets. This design choice prioritizes performance over safety, as offset validation would introduce computational overhead. However, in practice, tokenizers—especially custom ones—may contain bugs or edge-case oversights that violate these assumptions.

Potential Sources of Invalid Byte Offsets in FTS5 Tokenizers

2.1 Tokenizer Logic Errors in Start/End Offset Calculation

Custom tokenizers often involve complex logic for splitting text into tokens, especially when handling Unicode, hyphenation, or language-specific rules. A miscalculation in the tokenization loop—such as incorrect pointer arithmetic, mishandling of multibyte characters, or off-by-one errors—can produce invalid offsets. For instance, a tokenizer processing UTF-8 text might miscount the bytes in a multi-byte character, causing subsequent tokens to have start offsets that are larger than the input length.

2.2 Misaligned Tokenization Contexts

Tokenizers that maintain internal state (e.g., for handling hyphenated words across buffers) might fail to reset context between input chunks. This can lead to offsets being calculated relative to an incorrect base pointer, resulting in negative values or offsets that reference prior input buffers. Stateless tokenizers are not immune either: improper handling of text segmentation (e.g., splitting “don’t” into “don” and “t” without adjusting offsets) can cause misalignment.

2.3 Undefined Behavior in Tokenizer Callbacks

The xTokenize callback interface allows tokenizers to invoke user-defined functions for each token. If these functions modify the input text buffer or alter the tokenizer’s state mid-process, the offsets generated afterward may no longer correspond to the original text. This is particularly problematic in multi-threaded environments or when tokenizers reuse buffers across calls.

2.4 FTS5’s Reliance on Tokenizer-Generated Offsets

FTS5 does not cross-check reported offsets against the actual input text length. Functions like snippet or highlight blindly use the offsets to slice the original text, assuming they are valid. This creates a chain of dependency: a single invalid offset can corrupt the output of higher-level features or crash the entire query execution process.

Mitigating Risks: Validation Strategies and Safe Tokenizer Design

3.1 Implementing Byte Offset Sanity Checks in Tokenizers

Custom tokenizers must enforce strict validation of start and end offsets before invoking the xToken callback. At minimum, the following checks should be performed:

1. Non-Negative Offsets: Ensure iStart and iEnd are ≥ 0.
2. Monotonic Progression: Verify that iStart of the current token is ≥ iEnd of the previous token.
3. Bounds Checking: Confirm that iEnd does not exceed the length of the input text.
4. Order Consistency: Validate that iStart ≤ iEnd for each token.

If any check fails, the tokenizer should return SQLITE_RANGE (or another appropriate error code) to signal an invalid offset. This prevents corrupt data from propagating into FTS5’s internal structures. For example:

if (iStart < 0 || iEnd > textLength || iStart > iEnd) {  
    return SQLITE_RANGE;  
}  

3.2 Securing FTS5 Auxiliary Functions Against Invalid Offsets

Even with a correctly implemented tokenizer, defensive programming is necessary in functions that consume offsets (e.g., snippet). Before extracting text fragments, validate the offsets against the original text length:

if (iEnd > textLength) {  
    // Truncate to textLength or handle as an error  
}  

This prevents buffer overreads. SQLite’s built-in auxiliary functions do not perform such checks, so wrapper layers (e.g., Python extensions) must add them.

3.3 Testing Tokenizers with Fuzzing and Boundary Cases

Developers should subject custom tokenizers to rigorous testing, including:

• Edge Cases: Empty strings, single-byte inputs, and maximum-length texts.
• Invalid Offsets: Artificially injecting negative offsets or out-of-bounds values to verify error handling.
• Multibyte Encodings: UTF-8/16 texts with combining characters, emojis, or right-to-left scripts.

Fuzzing tools like libFuzzer or AFL can automate this process, uncovering offset calculation bugs that manual testing might miss.

3.4 Documenting Tokenizer Contracts and Offset Guarantees

Explicitly document the expectations for custom tokenizers, including:

• Offsets must be monotonically increasing.
• iEnd must not exceed the input text’s byte length.
• Tokenizers must return SQLITE_RANGE on validation failure.

This documentation should be mirrored in both the tokenizer’s code and any higher-level APIs (e.g., Python wrapper classes) to ensure consistent error handling.

3.5 Leveraging SQLITE_RANGE for Error Propagation

When invalid offsets are detected, tokenizers should return SQLITE_RANGE to abort the current operation and surface the error to the caller. This ensures that bugs are not silently ignored. Applications should handle this error code by logging diagnostic information (e.g., the input text and failing offsets) for debugging.

3.6 Wrapping Tokenizers in Sanitization Layers

For languages like Python, where developers interact with SQLite via wrappers (e.g., sqlite3 module or APSW), add a middleware layer that validates offsets before they reach FTS5. This layer can:

• Intercept tokens and their offsets.
• Apply the same sanity checks as in Section 3.1.
• Raise exceptions or log warnings for invalid offsets.

This approach provides defense-in-depth, protecting against both tokenizer bugs and FTS5’s lack of internal checks.

3.7 Monitoring for Memory Safety Violations

Use tools like AddressSanitizer (ASan) or Valgrind to detect out-of-bounds memory accesses during testing. These tools can identify buffer overreads caused by invalid offsets, even if they do not immediately crash the application. For example, when running SQLite’s test suite with ASan:

export CFLAGS="-fsanitize=address"  
./configure  
make  

3.8 Advocating for FTS5 Enhancements

While the current FTS5 implementation does not validate offsets, the SQLite team may consider adding optional runtime checks in future versions. Developers can advocate for:

• A compile-time flag to enable offset validation in FTS5.
• Documentation explicitly stating the risks of invalid offsets.
• Built-in helper functions for offset sanity checks.

Until then, the responsibility lies with tokenizer authors and application developers to enforce correctness.

By systematically addressing byte offset validation at the tokenizer level, reinforcing auxiliary functions with bounds checks, and employing rigorous testing practices, developers can mitigate the risks posed by invalid offsets in FTS5. This approach ensures memory safety and functional correctness while maintaining the performance benefits of SQLite’s lightweight design.