Custom Synonyms Tokenizer Fails During Query Tokenization in SQLite FTS5

Issue Overview: Custom Synonyms Tokenizer Works for Document Tokenization but Fails During Query Tokenization

The core issue revolves around the implementation of a custom synonyms tokenizer for SQLite’s Full-Text Search (FTS5) module. The tokenizer is designed to expand terms into their synonyms during both document indexing and query processing. While the tokenizer appears to function correctly during document tokenization—where it successfully adds synonyms to the FTS index—it fails during query tokenization, resulting in runtime errors and non-zero return codes (rc).

The tokenizer is built to mimic the behavior of the Porter tokenizer, which acts as a filter on top of the Unicode61 tokenizer. The custom tokenizer uses a lookup_synonyms function to generate a list of synonyms for a given term, separated by the | character. During document tokenization, the tokenizer processes terms and their synonyms, adding them to the FTS index. However, when attempting to tokenize a query, the tokenizer encounters errors, specifically when calling the xToken function. The error manifests as a runtime error with an unknown error code (224), and the return code (rc) from xToken is consistently negative (-1625625888).

The issue is further complicated by the fact that the tokenizer does not explicitly distinguish between FTS5_TOKENIZE_DOCUMENT and FTS5_TOKENIZE_QUERY modes. This lack of distinction suggests that the tokenizer may not be handling query-specific tokenization requirements correctly, leading to the observed failures.

Possible Causes: Misalignment Between Document and Query Tokenization Logic

The failure of the custom synonyms tokenizer during query tokenization can be attributed to several potential causes, each rooted in the nuances of how FTS5 handles document and query tokenization differently.

1. Incorrect Handling of Query Tokenization Flags: The tokenizer does not explicitly check for the FTS5_TOKENIZE_QUERY flag, which is crucial for distinguishing between document and query tokenization. Query tokenization often requires different handling, such as ensuring that the tokenizer processes terms in a way that aligns with the FTS5 query syntax. Without this distinction, the tokenizer may attempt to apply document-specific logic to queries, leading to errors.

2. Improper Return Code Handling in xToken: The negative return code (-1625625888) from xToken suggests that the tokenizer is encountering an internal error during query tokenization. This could be due to the tokenizer failing to properly handle the input parameters or the tokenization process itself. For example, the tokenizer might be passing incorrect or malformed data to xToken, causing it to fail.

3. Misalignment Between Tokenizer and FTS5 Query Parser: The FTS5 query parser expects tokens to be generated in a specific format and order. If the custom tokenizer deviates from this expectation—such as by generating tokens in an unexpected sequence or format—it can cause the query parser to fail. This misalignment is particularly problematic when dealing with synonyms, as the parser may not correctly interpret the expanded terms.

4. Version-Specific Behavior in SQLite: The tokenizer is based on SQLite version 3.40.0, which may have differences in FTS5 behavior compared to newer versions. While the user has indicated that this version suits their needs, it is possible that the observed issues are exacerbated by version-specific quirks or bugs that have since been addressed in later releases.

5. Incorrect Implementation of fts5SynonymsCb: The callback function fts5SynonymsCb is responsible for processing synonyms during tokenization. If this function is not correctly implemented—such as by failing to properly handle the FTS5_TOKENIZE_QUERY mode or by incorrectly passing data to xToken—it can lead to the observed errors. Additionally, the function may not be correctly handling the case where no synonyms are found for a given term, leading to unexpected behavior during query tokenization.

Troubleshooting Steps, Solutions & Fixes: Addressing Query Tokenization Failures in the Custom Synonyms Tokenizer

To resolve the issue, the following steps should be taken to ensure that the custom synonyms tokenizer correctly handles both document and query tokenization. These steps involve modifying the tokenizer implementation to explicitly handle query tokenization, improving error handling, and ensuring compatibility with the FTS5 query parser.

1. Explicitly Handle Query Tokenization Flags: Modify the tokenizer to explicitly check for the FTS5_TOKENIZE_QUERY flag and implement query-specific tokenization logic. This involves distinguishing between document and query tokenization modes and ensuring that the tokenizer processes terms appropriately in each mode. For example, during query tokenization, the tokenizer should ensure that synonyms are generated in a way that aligns with the FTS5 query syntax.

2. Improve Error Handling in xToken: Review the implementation of xToken to ensure that it correctly handles input parameters and returns appropriate error codes. This includes validating the input data before passing it to xToken and ensuring that the tokenizer does not attempt to process malformed or invalid data. Additionally, the tokenizer should handle cases where xToken returns an error code by logging the error and gracefully terminating the tokenization process.

3. Align Tokenizer Output with FTS5 Query Parser Expectations: Ensure that the tokenizer generates tokens in a format and sequence that aligns with the expectations of the FTS5 query parser. This may involve modifying the tokenizer to generate tokens in a specific order or format, particularly when dealing with synonyms. For example, the tokenizer should ensure that synonyms are generated as separate tokens and that they are correctly interpreted by the query parser.

4. Test with a Newer Version of SQLite: While the user has indicated that SQLite version 3.40.0 suits their needs, it is worth testing the tokenizer with a newer version of SQLite to determine if the observed issues are related to version-specific behavior. If the tokenizer works correctly with a newer version, it may be necessary to update the implementation to address any version-specific quirks or bugs.

5. Review and Refactor fts5SynonymsCb: Review the implementation of fts5SynonymsCb to ensure that it correctly handles both document and query tokenization. This includes ensuring that the function properly processes synonyms and correctly passes data to xToken. Additionally, the function should handle cases where no synonyms are found for a given term by returning an appropriate error code or by skipping the synonym generation process.

6. Add Debugging and Logging: Enhance the tokenizer with additional debugging and logging to provide more detailed information about the tokenization process. This includes logging the input parameters, the generated tokens, and any error codes returned by xToken. This information can be invaluable for diagnosing issues and understanding the behavior of the tokenizer during both document and query tokenization.

7. Validate Tokenizer Output: Implement validation checks to ensure that the tokenizer generates valid tokens that can be correctly interpreted by the FTS5 query parser. This includes checking the length and format of generated tokens and ensuring that they do not contain invalid characters or sequences. If invalid tokens are detected, the tokenizer should log an error and terminate the tokenization process.

8. Consult SQLite Documentation and Community: Review the SQLite documentation and consult the SQLite community to ensure that the tokenizer implementation aligns with best practices and recommended approaches. This includes reviewing the FTS5 documentation to understand the expected behavior of custom tokenizers and seeking advice from other developers who have implemented similar tokenizers.

By following these steps, the custom synonyms tokenizer can be modified to correctly handle both document and query tokenization, resolving the runtime errors and ensuring that the tokenizer functions as intended. This will enable the tokenizer to successfully expand terms into their synonyms during both indexing and query processing, providing a robust and reliable solution for synonym-based full-text search in SQLite.