FTS5 Unicode61 Tokenizer Fails to Recognize Polish “ł” and “Ł” Characters

FTS5 Unicode61 Tokenizer’s Inability to Handle Polish Diacritics

The core issue revolves around the FTS5 (Full-Text Search) Unicode61 tokenizer’s failure to correctly recognize and tokenize specific Polish characters, particularly "ł" and "Ł". This issue manifests when users attempt to index or search for text containing these characters. For example, a user might expect the term "Główna" to be found when searching for "glow*", but the tokenizer fails to map the Polish character "ł" to its base form "l". This problem is not limited to "ł" and "Ł"; other characters like "ɠ" and "ɠ" are also affected, indicating a broader issue with the tokenizer’s handling of certain Unicode characters.

The Unicode61 tokenizer is designed to handle Unicode text by normalizing characters and removing diacritics, which is essential for languages that use diacritical marks. However, the tokenizer’s current implementation does not correctly map "ł" and "Ł" to their base forms, leading to incorrect search results. This issue is particularly problematic for Polish language users, as these characters are integral to the language’s orthography.

The problem is exacerbated when users attempt to use the remove_diacritics option in FTS4 or FTS5, which is intended to strip diacritical marks from characters during tokenization. Despite enabling this option, the tokenizer still fails to correctly process "ł" and "Ł", rendering it ineffective for Polish text. This failure suggests that the issue lies deeper within the tokenizer’s handling of Unicode character mappings, specifically for characters that do not have a direct base character mapping in the Unicode standard.

Unicode Character Mapping Deficiencies in FTS5 Tokenizer

The root cause of this issue lies in the Unicode character mappings used by the FTS5 tokenizer. The Unicode standard provides mappings for characters to their base forms, which are used by tokenizers to normalize text. However, the mappings for "ł" and "Ł" are not correctly implemented in the FTS5 tokenizer. According to the UnicodeData.txt entries, the characters "ł" (U+0142) and "Ł" (U+0141) are defined as follows:

0141;LATIN CAPITAL LETTER L WITH STROKE;Lu;0;L;;;;;N;LATIN CAPITAL LETTER L SLASH;;;0142;
0142;LATIN SMALL LETTER L WITH STROKE;Ll;0;L;;;;;N;LATIN SMALL LETTER L SLASH;;0141;;0141

These entries indicate that "ł" and "Ł" are considered separate characters with no direct mapping to a base character. In contrast, other characters with diacritical marks, such as "Ŀ" (U+013F) and "ŀ" (U+0140), have explicit mappings to their base characters "L" (U+004C) and "l" (U+006C), respectively. This discrepancy in Unicode mappings is the primary reason why the FTS5 tokenizer fails to correctly process "ł" and "Ł".

The lack of a direct base character mapping for "ł" and "Ł" means that the tokenizer cannot normalize these characters to their base forms, leading to incorrect tokenization. This issue is not unique to SQLite’s FTS5 tokenizer; it is a limitation of the Unicode standard itself. However, the FTS5 tokenizer’s reliance on these mappings makes it particularly susceptible to this problem.

Additionally, the remove_diacritics option in FTS5 does not resolve this issue because it relies on the same Unicode mappings. When remove_diacritics is enabled, the tokenizer attempts to strip diacritical marks from characters, but it cannot do so for characters that lack a base character mapping. As a result, the tokenizer fails to correctly process "ł" and "Ł", leading to incorrect search results.

Custom Tokenizer Implementation and Unicode Reporting

To address this issue, users have two primary options: implementing a custom tokenizer or reporting the issue to the Unicode Consortium. Implementing a custom tokenizer is the most effective solution for users who require immediate functionality. SQLite’s FTS5 extension allows users to create custom tokenizers, which can be tailored to handle specific characters like "ł" and "Ł". A custom tokenizer can be designed to explicitly map these characters to their base forms, ensuring correct tokenization and search results.

Creating a custom tokenizer involves defining a new tokenizer class that implements the necessary logic to handle "ł" and "Ł". This class must be registered with SQLite’s FTS5 extension, after which it can be used in place of the default Unicode61 tokenizer. While this approach requires some development effort, it provides a robust solution for users who need to support Polish text in their applications.

For users who prefer not to implement a custom tokenizer, reporting the issue to the Unicode Consortium is another option. The Unicode Consortium is responsible for maintaining the Unicode standard, and they can be notified of issues through their official reporting channels. By reporting the lack of base character mappings for "ł" and "Ł", users can contribute to the long-term resolution of this issue. However, this approach does not provide an immediate solution, as changes to the Unicode standard may take time to be implemented and adopted.

In the meantime, users can work around the issue by using regular expressions to manually normalize text before indexing or searching. For example, a user could replace "ł" and "Ł" with "l" and "L" in their text before inserting it into the FTS5 index. While this approach is not ideal, it can provide a temporary solution for users who need to support Polish text.

In conclusion, the FTS5 Unicode61 tokenizer’s inability to correctly handle "ł" and "Ł" is a significant issue for Polish language users. The root cause lies in the Unicode character mappings used by the tokenizer, which do not provide a direct base character mapping for these characters. Implementing a custom tokenizer is the most effective solution, while reporting the issue to the Unicode Consortium can contribute to a long-term resolution. In the meantime, users can use regular expressions to manually normalize text, providing a temporary workaround for this issue.