SQLite 3.37.0 Test Failures on Big Endian Architectures Due to UTF-16 Encoding Issues

Issue Overview: Test Failures on Big Endian Architectures with UTF-16 Encoding

The core issue revolves around test failures observed in SQLite version 3.37.0 when running on big endian architectures, specifically PowerPC, PowerPC64, and S/390x. The failure manifests in the test suite, where the expected output of a UTF-16 encoded string does not match the actual output. The discrepancy is evident in the following test output:

[ 437s] ! windowB-2.0 expected: [{} 1 蕕郐䔓硑ᇍ䫎 1]
[ 437s] ! windowB-2.0 got:   [{} 1 喅킐ፅ典촑칊 1]

The issue is isolated to big endian architectures and does not occur on little endian systems, such as PPC64LE. This strongly suggests that the problem is related to how SQLite handles UTF-16 encoding on different endian architectures. The test in question is part of the windowC.test file, which was derived from windowB.test. The test was introduced in a specific commit that added a call to sqlite3_value_text(), a function that extracts UTF-16 strings in the native byte-order of the host machine. The test also includes a PRAGMA encoding=UTF16; directive, which sets the database encoding to UTF-16.

The failure occurs because the PRAGMA encoding=UTF16; directive does not explicitly specify whether the encoding should be big endian (UTF-16BE) or little endian (UTF-16LE). On big endian systems, the default behavior of PRAGMA encoding=UTF16; results in the extraction of UTF-16 strings in big endian format, which leads to the observed mismatch in the test output. This is further corroborated by the fact that on little endian systems, the expected result is obtained when using PRAGMA encoding=UTF16; or PRAGMA encoding=UTF16LE;, while PRAGMA encoding=UTF16BE; produces the same incorrect result as seen on big endian systems.

Possible Causes: UTF-16 Encoding Mismatch and Endianness Handling

The root cause of the issue lies in the handling of UTF-16 encoding and the implicit assumptions about endianness in SQLite’s implementation. The following factors contribute to the problem:

1. Implicit Endianness in PRAGMA encoding=UTF16;: The PRAGMA encoding=UTF16; directive does not explicitly specify whether the encoding should be big endian or little endian. Instead, it relies on the native byte-order of the host machine. On big endian systems, this results in UTF-16BE encoding, while on little endian systems, it results in UTF-16LE encoding. This implicit behavior leads to inconsistencies when the same test is run on different architectures.

2. Incorrect Test Prefix in windowC.test: The windowC.test file was created by copying parts of windowB.test, but it still contains the line set testprefix windowB. This incorrect test prefix could lead to confusion and misalignment in test execution, although it is not directly responsible for the encoding issue.

3. Function sqlite3_value_text() Behavior: The sqlite3_value_text() function extracts UTF-16 strings in the native byte-order of the host machine. This behavior is documented, but the test does not account for the possibility of different endianness across architectures. The function sqlite3_value_text16be() and sqlite3_value_text16le() provide explicit control over endianness, but they are not used in the test.

4. Lack of Explicit Encoding Specification in Tests: The test does not explicitly specify the endianness of the UTF-16 encoding, leading to different results on big endian and little endian systems. This lack of explicit specification is a critical oversight, as it assumes that the native byte-order of the host machine will always produce the expected result.

5. Architecture-Specific Behavior: The issue is specific to big endian architectures, which are less common than little endian architectures. This rarity may have contributed to the oversight, as the problem might not have been detected during initial testing on more common architectures.

Troubleshooting Steps, Solutions & Fixes: Addressing UTF-16 Encoding and Endianness

To resolve the issue, the following steps and solutions can be implemented:

1. Explicitly Specify UTF-16 Endianness in Tests: The most straightforward solution is to explicitly specify the endianness of the UTF-16 encoding in the test. Instead of using PRAGMA encoding=UTF16;, the test should use PRAGMA encoding=UTF16LE; or PRAGMA encoding=UTF16BE; depending on the desired behavior. This ensures consistent results across different architectures. For example, changing the test to use PRAGMA encoding=UTF16LE; would produce the expected result on both big endian and little endian systems.

2. Correct the Test Prefix in windowC.test: The incorrect test prefix in windowC.test should be corrected to avoid confusion and ensure proper test execution. The line set testprefix windowB should be changed to set testprefix windowC. This change does not directly address the encoding issue but ensures that the test is correctly identified and executed.

3. Use Explicit Endianness Functions: Instead of relying on sqlite3_value_text(), which uses the native byte-order of the host machine, the test should use sqlite3_value_text16be() or sqlite3_value_text16le() to explicitly control the endianness of the extracted UTF-16 strings. This provides greater control and ensures consistent behavior across different architectures.

4. Update Documentation and Test Guidelines: The SQLite documentation should be updated to highlight the importance of explicitly specifying endianness when working with UTF-16 encoding. Additionally, test guidelines should be updated to require explicit endianness specification in tests that involve UTF-16 encoding. This will help prevent similar issues in the future.

5. Implement Architecture-Specific Testing: To catch issues related to endianness and other architecture-specific behaviors, SQLite should implement testing on a wider range of architectures, including big endian systems. This will help identify and address issues that may not be apparent on more common architectures.

6. Apply the Fixes from the Commits: The fixes applied in the following commits should be reviewed and incorporated into the codebase:

   • Commit fb43456324c26879767b08febf1b5a2b46a289f25398a3872f81d845afd5d84e
   • Commit adf3a1e6f7575964e467f6813ff980e802cf5a37aaa9e1736af702c493f276b1

These commits address the issue by ensuring that the test explicitly specifies the endianness of the UTF-16 encoding and corrects the test prefix in windowC.test.

7. Validate Fixes on Big Endian Architectures: After applying the fixes, the test suite should be validated on big endian architectures to ensure that the issue has been resolved. This validation should include running the test suite on PowerPC, PowerPC64, and S/390x systems to confirm that the expected results are obtained.

By following these steps and implementing the suggested solutions, the issue of test failures on big endian architectures due to UTF-16 encoding mismatches can be effectively resolved. This will ensure consistent behavior across different architectures and improve the robustness of SQLite’s test suite.