FTS3 Test Failure on s390x Due to Endian-Sensitive Test Case

Understanding the FTS3 Test Case Failure on Big-Endian Architectures

The failure of the fts3corrupt4-25.6 test case on s390x systems when SQLite is compiled with --enable-fts3 reveals a critical dependency on byte order assumptions within test case logic. This discrepancy arises exclusively on big-endian architectures like s390x, while little-endian systems (x86_64, aarch64, ppc64le) execute the test successfully. The root cause lies in how the test case validates database corruption scenarios under FTS3, where the expected malformed database error (SQLITE_CORRUPT with message "database disk image is malformed") is not triggered on s390x. Instead, the test erroneously reports success (0 {}). The checkin 437849c80851da84 intended to resolve this test case instead exposed an underlying endianness sensitivity in the test’s implementation. The behavior is not indicative of a defect in SQLite’s core FTS3 functionality but rather a mismatch between the test case’s expectations and the byte order of the target architecture.

Architectural Endianness and FTS3 Corruption Detection Logic

The divergence in test outcomes stems from how the test case manipulates the database to simulate corruption and how SQLite’s FTS3 module detects such corruption. In the fts3corrupt4-25.6 test, a specific database file is altered to introduce structural inconsistencies that should force SQLite to return SQLITE_CORRUPT. However, the method of introducing this corruption inadvertently relies on byte order assumptions.

1. FTS3 Index Structure and Corruption Simulation:
   FTS3 uses inverted indices stored as B-trees, where term postings are encoded in binary formats. When the test case modifies the database to simulate corruption, it directly manipulates bytes in these structures. If the test case writes fixed byte patterns to specific offsets without considering the architecture’s byte order, the resulting "corruption" may not align with how SQLite’s FTS3 parser interprets the data. For example, a 4-byte integer written in little-endian format on x86_64 would be read incorrectly on s390x (big-endian), potentially bypassing the corruption checks.

2. Validation Logic in Test Scripts:
   The test case assumes that the corruption introduced will be detected uniformly across architectures. However, if the corruption involves multi-byte values (e.g., node sizes, page numbers), their interpretation depends on the platform’s endianness. On little-endian systems, a malformed integer might trigger an out-of-bounds read or invalid node size error, while on big-endian systems, the same byte sequence could represent a valid (but unintended) value, allowing the query to complete without error.

3. Compiler and Configuration Flags:
   The failure manifests only when --enable-fts3 is set, indicating that the test case’s corruption method is specific to FTS3’s storage format. Enabling FTS4 or FTS5 does not affect the outcome because their index structures differ, and the test case does not target those modules. The byte order sensitivity is thus confined to the FTS3-specific corruption scenario.

Resolving Endian-Sensitive Test Failures in FTS3

To diagnose and resolve the fts3corrupt4-25.6 failure on s390x, follow these steps:

1. Verify Endianness-Specific Behavior:
Confirm the test environment’s byte order using compile-time checks or runtime diagnostics. For example, execute a SQL query that returns the byte order of integer storage:

SELECT hex(CAST(1 AS BLOB));

On little-endian systems, this returns 01000000; on big-endian, 00000001. If the test case assumes little-endian byte order for corruption patterns, adjust the injected bytes to match the target architecture.

2. Audit Test Case Corruption Logic:
Review the test script’s database modification steps. Identify where fixed byte sequences are written to simulate corruption. Replace hard-coded little-endian values with architecture-agnostic methods. For instance, use SQLite’s CAST(expr AS BLOB) to generate platform-independent byte representations of integers:

-- Instead of writing X'01000000' for integer 1:
UPDATE tbl SET blob_col = CAST(1 AS BLOB) WHERE ...;

3. Apply Checkin 6216bfcb74273b78:
The fix involves updating the test case to handle endianness dynamically. Backport the changes from the checkin to the test suite:

• Modify the corruption injection code to use sqlite3_column_blob and sqlite3_bind_blob with byte-order-aware serialization.
• Replace static byte literals (e.g., X'12345678') with dynamically generated blobs that account for the host architecture’s endianness.

4. Conditional Test Execution:
If backporting the fix is impractical, disable the test case on big-endian architectures until the official 3.48.0 release includes the corrected test. Use conditional compilation or runtime checks:

#if defined(__s390x__)
  /* Skip fts3corrupt4-25.6 on s390x */
#else
  /* Run the test */
#endif

5. Validate with Forced Endianness Emulation:
Test the fix by emulating big-endian behavior on a little-endian system (or vice versa) using tools like qemu-s390x or compiler flags (e.g., -mbig-endian in GCC). Ensure the test now fails/passes as expected on both architectures.

6. Monitor Release Integration:
Track the integration of checkin 6216bfcb74273b78 into the SQLite release branch. Verify that the 3.48.0 release notes confirm the test case fix for big-endian systems.

By addressing the endianness assumptions in test case design and adopting platform-agnostic data manipulation techniques, developers can ensure consistent test behavior across architectures. This issue underscores the importance of considering byte order in low-level database tests, particularly when directly interfacing with binary storage formats.