Base85 Encoding Discrepancy Between SQLite and Online Tools

Understanding Base85 Encoding Variations in SQLite and Third-Party Tools

The core issue revolves around differing outputs produced by SQLite’s base85() function compared to online Base85 encoding tools when processing the same input data. A user observed that encoding the blob "test" using SQLite’s base85() yields KHkS=, while an online encoder returns FCfN8. The base64() function, by contrast, produces consistent results (dGVzdA==) that align with expectations. This discrepancy highlights critical differences in how Base85 encoding is implemented across systems, rooted in historical and technical design choices.

Root Causes of Divergent Base85 Encoding Results

1. Lack of Standardization in Base85 Character Sets

Base85 encoding is not governed by a universal standard, unlike Base64, which adheres to strict RFC guidelines. The absence of standardization has led to multiple variants of Base85, each defining its own set of printable ASCII characters for encoding binary data. SQLite’s implementation uses a character subset chosen by its contributor, Larry Brasfield, prioritizing exclusion of characters that were problematic in early computing contexts (e.g., quotes, control characters). Online tools such as RFC Tools’ encoder follow alternative conventions, such as RFC 1924, which selects a different character set. This fundamental mismatch in character mapping directly causes divergent encoded outputs.

2. Padding and Block Size Handling Differences

Base85 operates on 4-byte input blocks, converting them to 5-byte encoded strings. When input data length is not a multiple of 4 bytes, padding rules vary between implementations. SQLite’s base85() appends a = character to indicate padding, similar to Base64, whereas other implementations might omit padding or use alternative termination markers. The blob "test" is 4 bytes long, which aligns perfectly with the block size, eliminating padding as a factor here. However, inconsistent handling of edge cases (e.g., partial blocks) in other scenarios can amplify discrepancies.

3. Endianness and Byte Order Conventions

The order in which bytes are processed during encoding—big-endian vs. little-endian—can alter the final output. SQLite’s base85() treats input bytes as a single 32-bit integer in big-endian format, dividing the integer into five 85-based digits. Other implementations may reverse the byte order or use arithmetic partitioning methods that prioritize different digit sequences, leading to mismatched encoded strings even with identical input.

Resolving Base85 Encoding Mismatches: Strategies and Workarounds

1. Validate Implementation-Specific Conventions

Begin by reviewing the documentation or source code of the Base85 encoder in question. For SQLite, the ext/misc/base85.c source file explicitly defines its character set as 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!#$%&()*+-;<=>?@^_{|}~, omitting quotes (‘",) and space/backslash characters. Compare this with the online tool’s character set (e.g., RFC 1924 uses 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!#$%&()*+-;<=>?@^_{|}~ but includes " and excludes others). Tools that include " or \ will produce different outputs for the same input.

2. Replicate the Encoding Process Manually

To debug the discrepancy, manually encode "test" using both SQLite’s and the online tool’s conventions:

• Step 1: Convert "test" to hexadecimal bytes.
  t = 0x74, e = 0x65, s = 0x73, t = 0x74 → 0x74657374.
• Step 2: Treat the 4-byte sequence as a 32-bit integer.
  0x74657374 = 1,952,928,628 in decimal.
• Step 3: Divide the integer into five Base85 digits.
  • SQLite:
    1,952,928,628 ÷ 85^4 = 1,952,928,628 ÷ 52,200,625 ≈ 37 (digit 1)
    Remainder: 1,952,928,628 – (37 * 52,200,625) = 1,952,928,628 – 1,931,423,125 = 21,505,503
    21,505,503 ÷ 85^3 = 21,505,503 ÷ 614,125 ≈ 35 (digit 2)
    Remainder: 21,505,503 – (35 * 614,125) = 21,505,503 – 21,494,375 = 11,128
    11,128 ÷ 85^2 = 11,128 ÷ 7,225 ≈ 1 (digit 3)
    Remainder: 11,128 – 7,225 = 3,903
    3,903 ÷ 85 = 45 (digit 4)
    Remainder: 3,903 – (45 * 85) = 3,903 – 3,825 = 78 (digit 5)
    Digits: [37, 35, 1, 45, 78]
  • Map digits to SQLite’s character set:
    37 → K, 35 → H, 1 → k, 45 → S, 78 → ~ (but SQLite appends = here).
    Result: KHkS= (Note: The discrepancy in the final character arises from SQLite’s padding rule.)
  • RFC 1924 Example:
    Using a different digit-to-character mapping (e.g., including "), the same digits would resolve to FCfN8.

3. Adopt Cross-Platform Compatibility Measures

If interoperability with external tools is required:

• Option A: Use Base64 Instead
  Base64 is standardized (RFC 4648), ensuring consistent results across implementations. For blobs like "test", base64() will reliably produce dGVzdA==.
• Option B: Implement a Custom Base85 Variant
  Modify SQLite’s base85.c to align with the target character set and padding rules. For example, replacing the character array with RFC 1924’s set and adjusting the encoding logic.
• Option C: Preprocess/Postprocess Data
  Convert SQLite’s Base85 output to match third-party tools using string substitution. For instance, replace KHkS= characters with FCfN8 via regex or lookup tables.

4. Consult Documentation and Community Resources

SQLite’s base85() function is documented in its source code header, clarifying its non-standard approach. Developers encountering mismatches should verify whether their tools adhere to Adobe’s Ascii85, RFC 1924, or other variants, then adjust expectations or workflows accordingly.

By addressing the root causes—character set selection, padding conventions, and byte ordering—developers can resolve Base85 encoding discrepancies or opt for more standardized encoding methods when cross-platform consistency is critical.