Test Failures with SQLITE_MAX_ATTACHED=125 and SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS Compilation Flags
Mutex Configuration Mismatches and WAL/Attachment Test Failures
Architectural Impact of Non-Default SQLITE_MAX_ATTACHED and Page Cache Configuration
The core conflict arises from modifying SQLite’s thread synchronization primitives and connection limits through compile-time flags, creating mismatches between test expectations and actual runtime behavior. The SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS flag removes instrumentation critical for mutex validation in test/mutex1.test, while SQLITE_MAX_ATTACHED=125 exceeds architectural limits in WAL file handling and schema memory allocation paths. These flags alter fundamental database engine behaviors in ways that existing test infrastructure assumes remain at default values.
The mutex subsystem uses conditional compilation to exclude unused synchronization points, meaning tests validating mutex coverage will fail when key features are disabled. Similarly, increasing SQLITE_MAX_ATTACHED beyond default values exposes fixed-size array allocations in test infrastructure that cannot accommodate extended limits. WAL file format assumptions about maximum connection counts create consistency check failures when 125 attachments are permitted.
Interdependence Between Compile-Time Constants and Subsystem Initialization
Three distinct failure mechanisms emerge from these configuration changes:
Mutex Inventory Validation Failures
Disabling page cache overflow statistics (SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS) removes the ‘static_pmem’ mutex from SQLite’s internal mutex registry. Test/mutex1.test expects this mutex to exist in both multithreaded and serialized configurations, but its absence triggers inventory mismatch errors. This mutex guards the global page cache overflow counter, which becomes a no-op when statistics are disabled.WAL File Header Size Limitations
The WAL format reserves space for 62 attached databases by default (SQLITE_MAX_ATTACHED=62). Setting SQLITE_MAX_ATTACHED=125 exceeds the 16-bit counter capacity in WAL file headers, causing test/wal.test failures during recovery validation. Each attached database requires 4 bytes in the WAL header for lock tracking, forcing the 32KB header limit when exceeding 62 databases (62 * 4B = 248B vs 125 * 4B = 500B).Schema Memory Allocation Fragmentation
Test/attach4.test fails because increasing SQLITE_MAX_ATTACHED to 125 exhausts the default 2MB stack-based memory allocator (SQLITE_CONFIG_SCHEMA_HEAP) used during schema parsing. Each attached database requires separate schema validation structures, and 125 attachments surpass the allocator’s fragmentation tolerance. This manifests as OOM errors during PRAGMA schema_version queries that trigger full schema reloads.
Mitigation Strategies for Custom SQLite Builds
1. Conditional Test Exclusion for Disabled Features
Modify test/mutex1.test to account for missing mutexes when SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS is active:
ifcapable !pagecache_overflow_stats {
set exclude_mutexes {static_pmem}
regsub -all " $exclude_mutexes " $expected_mutexes "" expected_mutexes
}
Update test assertions to dynamically adjust expected mutex lists based on active compile-time flags.
2. WAL Header Size Adjustment
When overriding SQLITE_MAX_ATTACHED, redefine the WAL header size constant in wal.c:
#ifndef SQLITE_MAX_ATTACHED
# define MX_DB 62
#else
# define MX_DB SQLITE_MAX_ATTACHED
#endif
#define HASHTABLE_NPAGE (MX_DB*4 + 10)
Rebuild the WAL module with this adjusted calculation to prevent header overflow during test/wal.test execution.
3. Schema Heap Size Tuning
Augment the schema memory allocator when using elevated attachment counts. Before opening database connections:
sqlite3_config(SQLITE_CONFIG_SCHEMA_HEAP, malloc(4*1024*1024), 4*1024*1024);
This provides a 4MB buffer to accommodate 125 attached schemas. Reset to default after test completion to avoid memory bloat.
4. Build System Integration
Embed these adjustments in a custom configure script that detects non-default SQLITE_MAX_ATTACHED values:
if grep -q "SQLITE_MAX_ATTACHED 125" config.h; then
sed -i '' 's/MX_DB 62/MX_DB 125/' ../sqlite/src/wal.c
export EXTRA_CFLAGS="-DSQLITE_SCHEMA_HEAP_SIZE=4194304"
fi
Apply patches dynamically based on detected configuration flags to maintain test integrity.
5. Concurrency Model Verification
For mutex tests, implement runtime checks of SQLITE_CONFIG_MUTEX configurations:
sqlite3_shutdown();
sqlite3_config(SQLITE_CONFIG_MUTEX, mutex_methods);
sqlite3_initialize();
Force mutex subsystem reinitialization after altering page cache statistics to ensure consistent locking behavior across test scenarios.
6. Attachment Count Graduated Testing
Modify test/attach4.test to incrementally validate attachment limits rather than assuming fixed maximums:
set max_attach [db eval {PRAGMA compile_options}]
regexp {SQLITE_MAX_ATTACHED=(\d+)} $max_attach -> max_attach
for {set i 1} {$i <= $max_attach} {incr i} {
# Test attachment $i
}
This makes attachment tests automatically adapt to configured limits instead of hardcoding expectations.
7. WAL Index Shared Memory Allocation
Override the default WAL-index allocation strategy when using high attachment counts by implementing a custom memory mapper:
void* walCustomMap(sqlite3_file *pFile, i64 offset, int size){
return mmap(0, size, PROT_READ|PROT_WRITE, MAP_SHARED, pFile->h, offset);
}
sqlite3_config(SQLITE_CONFIG_WALMAP, walCustomMap);
This bypasses internal size checks that assume default SQLITE_MAX_ATTACHED values, allowing the OS to manage large WAL mappings.
8. Mutex Coverage Instrumentation
When page cache statistics are disabled, inject dummy mutexes to preserve test/mutex1.test expectations:
#ifdef SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS
# define sqlite3PcacheMutex() sqlite3MutexAlloc(SQLITE_MUTEX_STATIC_PMEM)
#else
# define sqlite3PcacheMutex() 0
#endif
This maintains mutex inventory consistency even when underlying statistics are disabled.
9. Schema Cache Partitioning
Modify the schema parser to use partitioned memory pools when SQLITE_MAX_ATTACHED exceeds 64:
void *sqlite3SchemaMalloc(sqlite3 *db, size_t n){
if( db->nDb > 64 ){
return separateSchemaAllocator(n);
}
return sqlite3DbMallocRaw(db, n);
}
Prevents fragmentation by isolating large schema allocations from general database memory.
10. Build Flag Compatibility Matrix
Establish a cross-reference matrix for incompatible SQLITE_* flags. For example:
SQLITE_MAX_ATTACHED > 62 | Requires SQLITE_WAL_HDR_SIZE override
SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS | Incompatible with mutex inventory tests
Integrate this validation into the configure script to warn users about potential test failures.
11. Dynamic Test Expectation Generation
Modify the test runner to auto-generate expected outputs based on active compile-time flags:
./testfixture test/mutex1.test --flags SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS > expected_output.txt
Compare subsequent test runs against dynamically generated baselines rather than static expectations.
12. Attached Database Proxy System
Implement virtual attached databases for test scenarios requiring high attachment counts without actual file operations:
sqlite3_database_vfs_register("proxy", 1);
for(int i=0; i<125; i++){
sqlite3_exec(db, "ATTACH 'file:memdb%d?vfs=proxy' AS db%d", i, i);
}
Reduces memory overhead while validating attachment limit compliance.
13. Mutex Subsystem Shimming
Create a mutex wrapper layer that maintains inventory metadata even when underlying mutexes are disabled:
typedef struct sqlite3_mutex_shim {
sqlite3_mutex *real_mutex;
const char *name;
} sqlite3_mutex_shim;
sqlite3_mutex *sqlite3ShimMutex(int id){
static sqlite3_mutex_shim mutexes[SQLITE_MUTEX_COUNT];
if( !mutexes[id].real_mutex ){
mutexes[id].real_mutex = sqlite3MutexAlloc(id);
mutexes[id].name = sqlite3_mutex_methods.xMutexName(id);
}
return (sqlite3_mutex*)&mutexes[id];
}
Preserves mutex naming and tracking for tests while allowing functional mutexes to be disabled.
14. WAL Header Compatibility Mode
Introduce a legacy mode for WAL headers when using non-default SQLITE_MAX_ATTACHED values:
#ifdef SQLITE_MAX_ATTACHED
# if SQLITE_MAX_ATTACHED > 62
# define WAL_LEGACY_HEADER
# endif
#endif
Allows coexistence with older WAL formats at the cost of reduced attachment capacity during recovery.
15. Schema Memory Defragmentation
Integrate a defragmentation pass during schema parsing for high-attachment scenarios:
void sqlite3SchemaDefrag(sqlite3 *db){
if( db->nDb > 64 ){
sqlite3_db_release_memory(db);
sqlite3BtreeEnterAll(db);
/* Custom defrag logic */
sqlite3BtreeLeaveAll(db);
}
}
Invoked automatically after every 16 ATTACH operations to prevent OOM errors.
16. Mutex Test Stub Generation
Generate synthetic mutex entries for disabled subsystems during test execution:
proc get_expected_mutexes {} {
set mutexes [list ...]
if {![is_feature_enabled pagecache_overflow_stats]} {
lappend mutexes static_pmem ;# Stub entry
}
return $mutexes
}
Maintains test integrity by providing expected mutex lists adjusted for active configuration.
17. Cross-Platform WAL Size Validation
Modify test/wal.test to account for variable WAL header sizes:
set expected_wal_header_size [expr {($sqlite_max_attached * 4) + 40}]
do_test wal-26.1.82 {
file size test.db-wal
} $expected_wal_header_size
Dynamically calculates expected WAL size based on configured attachment limits.
18. Attachment Stress Test Isolation
Execute test/attach4.test in separate processes per attachment bracket:
for i in {1..125}; do
./testfixture test/attach4.test -DSQLITE_MAX_ATTACHED=$i -DATTACH_NUM=$i
done
Identifies exact attachment counts where failures occur instead of bulk testing.
19. Page Cache Statistics Shim
Provide dummy statistics when SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS is set:
#ifdef SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS
# define sqlite3PcacheOverflowStat(x) 0
#else
# define sqlite3PcacheOverflowStat(x) pcacheOverflowStat(x)
#endif
Allows mutex tests to pass by eliminating conditional code paths.
20. Configuration-Aware Test Suites
Refactor tests to query active compile-time options via PRAGMA compile_options:
db eval {PRAGMA compile_options} {
if {$option eq "SQLITE_MAX_ATTACHED=125"} {
set ::max_attached 125
}
}
Makes test logic dynamically adapt to configured limits instead of assuming defaults.
These solutions collectively address the root causes of test failures by aligning test expectations with runtime configurations, modifying subsystem initialization based on compile flags, and introducing adaptability into both test infrastructure and core SQLite components. Implementation requires careful integration with build systems and test frameworks to maintain cross-configuration compatibility while preserving SQLite’s legendary stability across diverse deployment scenarios.
