SQLite Test Failure: Broken Pipe in fts5multiclient.test on Windows Build

Issue Overview: Broken Pipe During FTS5 Multi-Client Test Execution

The core problem manifests as a "broken pipe" error during execution of the fts5multiclient.test suite when building SQLite from source on Windows 10 using Visual Studio 2019 and IronTcl. This occurs after successful completion of foundational test phases like fuzztest, with failure specifically in the TCL-based test infrastructure during interprocess communication between testfixture.exe and TCL interpreters. The error stack trace reveals a failure in the puts_override procedure during attempted writes to channel file2a775f406e0, indicating disrupted communication between test components.

Key technical characteristics of the failure include:

• Environment: Clean SQLite source extraction (version 3.45.1) with modified build directory structure
• Build configuration: Customized Makefile.msc with TOP=.. and TCLSUFFIX=t
• Test infrastructure: IronTcl 8.6.7 configured with renamed directories and duplicated executables
• Failure context: Occurs during multi-client FTS5 virtual table tests requiring coordinated execution across multiple TCL interpreters

The error’s position in the test sequence indicates proper compilation of SQLite core functionality but reveals infrastructure-level communication breakdown. The broken pipe terminology in TCL context refers to failed writes to a closed channel between parent testfixture process and child TCL interpreters managing concurrent database connections.

Possible Causes: TCL Environment Configuration and IPC Channel Management

1. IronTcl Compatibility Issues with SQLite Test Harness

IronTcl (a .NET implementation of TCL) may implement pipes or file channels differently than standard TCL distributions. The SQLite test suite relies heavily on TCL’s interp command and channel redirection capabilities, which might exhibit divergent behavior in IronTcl regarding:

• Named pipe handling on Windows
• File channel inheritance between processes
• Synchronization of interprocess communication buffers
• Signal handling for channel closure detection

Example compatibility gaps could include:

• IronTcl’s implementation of tclsh lacking full support for auto_path variable resolution
• Differences in how child processes inherit handles in .NET runtime vs native TCL
• Variant behavior in fconfigure parameters for non-blocking I/O

2. Makefile.msc Configuration Errors Affecting TCL Integration

The modifications to Makefile.msc (line 11: TOP = .., line 921: TCLSUFFIX = t) introduce critical path resolution changes that may disrupt TCL integration:

• TOP = .. alters relative paths for:
  • TCL library inclusion (-I directives)
  • Linker references to tcllib.lib
  • Test script path resolution (..\test\*.test vs test\*.test)
• TCLSUFFIX = t modifies the expected TCL library filename pattern from tcl86t.lib to tcl86.lib (if suffix logic isn’t properly inverted)

These changes could cause:

• Incorrect linking against TCL runtime libraries
• Mismatched TCL header versions during compilation
• Test script path resolution failures
• TCL interpreter initialization errors

3. Windows-Specific File Handle Inheritance and Security Attributes

The Windows pipe implementation requires explicit handle inheritance configuration when spawning child processes. If the SQLite test harness or IronTcl doesn’t properly configure these attributes, pipe handles may not be inherited by child tclsh processes, leading to:

• Premature closure of pipe handles before child process initialization
• Access denied errors when attempting to write to pipes
• Race conditions in handle duplication between parent and child processes

Windows-specific factors to consider:

• Default security descriptors for anonymous pipes
• Handle inheritance flags in CreateProcess API calls
• CRT (C Runtime) layer file descriptor translation
• Anti-virus software intercepting pipe creation

Troubleshooting Steps, Solutions & Fixes

Phase 1: Validate TCL Environment Configuration

Step 1.1: Verify IronTcl Installation Integrity

• Confirm compat\tcl\bin contains:
  • tclsh86t.exe (original IronTcl binary)
  • tclsh.exe (copied duplicate)
  • tcl86t.dll (required runtime)
• Check environment variables:
  • PATH must include compat\tcl\bin
  • TCLLIBPATH should point to compat\tcl\lib

Step 1.2: Test TCL Interpreter Independently
Create testpipe.tcl:

set chan [open |[info nameofexecutable] r+]
puts $chan "puts 123"
flush $chan
gets $chan line
puts "Received: $line"
close $chan

Execute with:

compat\tcl\bin\tclsh.exe testpipe.tcl

Expected output: Received: 123. If this fails, IronTcl has fundamental pipe implementation issues.

Step 1.3: Switch to Official TCL Distribution
Replace IronTcl with ActiveState TCL 8.6:

1. Delete compat\tcl
2. Install ActiveState TCL to compat\tcl
3. Verify compat\tcl\bin\tclsh.exe exists
4. Update Makefile.msc line 921 to match actual library suffix (typically t for thread-enabled builds)

Step 1.4: Audit Makefile.msc TCL Directives
Ensure these lines match TCL installation:

TCLDIR = $(TOP)\compat\tcl
TCLINCDIR = $(TCLDIR)\include
TCLLIBDIR = $(TCLDIR)\lib
TCLLIB = $(TCLLIBDIR)\tcl$(TCLVERSION)$(TCLSUFFIX).lib

For ActiveState TCL 8.6, typical values would be:

• TCLVERSION = 86
• TCLSUFFIX = t

Phase 2: Diagnose Pipe Handling in Test Infrastructure

Step 2.1: Enable TCL Channel Debugging
Modify fts5multiclient.test to add channel tracing:

proc puts_override {args} {
    puts stderr "DEBUG: Writing to [lindex $args 0]"
    uplevel puts_original $args
}

Rebuild and run tests. Look for debug output preceding the broken pipe error to identify which channel is being closed prematurely.

Step 2.2: Investigate Handle Inheritance
Use Process Monitor (procmon) from Sysinternals to monitor:

• testfixture.exe process creation
• Pipe creation (CreatePipe API calls)
• Handle duplication (DupliateHandle)

Filter on:

• Process Name: testfixture.exe
• Operation: CreateFile, CreatePipe, DuplicateHandle

Look for ACCESS DENIED errors or handles closed before being inherited.

Step 2.3: Modify Pipe Security Attributes
Patch testfixture.c to explicitly set pipe security attributes:

SECURITY_ATTRIBUTES sa;
sa.nLength = sizeof(sa);
sa.bInheritHandle = TRUE;
sa.lpSecurityDescriptor = NULL;

if (!CreatePipe(&hReadPipe, &hWritePipe, &sa, 0)) {
    fprintf(stderr, "CreatePipe failed: %d\n", GetLastError());
    exit(1);
}

Rebuild testfixture.exe and retest.

Phase 3: Workarounds and Alternative Build Strategies

Step 3.1: Disable FTS5 Multi-Client Tests Temporarily
Modify veryquick.test to exclude problematic tests:

run_tests veryquick -files {
  ...
  fts5.test fts5aa.test fts5ac.test fts5af.test
  # fts5multiclient.test
  ...
}

This allows completing other tests while isolating the failure.

Step 3.2: Build with MinGW Instead of Visual Studio
Alternative compiler toolchain may handle process creation differently:

1. Install MSYS2 + MinGW-w64
2. Configure with:

./configure --with-tcl=$PWD/compat/tcl/lib

3. make test

Step 3.3: Use SQLite Amalgamation Build
Despite original requirement to build from source, temporarily test amalgamation:

nmake /f Makefile.msc sqlite3.c
nmake /f Makefile.msc test

If this succeeds, indicates issues in multi-directory build configuration.

Final Resolution Path:

1. Replace IronTcl with ActiveState TCL 8.6.13
2. Revert TCLSUFFIX = t to original value (empty) if using non-threaded TCL
3. Ensure TOP = .. doesn’t break relative paths to TCL headers
4. Apply pipe security attribute patch to testfixture.c

Post-resolution validation:

• Successful completion of fts5multiclient.test
• No broken pipe errors in test-out.txt
• All veryquick.test suites report "Ok" status