Resolving Missing API Exports When Building SQLite Extensions via EXTRA_SRC on Windows

Windows-Specific Library Export Failures with EXTRA_SRC and Amalgamation Conflicts

Issue Overview: EXTRA_SRC Extensions Break SQLite API Visibility in Windows DLLs

When compiling SQLite extensions from the ext/misc/ directory using the EXTRA_SRC build parameter on Windows platforms, developers encounter a critical issue where the resulting shared library (DLL) loses all standard SQLite API exports. This manifests as a DLL containing only the extension initialization function (e.g., sqlite3_eval_init) while missing essential symbols like sqlite3_open, sqlite3_exec, and other core API entries. The problem surfaces exclusively in Windows builds due to platform-specific symbol export handling through __declspec(dllexport) attributes.

The root technical conflict arises from dual compilation scenarios:

1. Amalgamation Builds: Standard SQLite compilation uses a single sqlite3.c amalgamation file that internally manages symbol exports via preprocessor macros.
2. Extension Integration: When adding extensions via EXTRA_SRC, their source files directly include sqlite3ext.h without guarding against amalgamation compilation contexts. This header contains Windows-specific export declarations that override the amalgamation’s export strategy.

The collision occurs because sqlite3ext.h unconditionally defines SQLITE_EXTENSION_INIT1 and SQLITE_EXTENSION_INIT2 macros, which are designed for building extensions as separate loadable modules. When these are compiled into the main library via EXTRA_SRC, they redefine the API export mechanism, stripping existing exports from the DLL’s symbol table. Windows linkers then produce a .def file and DLL with only the extension’s initialization symbols, rendering the library unusable for normal API calls.

Possible Causes: Unconditional Extension Headers and Declspec Export Overrides

Three primary factors contribute to this Windows-specific build failure:

1. Missing SQLITE_AMALGAMATION Guards in Extension Sources

SQLite extensions in ext/misc/ typically include sqlite3ext.h without checking whether they’re being compiled as part of the amalgamation. This header contains critical initialization macros (SQLITE_EXTENSION_INIT1, SQLITE_EXTENSION_INIT2) and Windows export attributes. When EXTRA_SRC injects these files into an amalgamation build:

/* eval.c (problematic inclusion) */
#include "sqlite3ext.h"  // Always included, even in amalgamation
SQLITE_EXTENSION_INIT1
// ... extension code ...

The sqlite3ext.h header redefines API symbol visibility by injecting __declspec(dllexport) directives that conflict with the amalgamation’s existing export strategy. This occurs because the amalgamation normally centralizes export definitions in sqlite3.c, while sqlite3ext.h assumes it’s building a standalone extension.

2. Declspec Attribute Collision in Windows DLLs

Windows platforms require explicit symbol export declarations via __declspec(dllexport) to expose functions from DLLs. The SQLite amalgamation carefully controls these exports through centralized macros. However, when extensions include sqlite3ext.h, they introduce additional __declspec attributes that override the amalgamation’s settings. For example:

/* sqlite3ext.h (excerpt) */
#ifdef _WIN32
# define SQLITE_EXTENSION_EXPORT __declspec(dllexport)
#else
# define SQLITE_EXTENSION_EXPORT
#endif

/* eval.c (after preprocessing) */
SQLITE_EXTENSION_EXPORT int sqlite3_eval_init(
  sqlite3 *db, 
  char **pzErrMsg, 
  const sqlite3_api_routines *pApi
){
  // ... init code ...
}

When compiled into the main DLL via EXTRA_SRC, this forces the linker to export only the sqlite3_eval_init symbol, discarding all other API exports defined in the amalgamation. The result is a DLL with an incomplete export table.

3. Build Script Limitations in Handling Extension Sources

The SQLite build system (specifically tools/mksqlite3c.tcl) processes EXTRA_SRC files differently than regular amalgamation components. When constructing shell.c or sqlite3.c, the script strips certain platform-specific attributes from extensions to prevent symbol conflicts. However, this sanitization doesn’t occur when building the library directly via make libsqlite3.la with EXTRA_SRC, leaving Windows-specific declarations intact. The oversight stems from two factors:

• Platform Agnosticism: Most SQLite build logic targets Unix-like systems, where symbol visibility is handled via linker scripts or compiler attributes without the need for __declspec.
• Undocumented EXTRA_SRC Behavior: The EXTRA_SRC parameter isn’t formally documented, leading to inconsistent handling of platform-specific code paths during source inclusion.

Troubleshooting Steps: Conditional Compilation and Build System Modifications

Step 1: Modify Extension Sources with SQLITE_AMALGAMATION Guards

Update extension source files to conditionally include sqlite3ext.h and wrap Windows export attributes:

/* eval.c (fixed) */
#ifndef SQLITE_AMALGAMATION  // Guard against amalgamation builds
#include "sqlite3ext.h"
SQLITE_EXTENSION_INIT1
#else
#include "sqlite3.h"  // Use amalgamation headers
#endif

// ... extension code ...

#ifndef SQLITE_AMALGAMATION
#ifdef _WIN32
__declspec(dllexport)  // Only export in standalone extension builds
#endif
#endif
int sqlite3_eval_init(
  sqlite3 *db, 
  char **pzErrMsg, 
  const sqlite3_api_routines *pApi
){
  SQLITE_EXTENSION_INIT2(pApi);
  // ... init code ...
}

Key Modifications:

• Conditional Header Inclusion: Prevents sqlite3ext.h from being included in amalgamation builds, avoiding macro redefinitions.
• Guarded Declspec Attributes: Ensures __declspec(dllexport) is only active when building the extension as a separate DLL, not when compiled into the main library via EXTRA_SRC.
• Selective Initialization: Uses SQLITE_EXTENSION_INIT2 only in non-amalgamation contexts to prevent API pointer mismanagement.

Step 2: Patch Build Scripts to Sanitize EXTRA_SRC Inputs

Modify the TCL script responsible for amalgamation generation (tools/mksqlite3c.tcl) to automatically strip __declspec(dllexport) attributes from EXTRA_SRC files:

# tools/mksqlite3c.tcl (excerpt)
proc copy_file_verbatim {filename {stripDeclspec 0}} {
  global out
  set in [open $filename rb]
  set tail [file tail $filename]
  fconfigure $in -translation binary
  section_comment "Begin EXTRA_SRC file $tail"
  while {![eof $in]} {
    set line [string trimright [gets $in]]
    if {$stripDeclspec} {
      # Remove Windows declspec attributes
      regsub -all {__declspec\(dllexport\)} $line {} line
    }
    puts $out $line
  }
  section_comment "End of EXTRA_SRC $tail"
  close $in
}

# Later in the script, when processing EXTRA_SRC:
foreach file $extrasrc {
  copy_file_verbatim $file 1  # Enable declspec stripping
}

Build Process Impact:

• Declspec Sanitization: Automatically removes __declspec(dllexport) from all lines in EXTRA_SRC files during amalgamation generation.
• Backward Compatibility: Preserves existing behavior for non-Windows builds while mitigating symbol conflicts on Windows.

Step 3: Update Windows Build Documentation and Makefiles

Add platform-specific warnings to SQLite’s Windows build infrastructure:

1. Makefile.msc Addendum:

# Microsoft Makefile (Makefile.msc) excerpt
# When using EXTRA_SRC on Windows, ensure extension sources do NOT contain
# __declspec(dllexport) attributes. These will be stripped automatically
# during amalgamation generation. See forum post [link] for details.

2. Build Process Documentation:

Although EXTRA_SRC remains undocumented, include remarks in the source tree’s README.md:

## Windows Build Notes

When including third-party extensions via `EXTRA_SRC` on Windows:
- Ensure extension sources guard against `__declspec(dllexport)` in amalgamation builds
- Verify generated `.def` files contain all expected SQLite API symbols
- Refer to [forum thread](link) for troubleshooting missing exports

Step 4: Validate Generated DEF Files and Exports

After applying fixes, inspect the linker-generated .def file to confirm all SQLite API symbols are present:

1. Post-Build Validation Command:

grep -E '^(sqlite3_(open|exec|prepare|step))' sqlite3.def

2. Expected Output:

sqlite3_open
sqlite3_open_v2
sqlite3_exec
sqlite3_prepare
sqlite3_prepare_v2
sqlite3_step

3. Diagnostic Actions for Missing Symbols:

• Reinspect Extension Guards: Verify #ifndef SQLITE_AMALGAMATION wraps all sqlite3ext.h inclusions and __declspec usage.
• Check Build Script Patches: Confirm tools/mksqlite3c.tcl correctly strips __declspec from EXTRA_SRC files.
• Test with Minimal Extensions: Rebuild with a single extension file to isolate conflicting sources.

Step 5: Implement Alternative Export Strategies (Advanced)

For complex scenarios requiring both amalgamation and extension exports, consider these advanced solutions:

1. Dual-Phase Symbol Export Macros

Define platform-specific export macros that adapt to amalgamation/extension contexts:

/* sqlite3_custom.h */
#if defined(_WIN32) && defined(SQLITE_AMALGAMATION)
# define SQLITE_EXPORT __declspec(dllexport)
#elif defined(_WIN32)
# define SQLITE_EXPORT __declspec(dllimport)
#else
# define SQLITE_EXPORT
#endif

/* eval.c */
#include "sqlite3_custom.h"
SQLITE_EXPORT int sqlite3_eval_init(...){...}

2. Linker Definition Files (.def)

Manually control exported symbols via a .def file instead of __declspec:

1. Create sqlite3.def with all public API functions
2. Modify build commands to use the definition file:

gcc -shared -o sqlite3.dll sqlite3.o -Wl,--export-all-symbols -Wl,--exclude-libs,ALL -Wl,--def,sqlite3.def

3. Version Scripts (Unix-style) for Windows

While less common, use GCC’s --version-script option on Windows MinGW builds:

gcc -shared -o sqlite3.dll sqlite3.o -Wl,--version-script=sqlite3.exports

Where sqlite3.exports contains:

{
  global:
    sqlite3_*;
  local:
    *;
};

Long-Term Maintenance Considerations

To prevent regression and streamline Windows builds:

1. CI Pipeline for Windows EXTRA_SRC Builds

   • Add automated Windows compilation tests using GitHub Actions or Azure Pipelines
   • Verify .def file completeness in CI artifacts

2. Extension Source Template Updates

   • Modify ext/misc template generators to include amalgamation guards by default:

# Code generator snippet for new extensions
echo "#ifndef SQLITE_AMALGAMATION" >> $filename
echo "#include \"sqlite3ext.h\"" >> $filename
echo "SQLITE_EXTENSION_INIT1" >> $filename
echo "#else" >> $filename
echo "#include \"sqlite3.h\"" >> $filename
echo "#endif" >> $filename

3. Community Outreach
   • Update SQLite’s How To Compile wiki with Windows-specific EXTRA_SRC caveats
   • Encourage extension contributors to test amalgamation builds on Windows

By systematically addressing header inclusion guards, build script sanitization, and platform-specific export mechanisms, developers can reliably integrate SQLite extensions via EXTRA_SRC on Windows without compromising core API visibility. The combination of source-level guards and build system patches provides a robust solution that maintains compatibility across all supported platforms.