Compiling SQLite Without Amalgamation: Fixing Missing Definitions and Build Errors

Missing SQLITE_OK, Implicit Function Declarations, and Geopoly Compilation Failures

Issue Overview: Compilation Errors When Bypassing the Amalgamation

The core issue arises when attempting to compile SQLite from its individual source files (as opposed to the standard amalgamation) to reduce build times through parallel compilation. The user encountered three primary errors during compilation:

1. Undeclared SQLITE_OK: The compiler reports that SQLITE_OK is undeclared in geopoly.c, despite including SQLite headers.
2. Implicit Function Declarations: Warnings about memset and other standard library functions not being declared, indicating missing header inclusions.
3. Unknown Type Name ‘sqlite3’: The compiler fails to recognize the sqlite3 type in geopoly.c, suggesting missing or incomplete header inclusion.

These errors occur because the SQLite amalgamation is not merely a concatenation of source files. It includes preprocessor directives, header inclusions, and macro definitions that are essential for resolving dependencies between components. When compiling individual files, these dependencies are not automatically satisfied, leading to missing definitions and type declarations.

For example, the geopoly.c file is designed to be included within rtree.c (as noted in the SQLite source comments). Compiling it separately bypasses the context provided by rtree.c, such as access to internal R-Tree structures and SQLite core definitions. Similarly, the absence of sqlite3.h or incorrect inclusion order can cause sqlite3 type declarations to be missing.

The amalgamation also defines critical macros like SQLITE_PRIVATE and SQLITE_EXTERN, which control symbol visibility and linkage. When compiling non-amalgamated sources, these macros must be explicitly defined or adjusted to avoid linkage errors or duplicate symbol definitions.

Possible Causes: Why Non-Amalgamation Builds Fail

1. Missing or Incorrect Preprocessor Definitions

The SQLite build system relies heavily on preprocessor macros to conditionally include code, manage internal APIs, and configure features. Key macros include:

• SQLITE_CORE: Indicates that the build is part of the SQLite core (not an extension). Required to suppress extension-specific code.
• SQLITE_AMALGAMATION: Signals that the amalgamation is being used, enabling certain optimizations and header inclusions.
• SQLITE_PRIVATE: Marks internal functions as static to limit their visibility.

When compiling individual files, omitting these macros (or defining them incorrectly) leads to:

• Undeclared constants (e.g., SQLITE_OK).
• Missing type definitions (e.g., sqlite3).
• Linkage errors due to symbol visibility mismatches.

2. Header Inclusion Order and Dependencies

SQLite’s headers (e.g., sqlite3.h, sqliteInt.h) have strict inclusion orders and dependencies. For example:

• sqliteInt.h depends on sqlite3.h for core type definitions.
• Platform-specific headers (e.g., os.h) require prior inclusion of configuration headers.

In the amalgamation, headers are included in a precise sequence. When compiling files individually, this order is disrupted, causing definitions to appear out of sequence or be missing entirely.

3. File-Specific Compilation Requirements

Certain SQLite source files are not designed to be compiled standalone:

• geopoly.c: Intended to be included within rtree.c to access R-Tree internals.
• tclsqlite.c: Part of the TCL bindings and requires TCL headers and libraries.
• shell.c: Contains the SQLite CLI entry point (main()) and conflicts with application code that defines its own main().

Attempting to compile these files separately without their required context results in errors like:

• 'SQLITE_OK' undeclared (missing sqlite3.h inclusion).
• unknown type name 'sqlite3' (missing forward declarations).

4. Missing Standard Headers

Warnings such as implicit declaration of 'memset' indicate that standard headers (e.g., string.h) are not included where needed. The amalgamation includes these headers upfront, but individual files may omit them, relying on the amalgamation’s inclusion order.

Troubleshooting Steps: Fixing Compilation Errors in Non-Amalgamation Builds

Step 1: Replicate the Amalgamation’s Preprocessor Environment

To compile SQLite’s individual source files successfully, replicate the macro definitions and header inclusion strategy used in the amalgamation.

a. Define Essential Macros
Add these compiler flags to your build system:

-DSQLITE_CORE=1 -DSQLITE_PRIVATE=static -DSQLITE_API=static

• SQLITE_CORE=1: Disables extension-specific code.
• SQLITE_PRIVATE=static: Marks internal functions as static to avoid linkage conflicts.
• SQLITE_API=static: Ensures API functions are correctly scoped.

Avoid defining SQLITE_AMALGAMATION, as this macro assumes headers are included in amalgamation order.

b. Include Headers Correctly
Ensure all source files include sqlite3.h before any internal headers (e.g., sqliteInt.h). Add the following to files causing errors:

#include "sqlite3.h"  
#include <string.h>   // For memset, memcpy, etc.  
#include <stddef.h>   // For NULL, size_t  

c. Fix Platform-Specific Omissions
For platform-dependent files (e.g., os_unix.c, os_win.c), include the appropriate system headers:

#if defined(_WIN32)  
#include <windows.h>  
#else  
#include <unistd.h>  
#endif  

Step 2: Exclude Non-Standalone Source Files

Identify and exclude files not meant for standalone compilation:

a. Remove geopoly.c from the Build
As noted in the SQLite source, geopoly.c is included within rtree.c. Compile rtree.c instead, which internally includes geopoly.c.

b. Exclude TCL and Shell Components

• tclsqlite.c: Requires TCL development packages (tcl.h, tclDecls.h). Exclude unless TCL bindings are needed.
• shell.c: Conflicts with application code defining main(). Remove it from the build.

c. Handle Extensions Conditionally
FTS3/4, RTREE, and JSON1 extensions require additional macros. For example, compile fts3.c with:

-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS  

Step 3: Resolve Missing Symbols and Type Errors

a. Add Missing sqlite3.h Inclusions
In files where sqlite3 types or constants (e.g., SQLITE_OK) are missing, explicitly include sqlite3.h:

#include "sqlite3.h"  

b. Forward-Declare Structures
If sqlite3 is still unrecognized, add forward declarations:

struct sqlite3;  
typedef struct sqlite3 sqlite3;  

c. Include Standard Headers
Fix implicit function warnings by including standard headers in files that use functions like memset:

#include <string.h>  

Step 4: Adjust the Build System for Parallel Compilation

a. Use a Unified Header Directory
Place all SQLite headers (sqlite3.h, sqliteInt.h, etc.) in a single directory (e.g., inc/) and add it to the compiler’s include path:

gcc -Iinc -c src/fts3.c -o obj/fts3.o  

b. Compile with Correct Flags
A minimal compile command for a source file should look like:

gcc -c src/insert.c -o obj/insert.o -Iinc -DSQLITE_CORE=1 -DSQLITE_PRIVATE=static  

c. Use ccache for Faster Rebuilds
As suggested in the forum thread, use ccache to cache object files and drastically reduce rebuild times:

export CC="ccache gcc"  
make clean  
make  

Step 5: Validate the Build Against the Amalgamation

a. Compare Preprocessor Output
Generate the preprocessor output for both amalgamated and non-amalgamated builds to identify discrepancies:

gcc -E sqlite3.c -o amalgamation.i  
gcc -E src/geopoly.c -Iinc -o geopoly.i  

Compare amalgamation.i and geopoly.i to ensure headers and macros are applied consistently.

b. Link Object Files Correctly
Ensure all SQLite object files are linked into the final binary:

gcc obj/*.o -o myapp -lpthread -ldl  

c. Test Core Functionality
After linking, run basic tests to verify SQLite operations:

sqlite3_open(":memory:", &db);  
sqlite3_exec(db, "SELECT sqlite_version()", ...);  

Solutions & Fixes: Ensuring a Successful Non-Amalgamation Build

1. Use a Custom Build Script

Automate the exclusion of non-standalone files and macro definitions. Example script:

#!/bin/bash  
SRC_DIR="src"  
OBJ_DIR="obj"  
INC_DIR="inc"  
EXCLUDES="geopoly.c tclsqlite.c shell.c"  

for file in $(ls $SRC_DIR/*.c); do  
  filename=$(basename $file)  
  if [[ " $EXCLUDES " =~ " $filename " ]]; then  
    continue  
  fi  
  gcc -c $file -o $OBJ_DIR/${filename%.c}.o -I$INC_DIR -DSQLITE_CORE=1 -DSQLITE_PRIVATE=static  
done  

2. Generate a Custom Header for Missing Definitions

Create a sqlite_custom.h header to forward-declare types and constants missing in individual files:

// sqlite_custom.h  
#ifndef SQLITE_CUSTOM_H  
#define SQLITE_CUSTOM_H  

#include "sqlite3.h"  
#include <string.h>  

// Forward declarations  
struct sqlite3;  
struct sqlite3_api_routines;  

// Constants missing in some files  
#ifndef SQLITE_OK  
#define SQLITE_OK 0  
#endif  

#endif  

Include this header in problematic files like geopoly.c.

3. Patch Source Files for Standalone Compilation

Modify files that rely on amalgamation-specific contexts. For geopoly.c:

a. Add Missing Includes
At the top of geopoly.c, add:

#include "sqlite3.h"  
#include "sqliteInt.h"  
#include <string.h>  

b. Comment Out or Fix R-Tree Dependencies
If geopoly.c references R-Tree internals, extract necessary definitions into a shared header or include rtree.c before geopoly.c in the build.

4. Precompile a Platform-Independent Library

As an alternative to compiling sources on every build, precompile SQLite as a static library:

gcc -c sqlite3.c -o sqlite3.o -DSQLITE_CORE=1  
ar rcs libsqlite3.a sqlite3.o  

Link against libsqlite3.a in your project. This avoids recompiling SQLite unless its source changes.

5. Adopt the "Split Amalgamation" Approach

Use the split amalgamation (e.g., sqlite3-1.c, sqlite3-2.c) to balance compilation parallelism and amalgamation benefits. While not fully parallel, this reduces per-file compilation time.

By addressing header inclusion order, preprocessor definitions, and file-specific dependencies, developers can successfully compile SQLite from individual source files. However, this approach requires ongoing maintenance, as SQLite’s internal structure may change between versions. For most projects, combining ccache with the amalgamation remains the simplest and most reliable option.