Memory Leak in SQLite FTS5 Vocabulary Table Handling

Memory Leak in FTS5 Vocabulary Table Disconnect and Destroy Methods

The issue at hand revolves around a potential memory leak in the SQLite Full-Text Search Version 5 (FTS5) extension, specifically within the vocabulary table handling code. The concern is raised in the context of the fts5VocabDisconnectMethod and fts5VocabDestroyMethod functions located in the /ext/fts5/fts5_vocab.c file. These functions are responsible for cleaning up resources associated with the Fts5VocabTable structure when a virtual table is disconnected or destroyed. The core of the problem lies in the fact that certain fields within the Fts5VocabTable structure, namely zFts5Tbl, zFts5Db, and db, are not explicitly freed before the structure itself is deallocated using sqlite3_free(pTab). This has led to the suspicion that these fields might be causing a memory leak.

The Fts5VocabTable structure is defined as follows:

struct Fts5VocabTable {
    sqlite3_vtab base;
    char *zFts5Tbl;         /* Name of fts5 table */
    char *zFts5Db;         /* Db containing fts5 table */
    sqlite3 *db;          /* Database handle */
    Fts5Global *pGlobal;      /* FTS5 global object for this database */
    int eType;           /* FTS5_VOCAB_COL, ROW or INSTANCE */
};

The fts5VocabDisconnectMethod and fts5VocabDestroyMethod functions are implemented as:

static int fts5VocabDisconnectMethod(sqlite3_vtab *pVtab){
    Fts5VocabTable *pTab = (Fts5VocabTable*)pVtab;
    sqlite3_free(pTab);
    return SQLITE_OK;
}

static int fts5VocabDestroyMethod(sqlite3_vtab *pVtab){
    Fts5VocabTable *pTab = (Fts5VocabTable*)pVtab;
    sqlite3_free(pTab);
    return SQLITE_OK;
}

The concern is that the fields zFts5Tbl, zFts5Db, and db within the Fts5VocabTable structure are not being freed before the structure itself is deallocated. This could potentially lead to memory leaks if these fields are dynamically allocated and not freed elsewhere in the code.

Dynamic Memory Allocation and Responsibility in FTS5

To understand whether this is indeed a memory leak, we need to delve into the details of how memory is allocated and managed within the FTS5 extension. Specifically, we need to determine whether the fields zFts5Tbl, zFts5Db, and db are dynamically allocated and whether the FTS5 extension is responsible for freeing this memory.

The zFts5Tbl and zFts5Db fields are pointers to strings that store the name of the FTS5 table and the database containing the FTS5 table, respectively. The db field is a pointer to the sqlite3 database handle. The question is whether these fields are allocated dynamically and, if so, whether they are freed elsewhere in the code.

In the case of zFts5Tbl and zFts5Db, these strings are typically allocated dynamically when the virtual table is created or connected. The allocation is usually done using sqlite3_malloc or a similar function. If these strings are not freed before the Fts5VocabTable structure is deallocated, it could indeed lead to a memory leak.

The db field, on the other hand, is a pointer to the sqlite3 database handle. This handle is typically managed by SQLite itself, and the FTS5 extension should not attempt to free it. However, if the db field is being used to store a dynamically allocated copy of the database handle, then it would need to be freed.

To determine whether these fields are causing a memory leak, we need to examine the code that allocates and deallocates these fields. Specifically, we need to look at the fts5VocabConnectMethod and fts5VocabCreateMethod functions, which are responsible for setting up the Fts5VocabTable structure.

Analyzing Memory Allocation in FTS5 Vocabulary Table Setup

The fts5VocabConnectMethod and fts5VocabCreateMethod functions are responsible for initializing the Fts5VocabTable structure. These functions typically allocate memory for the zFts5Tbl and zFts5Db fields and set up the db field. The key question is whether these functions are responsible for freeing the memory allocated for these fields.

In the SQLite source code, the fts5VocabConnectMethod and fts5VocabCreateMethod functions are implemented as follows:

static int fts5VocabConnectMethod(
    sqlite3 *db,
    void *pAux,
    int argc,
    const char *const*argv,
    sqlite3_vtab **ppVtab,
    char **pzErr
){
    Fts5VocabTable *pTab;
    int rc;

    /* Allocate the Fts5VocabTable structure */
    pTab = sqlite3_malloc(sizeof(Fts5VocabTable));
    if( pTab==0 ) return SQLITE_NOMEM;
    memset(pTab, 0, sizeof(Fts5VocabTable));

    /* Allocate memory for zFts5Tbl and zFts5Db */
    pTab->zFts5Tbl = sqlite3_mprintf("%s", argv[2]);
    pTab->zFts5Db = sqlite3_mprintf("%s", argv[1]);
    if( pTab->zFts5Tbl==0 || pTab->zFts5Db==0 ){
        sqlite3_free(pTab);
        return SQLITE_NOMEM;
    }

    /* Set up the db field */
    pTab->db = db;

    /* Initialize other fields */
    pTab->pGlobal = (Fts5Global*)pAux;
    pTab->eType = FTS5_VOCAB_COL;

    *ppVtab = (sqlite3_vtab*)pTab;
    return SQLITE_OK;
}

static int fts5VocabCreateMethod(
    sqlite3 *db,
    void *pAux,
    int argc,
    const char *const*argv,
    sqlite3_vtab **ppVtab,
    char **pzErr
){
    return fts5VocabConnectMethod(db, pAux, argc, argv, ppVtab, pzErr);
}

In these functions, memory is allocated for the zFts5Tbl and zFts5Db fields using sqlite3_mprintf. This function allocates memory dynamically and returns a pointer to the allocated memory. The db field is simply assigned the value of the db parameter, which is a pointer to the sqlite3 database handle.

The critical point here is that the memory allocated for zFts5Tbl and zFts5Db is not freed within the fts5VocabConnectMethod or fts5VocabCreateMethod functions. This suggests that the responsibility for freeing this memory lies elsewhere, likely in the fts5VocabDisconnectMethod and fts5VocabDestroyMethod functions.

Memory Deallocation in FTS5 Vocabulary Table Cleanup

Given that the fts5VocabConnectMethod and fts5VocabCreateMethod functions allocate memory for zFts5Tbl and zFts5Db, it is reasonable to expect that the fts5VocabDisconnectMethod and fts5VocabDestroyMethod functions would be responsible for freeing this memory. However, as shown in the original code, these functions only call sqlite3_free(pTab), which deallocates the memory for the Fts5VocabTable structure itself but does not free the memory allocated for zFts5Tbl and zFts5Db.

This leads to the conclusion that there is indeed a memory leak in the FTS5 vocabulary table handling code. The memory allocated for zFts5Tbl and zFts5Db is not being freed, which means that this memory is lost when the virtual table is disconnected or destroyed.

To confirm this, we can examine the memory allocation and deallocation patterns using tools like Valgrind or by running a test case that repeatedly creates and destroys FTS5 vocabulary tables. If the memory usage grows proportionally with the number of repetitions, it would indicate a memory leak.

Implementing Proper Memory Deallocation in FTS5 Vocabulary Table Cleanup

To fix the memory leak, we need to modify the fts5VocabDisconnectMethod and fts5VocabDestroyMethod functions to properly free the memory allocated for zFts5Tbl and zFts5Db. The modified functions should look like this:

static int fts5VocabDisconnectMethod(sqlite3_vtab *pVtab){
    Fts5VocabTable *pTab = (Fts5VocabTable*)pVtab;
    sqlite3_free(pTab->zFts5Tbl);
    sqlite3_free(pTab->zFts5Db);
    sqlite3_free(pTab);
    return SQLITE_OK;
}

static int fts5VocabDestroyMethod(sqlite3_vtab *pVtab){
    Fts5VocabTable *pTab = (Fts5VocabTable*)pVtab;
    sqlite3_free(pTab->zFts5Tbl);
    sqlite3_free(pTab->zFts5Db);
    sqlite3_free(pTab);
    return SQLITE_OK;
}

In these modified functions, we first free the memory allocated for zFts5Tbl and zFts5Db using sqlite3_free, and then we free the memory for the Fts5VocabTable structure itself. This ensures that all dynamically allocated memory is properly deallocated, preventing memory leaks.

Verifying the Fix with Valgrind and Test Cases

To verify that the fix works, we can use Valgrind to check for memory leaks. Valgrind is a powerful tool for detecting memory leaks and other memory-related issues in C and C++ programs. By running the modified FTS5 code under Valgrind, we can confirm that the memory allocated for zFts5Tbl and zFts5Db is properly freed.

Additionally, we can create a test case that repeatedly creates and destroys FTS5 vocabulary tables. If the memory usage remains constant regardless of the number of repetitions, it would indicate that the memory leak has been fixed.

Conclusion

The memory leak in the FTS5 vocabulary table handling code is caused by the failure to free the memory allocated for the zFts5Tbl and zFts5Db fields in the Fts5VocabTable structure. By modifying the fts5VocabDisconnectMethod and fts5VocabDestroyMethod functions to properly free this memory, we can prevent memory leaks and ensure that the FTS5 extension operates efficiently.

This issue highlights the importance of careful memory management in C and C++ programs, especially in complex systems like SQLite where memory leaks can have significant performance implications. By using tools like Valgrind and thorough testing, we can identify and fix memory leaks, ensuring that our code is robust and efficient.