Memory Leak in SQLite with ICU Extension in Python

Memory Growth Detected During Repeated ICU Collation Loading

The core issue revolves around a memory leak observed when repeatedly opening and closing SQLite databases while using the ICU extension to load a French collation. The memory leak manifests as a steady increase in memory usage, as monitored by tools like htop. The leak is not attributable to the Python code itself, as profiling the Python script did not reveal any memory-related issues. However, when the ICU collation loading step is omitted, the memory leak disappears, suggesting that the issue lies in the interaction between SQLite, the ICU extension, and the Python SQLite3 module.

The problem is particularly noticeable when thousands of databases are opened and closed in rapid succession, with each operation involving the loading of the ICU extension and the execution of a collation-loading query. The memory growth persists even after the database connections are closed, indicating that some resources allocated during the ICU collation loading process are not being properly released.

ICU Resource Management and SQLite Integration

The memory leak is likely caused by improper resource management within the ICU extension or its integration with SQLite. The ICU (International Components for Unicode) library provides robust support for Unicode text processing, including collation, which is essential for handling locale-specific sorting rules. When SQLite loads the ICU extension, it dynamically links to the ICU library and uses its functions to perform collation operations.

One possible cause of the memory leak is that the ICU resources allocated during the collation loading process are not being fully released when the database connection is closed. Specifically, the icu_load_collation function in SQLite’s ICU extension may be failing to clean up resources such as collator objects (ucol), which are created using the ucol_open function from the ICU library. While the SQLite code includes a ucol_close call to release these resources, there may be scenarios where this cleanup is not executed correctly, leading to memory leaks.

Another potential cause is the interaction between Python’s garbage collection and the ICU extension. Python’s SQLite3 module manages database connections and cursors, but it may not fully account for resources allocated by external libraries like ICU. If the ICU extension retains references to allocated memory or fails to release resources in a way that Python’s garbage collector can handle, memory leaks can occur.

Additionally, the repeated loading and unloading of the ICU extension itself could contribute to the memory leak. Each time the extension is loaded, it may allocate memory for internal structures or caches that are not fully released when the extension is unloaded. Over thousands of iterations, these small memory allocations can accumulate, leading to significant memory growth.

Profiling ICU with Valgrind and Implementing Resource Cleanup

To diagnose and resolve the memory leak, a systematic approach involving profiling, code inspection, and resource management improvements is necessary. The first step is to use Valgrind to profile the ICU library and identify any memory allocations that are not being properly released. Valgrind is a powerful tool for detecting memory leaks in C and C++ code, and it can be used to trace memory allocations and deallocations within the ICU library.

To profile the ICU library through Python, you can use Valgrind’s memcheck tool to run the Python interpreter and monitor memory usage during the execution of the script. This will help identify any memory allocations within the ICU library that are not being released. The following command can be used to run the Python script under Valgrind:

valgrind --tool=memcheck --leak-check=full python3 script.py

Once the memory leak is confirmed and its source identified, the next step is to ensure that all ICU resources are properly released. This involves modifying the SQLite ICU extension code to include additional cleanup steps, such as explicitly closing collator objects and freeing any allocated memory. For example, the icu_load_collation function should be updated to ensure that ucol_close is called for every ucol_open, even in error scenarios.

In addition to modifying the ICU extension, it may be necessary to adjust the way the ICU extension is loaded and unloaded in Python. Instead of enabling and disabling the extension for each database connection, consider loading the ICU extension once at the start of the program and keeping it loaded for the duration of the program’s execution. This reduces the overhead of repeatedly loading and unloading the extension and minimizes the risk of memory leaks.

Finally, implementing a robust error handling and resource cleanup mechanism in the Python script can help mitigate memory leaks. This includes using context managers (with statements) to ensure that database connections and cursors are properly closed, even in the event of an error. The following example demonstrates how to use a context manager to handle database connections:

import sqlite3
from contextlib import closing

path_ = "test_run_41.db"
icu_extension = "./icu.cpython-38-x86_64-linux-gnu.so"

for i in range(0, 5000):
    print("OPEN {}".format(path_))
    with closing(sqlite3.connect(path_)) as conn:
        conn.isolation_level = None
        with closing(conn.cursor()) as cursor:
            conn.enable_load_extension(True)
            conn.load_extension(icu_extension)
            cursor.execute("select icu_load_collation(?, ?)", ("fr-u-kn", "fr"))
            conn.enable_load_extension(False)

By following these steps, you can identify and resolve the memory leak in SQLite when using the ICU extension. Profiling with Valgrind will help pinpoint the source of the leak, while code modifications and improved resource management will ensure that all ICU resources are properly released. Implementing these changes will result in a more stable and memory-efficient application.