Transaction-Level Timestamp Consistency in SQLite vs. PostgreSQL

Understanding Timestamp Stability in SQLite and PostgreSQL

Core Issue: Step vs. Transaction Stability

SQLite and PostgreSQL differ fundamentally in how they handle temporal functions like NOW(), CURRENT_TIMESTAMP, and datetime('now').

1. SQLite Behavior (Step Stability)

   • The datetime('now') function returns the exact same value for all calls within a single sqlite3_step() operation (i.e., during the execution of one row retrieval in a query).
   • Example:

     SELECT *, julianday('now') - julianday(modification) AS age FROM player;
     

     If this query processes 1,000 rows, julianday('now') may update multiple times during execution, leading to inconsistent age calculations.

2. PostgreSQL Behavior (Transaction Stability)

   • NOW() returns the start time of the transaction for all calls within that transaction.
   • Guarantees consistency: All modifications in a transaction share the same timestamp.

Why This Matters
Applications expecting PostgreSQL-like behavior may encounter:

• Inconsistent timestamps across related updates/inserts.
• Drift in time-sensitive calculations (e.g., session expiration checks).

Root Causes of SQLite’s Timestamp Behavior

1. Step-Level Time Retrieval
   SQLite fetches the current time from the Virtual File System (VFS) each time sqlite3_step() is called, unless optimized as a constant.

2. Absence of Transaction Timestamp Storage
   Unlike PostgreSQL, SQLite doesn’t store a “transaction start time” at the connection level.

3. Autocommit Mode Interactions

   • Implicit transactions (autocommit mode) reset timestamps for each statement.
   • Explicit transactions (BEGIN/COMMIT) don’t freeze timestamps unless manually managed.

4. Backward Compatibility Constraints
   Changing datetime('now') to be transaction-stable would break existing applications relying on step-level behavior (e.g., long-running queries expecting real-time updates).

Solutions for Consistent Timestamps in SQLite

1. Manual Timestamp Binding

Capture the timestamp at the start of the transaction and propagate it via bound parameters.

Steps:

• Fetch the timestamp before starting the transaction:

  sqlite3_int64 txn_time;
  sqlite3OsCurrentTimeInt64(db->pVfs, &txn_time);
  

• Use this value in all subsequent operations:

  UPDATE player SET score = score + 1, modification = ?;
  

• Advantage: Full control over timestamp consistency.
• Drawback: Requires application-level changes.

2. Custom “Transaction Time” Function

Implement a txn modifier via SQLite’s extension API or source modification.

Patch Example (abridged):

Index: src/date.c
===================================================================
--- src/date.c
+++ src/date.c
@@ -379,10 +379,30 @@
   return 0;
  }else{
   return 1;
  }
 }
+
+static int setDateTimeToTransaction(sqlite3_context *context, DateTime *p){
+ p->iJD = sqlite3ConnCurrentTime(context);
+ if( p->iJD>0 ){
+  p->validJD = 1;
+  return 0;
+ }else{
+  return 1;
+ }
+}
+
 static int parseModifier(
   sqlite3_context *context,
   const char *zMod,
   DateTime *p
 ){
@@ -439,10 +459,12 @@
   return 0;
  }else if( !context ){
   return 1;
  }else if( sqlite3StrICmp(zDate,"now")==0 && sqlite3NotPureFunc(context) ){
   return setDateTimeToCurrent(context, p);
+ }else if( sqlite3StrICmp(zDate,"txn")==0 && sqlite3NotPureFunc(context) ){
+  return context->pVdbe->db->autoCommit ? setDateTimeToCurrent(context, p) : setDateTimeToTransaction(context, p);
  #ifndef SQLITE_OMIT_LOCALTIME
   }else if( sqlite3StrICmp(zDate, "local")==0 && sqlite3NotPureFunc(context) ){
    if(setDateTimeToCurrent(context, p)==0) return toLocaltime(p, context);
  #endif
   }else if( strchr(zDate, '/')>0 ) {

Usage:

SELECT datetime('txn'); -- Returns the transaction start time

Behavior:

• txn uses a connection-level timestamp reset when a transaction begins.
• Falls back to now semantics in autocommit mode.

3. Modify CURRENT_TIMESTAMP Semantics

Adjust CURRENT_TIMESTAMP to use transaction-level stability via compile-time options or pragmas.

Code Change:

// In date.c, change the handling of CURRENT_TIMESTAMP
- datetimeFunc(context, 0, 0);
+ datetimeFunc(context, -1, 0); // Force 'txn' semantics

Considerations:

• May break applications relying on step-level CURRENT_TIMESTAMP.
• Use a pragma (e.g., PRAGMA legacy_timestamp_behavior=ON) to revert.

4. Application-Level Transaction Time Tracking

Store the transaction start time in the application layer and use it in SQL:

# Python example using sqlite3
import sqlite3
import time

conn = sqlite3.connect('test.db')
conn.execute('CREATE TABLE events (id INTEGER, ts TEXT)')
txn_time = time.time()

with conn:
    conn.execute('INSERT INTO events VALUES (1, ?)', (txn_time,))
    conn.execute('INSERT INTO events VALUES (2, ?)', (txn_time,))

Advantage: No SQLite modifications required.

Key Takeaways

1. SQLite Prioritizes Flexibility Over Implicit Guarantees
   Its lightweight design avoids storing transaction metadata unless explicitly requested.

2. Transaction Stability Requires Explicit Action
   Achieve PostgreSQL-like behavior through:

   • Pre-binding timestamps.
   • Custom txn functions.
   • Application-layer time management.

3. Backward Compatibility Limits Built-In Solutions
   Fundamental changes to datetime('now') are unlikely due to the risk of breaking legacy applications.

By understanding these nuances, developers can implement robust timestamp handling tailored to SQLite’s architecture while meeting application-specific consistency requirements.