Resolving Mobile Access Errors Due to Anti-Robot JavaScript Checks in SQLite Forum

---

Understanding the Mobile Access Error and Anti-Robot Defense Mechanism

The core issue revolves around users encountering access errors when attempting to view SQLite Forum content on mobile devices. This problem emerged after server-side changes were implemented to combat excessive resource consumption by web crawlers (robots). The forum software uses JavaScript-based validation to differentiate human users from automated bots. When a client (browser) without JavaScript support attempts to access protected resources, they are redirected to an error page or blocked from viewing content. Mobile users reported persistent errors even with JavaScript enabled and while logged in, suggesting a conflict between device-specific configurations and server-side robot detection logic.

The anti-robot system employs two primary defenses:

1. JavaScript Hyperlink Rewriting: All forum hyperlinks are initially set to point to a /honeypot endpoint. Legitimate browsers execute JavaScript to rewrite these links to their true destinations during page load or user interaction (mouse movement/touch).
2. User-Agent Analysis: Requests from clients with User-Agent strings associated with known crawlers are blocked or subjected to additional checks.

Mobile devices triggered false positives in this system due to three factors:

• JavaScript Execution Timing: Some mobile browsers delay or optimize JavaScript execution to conserve resources, causing link rewriting to fail.
• User-Agent String Ambiguity: Android and iOS browsers often omit critical identifiers in their User-Agent strings, making them resemble crawlers.
• Touchscreen Interaction Logic: The original defense mechanism relied on mouse events (mousemove/mousedown) to activate link rewriting, which doesn’t translate cleanly to touchscreen devices.

This created a paradox where legitimate users were blocked unless they explicitly enabled JavaScript (which was already active) or logged in with credentials. The server’s inability to reliably distinguish between mobile browsers and crawlers led to widespread access issues, particularly for Android devices and Safari-based browsers.

---

Root Causes of False Positives in Robot Detection

JavaScript Dependency in Resource Loading

The forum’s anti-robot system assumes all human-operated browsers will execute JavaScript synchronously during page load. Mobile browsers often defer non-essential JavaScript execution to improve perceived performance, especially on slower networks. This delay caused the link rewriting logic to miss the initial rendering cycle, leaving /honeypot links intact. When users clicked these unmodified links, the server interpreted the request as coming from a bot.

User-Agent String Misclassification

The server’s User-Agent filtering logic used overly broad patterns to flag potential crawlers. For example:

• Android Browsers: User-Agent strings like Mozilla/5.0 (Linux; Android 12; SM-G998U) AppleWebKit/537.36 lack explicit browser identifiers (e.g., "Chrome" or "Firefox"), causing them to be misclassified.
• Safari on iOS: All iOS browsers use WebKit under Apple’s restrictions, making their User-Agent strings nearly identical to crawlers that spoof Safari.
• Legacy Version Strings: Older browser versions (e.g., Version/8.0 Safari/537.36) were disproportionately flagged due to crawlers often using outdated version numbers.

Browser-Specific Handling of Visited Links

The link rewriting mechanism interfered with how browsers track visited URLs. Safari-based browsers (including all iOS browsers) do not update link coloration when JavaScript modifies the href attribute. This created confusion for users relying on color cues to identify viewed content. Chrome and Firefox handle this correctly but only if the User-Agent is explicitly whitelisted in the server’s detection logic.

Server-Side Configuration Overreach

Initial implementations of the anti-robot system used aggressive null-routing for any client that:

• Lacked a Referer header
• Accessed more than 50 pages per minute
• Had a User-Agent string containing keywords like "bot" or "crawler"

These heuristics failed to account for legitimate mobile use cases where users might:

• Open multiple forum threads in rapid succession
• Use browsers that omit the Referer header for privacy
• Have User-Agent strings truncated by carrier-grade proxies

---

Comprehensive Solutions for Restoring Mobile Access

Step 1: Validate JavaScript Execution and User-Agent Compatibility

For End Users:

• Navigate to the forum’s /test_env page (e.g., https://sqlite.org/forum/test_env).
• Confirm the g.javascriptHyperlink parameter is set to 1, indicating anti-robot defenses are active.
• Check the HTTP_USER_AGENT value matches your device’s browser. For Android Firefox, expect:
  Mozilla/5.0 (Android 12; Mobile; rv:97.0) Gecko/97.0 Firefox/97.0
• If the User-Agent is incorrect, update the browser or disable data-saving modes that might alter the header.

For Administrators:

• Modify User-Agent whitelisting rules to include partial matches for mobile browsers:

  -- Example: Whitelist Android and iOS without requiring full browser identification
  UPDATE robot_filter SET is_whitelisted = 1 
  WHERE user_agent LIKE '%Android%' 
     OR user_agent LIKE '%iPhone%' 
     OR user_agent LIKE '%iPad%';
  

• Implement a secondary validation step for ambiguous User-Agents using TLS fingerprinting or HTTP/2 protocol characteristics, which are harder for crawlers to spoof.

Step 2: Adjust JavaScript Link Rewriting Logic for Touchscreen Devices

Modify the forum’s JavaScript to account for touch events and mobile rendering behaviors:

// Replace mouse event listeners with a hybrid approach
document.addEventListener('DOMContentLoaded', function() {
  // Immediate rewrite for whitelisted User-Agents
  if (/Android|webOS|iPhone|iPad/i.test(navigator.userAgent)) {
    rewriteHyperlinks();
  } else {
    // Fallback to mouse/touch events for others
    document.addEventListener('mousemove', rewriteHyperlinks, {once: true});
    document.addEventListener('touchstart', rewriteHyperlinks, {once: true});
  }
});

function rewriteHyperlinks() {
  document.querySelectorAll('a[href^="/honeypot"]').forEach(a => {
    const truePath = a.dataset.trueHref; // Server should preload this
    a.href = truePath;
  });
}

This script prioritizes immediate link rewriting for mobile User-Agents while retaining event-based activation for desktop browsers. The server must precompute the data-true-href attribute during page generation to avoid dependency on client-side parsing.

Step 3: Implement Browser-Specific CSS for Visited Links

To address Safari’s limitation in updating link colors after JavaScript modifications, add a server-side CSS override for Safari-based browsers:

/* Detect Safari using User-Agent */
@media not all and (min-resolution:.001dpcm) { 
  @supports (-webkit-appearance:none) {
    a:visited {
      color: #551a8b !important; /* Standard visited link color */
    }
  }
}

Combine this with a cookie-based tracking system that marks visited links:

// After rewriting links, mark visited state via cookies
function trackVisitedLinks() {
  const links = document.querySelectorAll('a[href]');
  links.forEach(link => {
    if (document.cookie.includes(`visited=${link.href}`)) {
      link.classList.add('visited');
    }
    link.addEventListener('click', () => {
      document.cookie = `visited=${link.href}; max-age=${30*24*60*60}`;
    });
  });
}

Step 4: Deploy Progressive Challenges for Ambiguous Clients

Instead of blanket blocking, use incremental challenges:

1. First-Layer Defense: Serve unmodified content to clients with whitelisted User-Agents or valid cookies.
2. Second-Layer Defense: For unknown User-Agents, inject a hidden CAPTCHA challenge within a <noscript> tag. Bots parsing the page will trigger the CAPTCHA, while humans remain unaffected.
3. Third-Layer Defense: Rate-limit clients that exceed 90 requests per minute, presenting a login prompt with the CAPTCHA password (e.g., "anonymous" login).

Example CAPTCHA integration:

<noscript>
  <form action="/challenge" method="POST" style="display:none;">
    <input type="hidden" name="redirect" value="<!-- Current URL -->">
    <input type="text" name="captcha_response">
    <input type="submit">
  </form>
  <script>
    // Auto-submit CAPTCHA for bots that parse JS but don't execute it
    document.forms[0].submit();
  </script>
</noscript>

Step 5: Enable Optional Authentication Bypass

Allow users to opt into a low-friction experience by promoting the "anonymous" account:

• Publicize the CAPTCHA password during error conditions:
  Login with username "anonymous" and password displayed below:
• Implement a persistent cookie for anonymous users that skips future CAPTCHAs:

  Set-Cookie: anonymous_auth=1; Path=/; Max-Age=2592000; SameSite=Lax
  

Step 6: Continuous Monitoring and User-Agent Database Updates

Maintain a dynamically updated database of User-Agent strings with the following schema:

CREATE TABLE user_agent_profiles (
  id INTEGER PRIMARY KEY,
  user_agent_hash TEXT UNIQUE,  -- SHA-256 of normalized User-Agent
  classification TEXT CHECK(classification IN ('human', 'bot', 'unknown')),
  first_seen TIMESTAMP,
  last_seen TIMESTAMP,
  request_count INTEGER
);

CREATE TABLE user_agent_patterns (
  id INTEGER PRIMARY KEY,
  pattern TEXT UNIQUE,  -- SQL LIKE pattern with % wildcards
  classification TEXT
);

Populate the user_agent_patterns table with known mobile browser signatures and schedule weekly updates based on crowd-sourced data from legitimate users.

---

By systematically addressing the interplay between JavaScript execution timing, User-Agent detection inaccuracies, and mobile browser quirks, administrators can restore access for legitimate users while maintaining robust defenses against crawlers. End users should ensure their browsers are updated, JavaScript is enabled without restrictions, and consider using the "anonymous" login during transitional periods while server-side improvements are deployed.