Exclusion of default.test Due to *fault* Wildcard in Test Suite Configuration

Understanding the Unintended Exclusion of default.test in SQLite Test Suites

The core issue revolves around the accidental exclusion of the default.test file from SQLite’s test suite execution due to a wildcard pattern (*fault*) used in the test configuration. This problem arises in the test/permutations.test file, specifically in the configuration of the "veryquick" test suite. The exclusion pattern *fault* inadvertently matches default.test because the substring "fault" appears in its filename, even though the test itself is unrelated to fault testing. This leads to gaps in test coverage and potential undetected regressions in SQLite’s functionality.

The misconfiguration occurs in the section of code that defines which test files are excluded from the "veryquick" test suite. The -exclude flag uses the pattern *fault* to omit tests related to fault injection or error handling. However, the default.test file, which likely contains baseline tests for default SQLite behaviors, is excluded solely because its name contains the letters "fault" as a substring. This is a classic case of overbroad wildcard matching, where a pattern intended to filter specific test categories ends up excluding unrelated tests due to naming overlaps.

The implications of this issue extend beyond the immediate exclusion of default.test. Similar wildcard-based exclusions in other parts of the permutations.test file (and potentially other test configurations) may unintentionally omit additional test files. This undermines the integrity of the test suite, as critical tests may be skipped without explicit intent.

Root Causes of Overbroad Wildcard Matching in Test Suite Configuration

The accidental exclusion of default.test stems from two primary factors:

1. Ambiguous Wildcard Usage in Exclusion Patterns
   The *fault* wildcard pattern is designed to exclude test files related to fault injection, error handling, or system failure simulations. However, the * wildcard in SQLite’s test suite configuration operates as a substring matcher, not a targeted identifier. This means any file containing "fault" in its name—regardless of its position or context—is excluded. The default.test file, despite not being related to fault testing, is caught by this pattern because "fault" is a substring of its name.

   Wildcard patterns in test configurations are powerful but risky. They rely on precise naming conventions to avoid false positives. In this case, the absence of a delimiter or positional constraint (e.g., fault_*.test to match files starting with "fault_") allows unrelated files to be excluded.

2. Insufficient Guardrails for Test File Naming Conventions
   SQLite’s test files follow a naming convention that reflects their purpose (e.g., malloc.test for memory allocation tests, ioerr.test for I/O error tests). However, the presence of a file named default.test violates an implicit rule: avoiding substrings that overlap with exclusion patterns. The name "default.test" includes the substring "fault," which conflicts with the *fault* exclusion pattern.

   This highlights a broader issue in test suite maintenance: the lack of enforced naming conventions or automated checks to prevent such overlaps. Without safeguards, new test files (or legacy ones) risk being excluded unintentionally if their names inadvertently match exclusion patterns.

3. Cascading Exclusions Across Multiple Test Suites
   The permutations.test file defines multiple test suites (e.g., "quick," "veryquick") with shared exclusion patterns. A single overbroad wildcard in one suite’s configuration can propagate exclusions to other suites if patterns are reused or inherited. The original discussion notes 13 locations in the source file where similar exclusions might occur, suggesting systemic risk.

Resolving the Exclusion Conflict: Targeted Fixes and Preventative Measures

To address the unintended exclusion of default.test and prevent similar issues, the following steps are recommended:

Step 1: Refine Exclusion Patterns to Avoid Substring Collisions

Modify the -exclude pattern in the "veryquick" test suite to exclude only test files explicitly related to fault injection. Instead of *fault*, use a more precise pattern that targets files with "fault" as a standalone term or prefix/suffix. For example:

• fault*.test to match files starting with "fault"
• *_fault.test to match files ending with "_fault"

This narrows the scope of the exclusion while retaining the intended tests.

Code Change Example:

test_suite "veryquick" -prefix "" -description {
 "Very" quick test suite. Runs in minutes on a workstation.
 This test suite is the same as the "quick" tests, except that some files
 that test malloc and IO errors are omitted.
} -files [
 test_set $allquicktests -exclude *malloc* *ioerr* fault* *bigfile* *_err* \
   *fts5corrupt* *fts5big* *fts5aj*
]  

Here, *fault* is replaced with fault*, which excludes only files whose names begin with "fault."

Step 2: Rename Conflicting Test Files

If refining exclusion patterns is impractical (e.g., due to many existing patterns), rename default.test to eliminate the "fault" substring. For instance, renaming it to baseline.test or default_behavior.test would prevent it from being matched by *fault*.

Advantages:

• Avoids changes to exclusion logic.
• Future-proofs against similar substring collisions.

Considerations:

• Ensure all references to default.test in the codebase are updated.
• Verify that the renamed file is included in relevant test suites.

Step 3: Audit All Exclusion Patterns in permutations.test

The original discussion mentions 13 locations where similar exclusions might occur. Perform a systematic audit of all -exclude patterns in permutations.test to identify:

• Overbroad wildcards (e.g., *err* excluding "interrupt.test" due to "err" substring).
• Test files with names that unintentionally match exclusion patterns.

Audit Workflow:

1. Extract all -exclude patterns from the file.
2. For each pattern, list all test files that match it.
3. Cross-reference matched files with their actual purposes to confirm exclusion validity.

Step 4: Implement Naming Conventions and Automated Checks

To prevent future issues, establish formal guidelines for test file naming:

• Avoid common substrings (e.g., "fault," "err") unless they align with the test’s purpose.
• Use delimiters like underscores to separate terms (e.g., fault_injection.test instead of fault.test).

Additionally, integrate automated checks into the development workflow:

• A pre-commit hook that flags test files with names matching active exclusion patterns.
• CI/CD pipeline checks to verify that no critical tests are excluded unintentionally.

Step 5: Validate Test Suite Coverage Post-Fix

After applying fixes, validate that default.test (or its renamed counterpart) is included in the "veryquick" test suite and other relevant suites. Use SQLite’s test runner to execute the suite and confirm:

./testfixture permutations.test veryquick  

Check the output for the presence of default.test (or its new name) in the list of executed tests.

Step 6: Document Changes and Update Maintenance Guidelines

Update SQLite’s internal documentation to reflect:

• The rationale behind exclusion pattern changes.
• New test file naming conventions.
• Procedures for auditing exclusion patterns in the future.

By addressing the root causes and implementing preventative measures, SQLite’s test suites can maintain their rigor while avoiding unintended gaps in test coverage. This approach balances immediate fixes with long-term safeguards, ensuring that the test configuration remains robust against similar issues.