Resolving “ERROR: stale evidence at build.c:3712” in SQLite Documentation Builds

Understanding the "ERROR: stale evidence at build.c:3712" Issue

The "ERROR: stale evidence at build.c:3712" error is a recurring issue encountered during the build process of SQLite’s documentation. This error is not a traditional compilation error but rather a discrepancy between the documentation’s claims and the corresponding test evidence in the SQLite test suite. The error message indicates that the evidence markers, which link specific statements in the documentation to test cases that validate those statements, are outdated or mismatched. These evidence markers are critical for ensuring that the documentation accurately reflects the behavior of the SQLite library.

The error typically arises during the execution of the make all command, which builds the full documentation suite, including the evidence matrix. The evidence matrix is a comprehensive mapping of documentation claims to test cases, ensuring that every documented feature or behavior is backed by verifiable tests. When the evidence markers are stale, the build process flags these discrepancies, resulting in the "stale evidence" errors.

The issue is particularly noticeable when building the documentation from the latest checkouts of the SQLite source and documentation repositories. The documentation and the test suite are maintained separately, and there can be a lag between updates to the documentation and the corresponding updates to the test evidence. This lag is often due to the rapid pace of development in SQLite, where new features and changes are frequently introduced.

Root Causes of Stale Evidence Errors

The primary cause of the "stale evidence" errors is the asynchronous development of the SQLite documentation and the test suite. The documentation is updated to reflect new features, changes, or clarifications, but the corresponding test evidence may not be immediately updated to match these changes. This discrepancy can occur for several reasons:

1. Documentation Updates Without Test Updates: When developers update the documentation to reflect new features or changes, they may not immediately update the test suite to include evidence markers for these changes. This is especially common during active development phases, where documentation is updated frequently.

2. Test Suite Refactoring: The SQLite test suite is extensive and undergoes regular refactoring to improve coverage and maintainability. During these refactoring efforts, evidence markers may be inadvertently removed or modified, leading to mismatches with the documentation.

3. Version Mismatch Between Documentation and Source: The SQLite documentation and source code are maintained in separate repositories. If the documentation is built against a version of the source code that does not match the latest test suite, evidence markers may not align correctly, resulting in stale evidence errors.

4. Incremental Builds: The SQLite documentation build process supports incremental builds to improve efficiency. However, incremental builds can sometimes fail to update evidence markers correctly, especially if the build process is interrupted or if there are changes to the build environment.

5. Manual Evidence Marker Updates: Evidence markers are sometimes updated manually, and human error can lead to inconsistencies. For example, a developer might forget to update a marker after making a change to the documentation or the test suite.

Steps to Resolve Stale Evidence Errors

Resolving the "stale evidence" errors requires a combination of manual intervention and adjustments to the build process. Below are detailed steps to address the issue:

1. Use make fast for Documentation Builds

The simplest way to avoid stale evidence errors is to use the make fast command instead of make all. The make fast command builds a subset of the documentation that excludes the evidence matrix, thereby avoiding the errors related to stale evidence. This approach is recommended for users who do not require the full evidence matrix and are primarily interested in the documentation content.

Command:

make fast

Considerations:

• The make fast command does not generate the evidence matrix, which means that the documentation will not include the links between claims and test cases.
• This approach is suitable for most users who do not need to validate the documentation against the test suite.

2. Update Evidence Markers Manually

If you require the full documentation suite, including the evidence matrix, you can update the evidence markers manually. This process involves identifying the stale markers and updating them to match the current state of the documentation and the test suite.

Steps:

• Identify the stale evidence markers by examining the build output. Each error message includes the file and line number where the discrepancy occurs.
• Locate the corresponding test case in the SQLite test suite. The test case should validate the claim made in the documentation.
• Update the evidence marker in the documentation to reference the correct test case. This may involve modifying the documentation source files or the test suite.

Example:
If the error message indicates a stale marker in foreignkeys.html, you would:

• Open foreignkeys.html and locate the claim that is flagged as having stale evidence.
• Find the corresponding test case in the SQLite test suite (e.g., e_fkey.test).
• Update the evidence marker in foreignkeys.html to reference the correct test case.

Considerations:

• Manual updates require a deep understanding of both the documentation and the test suite.
• This process can be time-consuming, especially if there are many stale markers.

3. Synchronize Documentation and Source Code Versions

Ensure that the documentation and source code are synchronized to the same version. This reduces the likelihood of discrepancies between the documentation and the test suite.

Steps:

• Check out the latest versions of both the SQLite source code and documentation repositories.
• Verify that the versions are compatible by examining the commit history and release notes.

Example:

fossil open /path/to/sqlite.fossil trunk
fossil open /path/to/sqliteDoc.fossil trunk

Considerations:

• Synchronizing versions may not always resolve stale evidence errors, especially if the test suite has not been updated to match the latest documentation.

4. Rebuild the Evidence Matrix from Scratch

If the stale evidence errors persist, you can rebuild the evidence matrix from scratch. This process involves regenerating the evidence markers based on the current state of the documentation and the test suite.

Steps:

• Clean the build environment to remove any cached or intermediate files.
• Rebuild the documentation using the make all command.

Example:

make clean
make all

Considerations:

• Rebuilding the evidence matrix from scratch can be resource-intensive and time-consuming.
• This approach may not resolve all stale evidence errors if there are underlying issues with the documentation or test suite.

5. Contact SQLite Consortium for Support

If the above steps do not resolve the issue, consider reaching out to the SQLite Consortium for support. Consortium members have access to priority support and can request updates to the documentation and test suite to address stale evidence errors.

Steps:

• Join the SQLite Consortium to gain access to priority support.
• Submit a support request detailing the stale evidence errors and their impact on your workflow.

Considerations:

• Consortium membership requires a financial commitment.
• This approach is recommended for organizations that rely heavily on SQLite and require timely resolution of documentation issues.

Conclusion

The "ERROR: stale evidence at build.c:3712" error is a known issue in the SQLite documentation build process, arising from discrepancies between the documentation and the test suite. While the error does not affect the functionality of the SQLite library, it can be a concern for organizations with strict compliance requirements. By using the make fast command, manually updating evidence markers, synchronizing documentation and source code versions, or rebuilding the evidence matrix, you can mitigate or resolve these errors. For organizations requiring guaranteed resolution, joining the SQLite Consortium provides access to priority support and ensures that documentation issues are addressed promptly.