Evaluating synthetic evidence

Purpose and when to use it

Use this method when a claim about AI-related content or activity needs an evidence-supported answer. Begin with a bounded question such as whether a particular agent task produced a named external file. A broad question such as whether AI was involved often hides several different propositions.

Examine complementary evidence layers

From question to conclusion

• 1. Question. State the object, event, period, and claim that could be supported or contradicted.
• 2. Preserve. Retain original objects, context, handling records, and integrity measurements.
• 3. Establish Provenance. Identify origin claims and gaps without assuming that absence establishes falsity.
• 4. Inspect Metadata / System Artifacts. Keep raw values and test their interpretation against the relevant format and version.
• 5. Analyze Content. Choose methods suited to this object and preserve preprocessing and tool output.
• 6. Develop Competing Hypotheses. Include ordinary editing, copied content, incomplete logs, and unknown sources when relevant.
• 7. Seek Independent Corroboration. Map dependencies and look for contradictions as well as agreement.
• 8. Test Reproducibility. Retain inputs, settings, controls, derived views, and enough detail to repeat the consequential steps.
• 9. Assess Limitations. Identify missing evidence, method errors, unsupported conditions, and redaction effects.
• 10. State Calibrated Confidence. Connect each conclusion to observations, support, alternatives, and an explicit boundary.

Documentation and reporting

Keep a claim-to-evidence table alongside the source inventory. Record each tool version, transformation, test, result, and failure. Report direct observations separately from inference and attribution. Use the published SFIF as an organizing framework and the Artifact Report format for tested artifact behavior.

Common failures and limitations

A plausible story can conceal missing branches, circular support, untested clock conversions, or a detector used outside its evaluation conditions. Repeating a test does not make it valid, and adding copies of the same record does not make its origin independent. A reliable examination can end with an unresolved proposition.

A hypothetical agent-file question

An agent transcript says “report saved.” A tool record shows an attempted write. The acquired directory contains no file, but collection occurred after a cleanup job. The supported finding is an attempted write with incomplete evidence of persistence. A historical filesystem record or independent destination log could distinguish creation followed by deletion from a failed write. This is a teaching scenario, not a real incident.

Continue with specific methods

• Acquiring AI-related evidence
• Preserving synthetic evidence
• Examining provenance and origin claims
• Interpreting metadata and system artifacts
• Making bounded attribution claims
• Reconstructing events and workflows
• Seeking independent corroboration
• Writing calibrated confidence statements
• Reporting synthetic-evidence findings
• Documenting chain of custody
• AI detection limitations

Terms used in this method

• Competing hypothesis
• Reproducibility
• Confidence statement

How do you turn a broad question into a test?

Replace “Was AI involved?” with a specific proposition such as whether a named agent task produced an acquired file. Identify the sources that can test creation, timing, and content separately, then preserve them before interpreting tool results.

Compare source-system records, provenance, object measurements, and independent effects. The final answer can support one part of the claim while leaving another unresolved; keep those boundaries in the main conclusion.