Stop treating HTTP file sources with no cache headers as always-changed

[miharp]

Fixes #581.

AI disclosure: this contribution (investigation, code, tests, docs, and this description) was produced with substantial assistance from Claude (Anthropic), per the project's AI usage policy. I've reviewed it and take full responsibility for it.

Summary

A file resource with an http(s) source pointed at a server that sends no Last-Modified, ETag, or checksum header at all — or sends an ETag that's never consulted because checksum => etag wasn't requested — is treated as "changed" on every single run, forever. Puppet::FileServing::HttpMetadata fabricates the current wall-clock time as a fake mtime whenever headers give nothing usable, and since :mtime is the last resort in the checksum fallback chain, the comparison is guaranteed to conclude "changed" independent of whether the remote content actually changed. This fires notify/subscribe (e.g. restarting a service) on every Puppet run against origins like Artifactory behind a caching proxy, or plain raw.githubusercontent.com sources.

See #581 for the full root-cause writeup, real-world reproduction, and a link to a self-contained Docker repro.

What changed

1. lib/puppet/file_serving/http_metadata.rb — no more fabricated Time.now. No usable header now resolves to :none ("unverifiable, assume unchanged" — the same semantics checksum => none already has elsewhere) instead of a checksum guaranteed to differ every run. Adds #verify!, called by the terminus below to override that :none verdict with a real, earned digest.
2. lib/puppet/indirector/file_metadata/http.rb — when metadata resolves to :none and the resource asked for a real digest (the default, or any explicit type other than mtime/ctime/none), downloads the body once and hashes it as it streams by, discarding the bytes. If a rewrite turns out to be needed (content actually changed), the normal content fetch downloads it again — one extra request only in the uncommon changed case, and no open file handles for a long-running agent to accumulate in the common unchanged case. A failed verification GET is a real failure, not "unchanged": a non-success response returns nil (not found, same as every other failure branch in #find — which also preserves next-source fallback for source => [...] arrays), and a raised network error propagates, exactly like the existing HEAD request already behaves.
3. lib/puppet/type/file/source.rb — docs only. Brings the source parameter docs in line with actual behavior: they still described the old, buggy Last-Modified-or-nothing fallback as intended, and never documented checksum => etag at all (which shipped in Feature: file etag support #329).
4. lib/puppet/type/file/checksum.rb — documents checksum => etag (same pre-existing gap), and fixes the pre-existing :etag fallback (also from Feature: file etag support #329) that hardcoded :md5 when no ETag-derived type resolves — which breaks under FIPS. Both that fallback and the terminus's equivalent now use Puppet[:digest_algorithm], which is always FIPS-safe, and — since the local and remote sides of the comparison must agree on an algorithm — fixing one without the other would have made them diverge.

Explicitly not fixed: a server whose Last-Modified header itself changes on every request despite unchanged content (e.g. a dynamic backend behind a caching proxy). That still resolves to :mtime, not :none, so the verification path never triggers — the gate only earns a checksum when there's no header to trust, not when there's an untrustworthy one. Distinguishing the two would mean paying for a full download on every apply for any HTTP source with a Last-Modified header at all, including the well-behaved majority where it's perfectly reliable.

Known minor cost: a resource combining checksum_value with a headerless http(s) source still pays for the verification download even though the checksum_value comparison doesn't use the metadata checksum — the terminus only sees checksum_type in the request options. Rare combination; threading checksum_value through the indirection felt like scope creep for this fix.

Testing

Unit specs updated/added in spec/unit/file_serving/http_metadata_spec.rb and spec/unit/indirector/file_metadata/http_spec.rb, including coverage for: earning a checksum via GET, the mtime/ctime/none/unspecified opt-outs (asserted by omitting the GET stub, so WebMock fails the example if a request sneaks through), a failed verification GET returning nil, a raised network error propagating, and the FIPS-safe etag fallback.

$ bundle exec rspec spec/unit/file_serving/http_metadata_spec.rb \
    spec/unit/indirector/file_metadata/http_spec.rb \
    spec/unit/type/file/source_spec.rb \
    spec/unit/type/file/content_spec.rb
272 examples, 0 failures, 1 pending (pre-existing, unrelated)

$ bundle exec rspec spec/unit/file_serving/ spec/unit/type/file_spec.rb spec/unit/type/file/
1083 examples, 0 failures, 10 pending (all pre-existing, unrelated)

For an end-to-end check beyond mocks, see miharp/openvox#1 (not for merge) — a Docker harness that runs real puppet apply against a hand-rolled HTTP server able to withhold, churn, or fail cache-validation responses on demand, asserting both notify behavior and exact --detailed-exitcodes exit codes, with console output from both an unpatched run (bug reproduces) and this branch (resolved). Its scenario 7 proves the failure-propagation property end-to-end: HEAD succeeding with no validators followed by a failing verification GET produces a failed run (exit 4), not a silently clean one.

Opening as draft to get early feedback on the fix boundary described above (only :none triggers verification, not an untrustworthy-but-present header) before final review.

[miharp]

Should this be labeled backwards-incompatible? The behavior change here means a file resource with an http(s) source and no usable cache-validation headers, previously reported as "changed" (and thus firing notify/subscribe) on every run, will now be treated as unchanged unless content actually differs. That's a real behavior change for anyone relying on the old always-changed side effect, similar to prior bug-fix PRs like #170 that got this label.