refactor(aggregator): deterministic snapshots archives

[Alenar]

Content

This PR harden the archives building of the mithril-aggregator in order to makes them bytes identical for a given content.

It extract the FileArchiver api from the Aggregator into introduces a new library, mithril-file-archiver, with additional changes to ensure deterministic, byte-stable .tar.zst archives creations.

Main Changes

New Library Added:

• Introduces mithril-file-archiver as a standalone library for generating deterministic archives. Like the previous api, it supports archiving files, directories, or serializable data with compression.

  • Includes support for configurable Zstandard compression parameters.
  • Ensures reproducibility and determinism across platforms.
  • Refactored the appender system around a ArchiveEntry enum that hold one entry of the supported type (file, directory, or data), ArchiveEntry are Ord and the existing appender now return them instead of directly writting to the tar archive, this allows AppenderChain to be deterministic as long as there's no entries overlap between the chained appenders.
  • Comprehensive tests for functionality, compatibility, cross-platform reproducibility, and golden master validation were implemented in the new library.

• Existing File Archiver Logic Removed: The previously internal file_archiver and associated modules (appender, tools, etc.) in the mithril-aggregator folder have been moved to the new library.

  • Updated mithril-aggregator to integrate mithril-file-archiver for tasks like snapshot generation and ancillary file handling.
  • Adjusted tests to use the new library's provided abstractions and utilities (e.g., FileArchiver::new_with_default_parameters and FileArchiveTestExtension). - Existing tests in mithril-aggregator were adapted accordingly.
  • Removed direct dependencies on zstd and tar in mithril-aggregator.

• Documentation:

  • Enhanced documentation to describe mithril-file-archiver, its API, and usage constraints.

• Other:

  • follow_symlinks support is now disabled in the FileArchiver (as recommended in their doc)
  • some unused dependencies were removed from the aggregator: tempfile, and mithril-test-http-server

How deterministic archives were achieved

• Changes to the tar header:
  • switch to HeaderMode::Deterministic: this set the mtime to a fix value (timestamp 1153704088, date: Jul 23, 2006), the guid and gid to 0, and the permissions to 644 for files and 755 for directories (see the code here)
  • sparse file support is disabled as it's not consistent across systems and the zstandard will compress the sparse part itself anyway
• A normalization path is now run on the AppenderEntries entries list:
  • removal of trailing / from directory names (or \ on Windows)
  • removal of leading ./ from path
  • the entries list is now sorted using rust default sort
• changes to AppenderData metadata:
  • a deterministic mtime is set instead of the current time: randomly choose to 1767225600 (2026-01-01 00:00:00 UTC)
• Removal of AppenderDirAll: the order of the files and folder is not stable across systems, if needed, it could be re-created as a constructor of AppenderEntry (from_directory) which would do the same job of recursively adding all entries but would then leverage its normalization pass
• Reworking the appender to output a list of sortable ArchiveEntry allowed to add commutativity and associativity to the ChainAppender, as long as there's no overlapping path between chained appenders, else the rightmost take priority

Pre-submit checklist

• Branch
  • Tests are provided (if possible)
  • Crates versions are updated (if relevant)
  • CHANGELOG file is updated (if relevant)
  • Commit sequence broadly makes sense
  • Key commits have useful messages
• PR
  • All check jobs of the CI have succeeded
  • Self-reviewed the diff
  • Useful pull request description
  • Reviewer requested
• Documentation
  • Update README file (if relevant)
  • No new TODOs introduced

Issue(s)

Relates to #3435

[github-actions]

Test Results

     5 files  ± 0     221 suites  +12   1h 0m 30s ⏱️ - 1h 25m 6s
 3 408 tests  - 42   3 408 ✅  - 42  0 💤 ±0  0 ❌ ±0 
11 383 runs  +70  11 383 ✅ +70  0 💤 ±0  0 ❌ ±0 

Results for commit 16dfed5. ± Comparison against base commit 58b575a.

This pull request removes 109 and adds 67 tests. Note that renamed tests count towards both.

mithril-aggregator ‑ tools::file_archiver::api::tests::compute_size_of_uncompressed_data_and_archive
mithril-aggregator ‑ tools::file_archiver::api::tests::overwrite_already_existing_archive_when_archiving_succeed
mithril-aggregator ‑ tools::file_archiver::api::tests::should_create_a_valid_archive_with_zstandard_compression
mithril-aggregator ‑ tools::file_archiver::api::tests::should_delete_tmp_file_in_target_directory_if_archiving_fail
mithril-aggregator ‑ tools::file_archiver::api::tests::should_not_delete_an_already_existing_archive_with_same_name_if_archiving_fail
mithril-aggregator ‑ tools::file_archiver::appender::tests::appender_data::append_serializable_json
mithril-aggregator ‑ tools::file_archiver::appender::tests::appender_data::appended_entry_have_read_write_permissions_and_time_metadata
mithril-aggregator ‑ tools::file_archiver::appender::tests::appender_data::compute_uncompressed_size
mithril-aggregator ‑ tools::file_archiver::appender::tests::appender_dir_all::compute_uncompressed_size
mithril-aggregator ‑ tools::file_archiver::appender::tests::appender_entries::can_append_duplicate_files_and_directories
…

mithril-file-archiver ‑ api::tests::compute_size_of_uncompressed_data_and_archive
mithril-file-archiver ‑ api::tests::overwrite_already_existing_archive_when_archiving_succeed
mithril-file-archiver ‑ api::tests::should_create_a_valid_archive_with_zstandard_compression
mithril-file-archiver ‑ api::tests::should_delete_tmp_file_in_target_directory_if_archiving_fail
mithril-file-archiver ‑ api::tests::should_not_delete_an_already_existing_archive_with_same_name_if_archiving_fail
mithril-file-archiver ‑ appender::tests::appender_data::append_serializable_json
mithril-file-archiver ‑ appender::tests::appender_data::appended_entry_have_read_write_permissions_and_fixed_time_metadata
mithril-file-archiver ‑ appender::tests::appender_data::compute_uncompressed_size
mithril-file-archiver ‑ appender::tests::appender_entries::can_append_duplicate_files_and_directories
mithril-file-archiver ‑ appender::tests::appender_entries::compute_uncompressed_size_of_its_paths
…

♻️ This comment has been updated with latest results.