Content identity and duplicates
A path tells the archive where a file was found. A content hash gives it a way to group repeated bytes. Avichive retains both: scanner rows represent occurrences, while UniqueFile represents content shared by those occurrences. This lets the application preserve one original without erasing the history of its locations. UniqueFile model (archive-browser/app/models/unique_file.rb).
Consider three invented observations:
| Original path | Content fingerprint | Interpretation |
|---|---|---|
/Archive/Photos/beach.jpg |
0123456789abcdef |
One occurrence |
/Backup/OldLaptop/beach-copy.jpg |
0123456789abcdef |
Another occurrence of the same indexed content |
/Archive/Photos/beach-edited.jpg |
fedcba9876543210 |
Separate content |
The first two can share a preserved original and content-level analysis. The edited image remains separate even if it looks almost identical. Exact-content grouping does not establish visual similarity, chronological version order, or which edit a person prefers.
How the content record is built
The population task selects distinct non-null scanner hashes, loads their occurrences in batches, and creates a UniqueFile for each hash. Occurrences without hashes remain in the scanner catalog but do not gain this content identity. Skipped hashing, read failures, and empty files can leave hashes absent. The content record’s fields include byte size, category, extension, occurrence count, canonical scanner entry and canonical path. Population implementation (archive-browser/lib/populators/unique_file_populator.rb).
The initial canonical choice is the shallowest path, with the lowest scanner ID breaking ties. This is a stable heuristic for choosing an occurrence, not a judgment about historical authenticity or better organization. A shallow temporary directory can still win. The content record’s all_occurrences method retrieves every scanner row sharing the hash, so the chosen path is not the only provenance retained.
Repeated population uses create-or-find behavior. Existing content records are not automatically refreshed with new occurrence counts or canonical choices by the creation block. That distinction matters when reasoning about later scans: avoiding duplicate records is different from reconciling an evolving inventory.
Identity, integrity and availability
The scanner uses xxHash64, a fast non-cryptographic fingerprint. The application groups on that value; it does not perform exhaustive byte comparison during population. A hash match is useful operational evidence, not a mathematical guarantee that collisions cannot occur.
Preservation adds checks around the actual bytes. The original uploader verifies a readable source against the expected hash and can try alternate occurrences when the canonical path is unavailable. The cloud reader also offers an explicit full-download hash verification operation. Ordinary materialization downloads bounded content for processing; it does not automatically perform that hash check. Source hashes, storage-specific size checks, and upload checks address different failure modes. Original storage service (archive-browser/app/services/archive_originals.rb).
canonical_path remains a local provenance field. The preserved original’s URI is separate, so losing access to the original drive need not erase the catalog’s record of where the content came from. Conversely, a database entry alone does not prove an original was uploaded successfully.
Savings without deleting history
For a content record with size s and n occurrences, the model estimates duplicate bytes as s × (n − 1). This is the extra indexed storage occupied by copies beyond one. It is not a report of disk space actually freed: Avichive can deduplicate cloud preservation while leaving local files untouched.
The February guide reported about 735 GB of duplicate storage. That discovery motivated the design, but today’s reader should use dated capability evidence rather than treat the original savings estimate as a current measurement.
A keep/skip curation decision, a backup status and an exact-duplicate group answer different questions. Skip records an archival decision; it does not itself delete an occurrence. Multiple copies can carry useful folder context even when their bytes are identical. File analysis and review explain those decisions; cloud preservation explains how one original serves the content identity.
Implementation notes refer to repository sources reviewed at e76e74c on October 13, 2026.