Scanning the original archive

The archive began with a practical question: what files survived across years of backups, and where were all their copies? The original guide, written in February 2026, described roughly 3.5 million files and 1.47 TB of scanned data. Those figures explain the scale of the discovery problem; they are historical observations rather than a current dashboard reading.

The Go scanner separates discovery from the application that interprets it. It walks a selected filesystem tree, records paths and basic metadata, optionally hashes contents, and extracts available EXIF. Rails subsequently builds browsable records, preservation state, and human decisions from that inventory. Scanner source (scanner/main.go), application reader (archive-browser/app/models/scanner/file_entry.rb).

A bounded discovery pipeline

One directory walker discovers entries. For each eligible file, it acquires a semaphore permit and starts a processing goroutine. The default is eight workers. A buffered channel carries completed records to a separate database writer, which collects batches of 1,000 by default and sends them through PostgreSQL COPY.

This arrangement overlaps file processing and database writes while bounding concurrency. When all worker permits are occupied, the walker waits: backpressure is part of the design. It avoids creating millions of simultaneous goroutines and makes disk throughput a practical tuning constraint. COPY writes ordinary latitude/longitude columns alongside the other observations; the implementation does not attempt to ingest a PostGIS geography value directly. Traversal and ingestion implementation (scanner/main.go).

What each observation contains

A record includes the supplied path, filename, lowercase extension, byte size and modification time. The scanner streams nonempty files through xxHash64 and stores a 16-character hexadecimal fingerprint. Hashing is optional, and a failed hash read leaves that field empty. Empty files are not hashed by this implementation.

Extension rules assign categories such as photo, video, document, code and configuration, with textual confidence labels. These are format clues. A JPEG extension does not establish whether an image is a family photograph, a screenshot or a scanned invoice. Extensionless files are marked unknown and needing review; an unrecognized nonempty extension can remain uncategorized. A low-confidence mapped extension is not automatically a review flag in this source.

EXIF processing attempts camera information, exposure settings, capture date, dimensions and GPS for selected extensions. Dimensions are accepted only within a positive, bounded range. An extension appearing in the EXIF list means extraction is attempted, not that every file in that format will yield metadata. Later file analysis and photo analysis add different kinds of evidence.

Resume and exclusion behavior

At startup, the scanner loads existing database paths into memory. An already recorded path is skipped. This saves repeated work after an interruption, but it does not detect changed bytes at an existing path, refresh old categories or track deletion. A second scan is therefore not a general synchronization mechanism.

Directory exclusions use exact names and skip entire subtrees, including selected system, dependency and Git directories. Sensitive flags use case-sensitive substring checks against paths. Both policies affect what can be discovered; neither proves that omitted files are unimportant or that every sensitive file is caught. The walker does not recurse through symbolic-link directories, although a file open may resolve a link target.

The scanner emits file rows, not directory rows. The application derives ancestor directories and rolls their statistics upward. Directory population (archive-browser/lib/populators/directory_populator.rb).

Reading a completed scan

The source exposes -source, -db, -workers, -batch, -skip-hash, -skip-exif and -verbose. Its displayed processed count measures records sent to the writer, not confirmed durable inserts: failed batches are logged and cleared without a built-in retry loop. Successful completion therefore needs reconciliation against stored rows and errors, rather than trust in the spinner alone.

This discovery layer supplies the foundation for content identity. It records evidence without deciding what deserves preservation or how a recovered project should be rebuilt.

Implementation notes refer to repository sources reviewed at e76e74c on October 13, 2026.