Read raw images through the shared raw_image module

.github/workflows/python_lint.yml

@@ -83,8 +83,10 @@ jobs:
       # scripts/vendor/ holds code copied verbatim from another repository, so it
       # drifts in two directions and both are silent: an edit here looks like a fix
       # until the next re-vendor reverts it, and an upstream release leaves this copy
-      # quietly old. Unconditional rather than gated on changed Python files, because
-      # the recorded hashes live in a .json and a stale record is the same defect.
+      # quietly old. The manifest records each copy's hash and upstream commit; this
+      # checks the hash and fetches the upstream file at that commit. Unconditional
+      # rather than gated on changed Python files, because the recorded hashes live in
+      # a .json and a stale record is the same defect.
       - name: Check vendored files match what was vendored
         run: python admin/scripts/check_vendored.py
 

.github/workflows/test_builds.yml

@@ -73,6 +73,23 @@ jobs:
           dist_build\vleapp.exe -t fs -i smoke_in -o smoke_out
           if ($LASTEXITCODE -ne 0) { exit 1 }
 
+      # A raw image is read through the vendored reader imported as a module, which
+      # is the path a frozen build is most likely to break: an earlier design ran the
+      # reader as a subprocess through sys.executable, and in a bundle that is the
+      # tool itself, so -t raw could never work frozen and nothing smoke-tested it.
+      - name: Smoke test the frozen CLI on a raw image
+        shell: pwsh
+        run: |
+          New-Item -ItemType Directory -Path smoke_raw_out | Out-Null
+          python -c "import gzip, shutil; shutil.copyfileobj(gzip.open('admin/test/data/raw_images/ntfs-fixture.img.gz'), open('smoke_in/ntfs-fixture.img', 'wb'))"
+          dist_build\vleapp.exe -t raw -i smoke_in\ntfs-fixture.img -o smoke_raw_out
+          if ($LASTEXITCODE -ne 0) { exit 1 }
+          $log = Get-ChildItem -Path smoke_raw_out -Recurse -Filter "Screen_Output.html" | Select-Object -First 1
+          if (-not $log) { Write-Error "no run log was written"; exit 1 }
+          if (-not (Select-String -Path $log.FullName -Pattern "walked lba0: 4" -Quiet)) {
+            Write-Error "the frozen build did not walk the NTFS fixture"; exit 1
+          }
+
       - name: Package artifact
         run: |
           New-Item -ItemType Directory -Path artifact | Out-Null
@@ -140,6 +157,15 @@ jobs:
           echo placeholder > smoke_in/placeholder.txt
           dist_build/vleapp -t fs -i smoke_in -o smoke_out
 
+      # Same on Linux: the frozen build has to read a raw image through the vendored
+      # reader imported as a module, and the run log has to show the walk.
+      - name: Smoke test the frozen CLI on a raw image
+        run: |
+          mkdir smoke_raw_out
+          python -c "import gzip, shutil; shutil.copyfileobj(gzip.open('admin/test/data/raw_images/ntfs-fixture.img.gz'), open('smoke_in/ntfs-fixture.img', 'wb'))"
+          dist_build/vleapp -t raw -i smoke_in/ntfs-fixture.img -o smoke_raw_out
+          grep -rq "walked lba0: 4" smoke_raw_out --include="Screen_Output.html" || { echo "the frozen build did not walk the NTFS fixture"; exit 1; }
+
       - name: Package artifact
         run: |
           mkdir artifact

README.md

@@ -31,10 +31,14 @@ https://twitter.com/TroySchnack/status/1266085323651444736?s=19
 $ python vleapp.py -t <fs | zip | tar | gz | file | raw | iva> -i <path_to_extraction> -o <path_for_report_output>
 ```
 
-`raw` reads a disk image, or an EnCase/EWF `.E01` acquisition and the segments beside
-it, without mounting and without administrator rights: its QNX6, QNX4, ext2/3/4, FAT32,
-exFAT, NTFS, HFS+ or APFS volumes and QNX IFS boot images are read directly. `iva` reads a Berla iVe
-export as it stands.
+`raw` reads a disk image (`.img`, `.dd`, `.bin`, or any numbered `.001` segment of
+a split set), or an EnCase/EWF `.E01` acquisition and the segments beside it, in
+place: no mounting and no administrator rights. Its NTFS, FAT32, exFAT, ext2/3/4,
+HFS+, APFS, QNX6, QNX4, ETFS, EFS and QNX IFS volumes are searched directly, and
+only the files an artifact asks for are read out of the image. The GUI picks
+`raw` on its own for those extensions. See `admin/docs/raw_image_input.md`.
+`iva` reads a Berla iVe export as it stands: the raw image inside it is read the
+same way, and the export's `Vehicle.json` is reported beside the vehicle data.
 
 ### GUI
 

admin/docs/raw_image_input.md

@@ -1,79 +1,122 @@
-# Raw image input, and what it changed in core
-
-For maintainers, and for whoever ports this to the other LEAPP cores.
-
-`-t raw` takes a raw disk image and reads its QNX6, QNX4, ETFS, EFS, ext2/3/4, FAT32, exFAT,
-NTFS, HFS+ and APFS volumes, and QNX IFS boot images, without
-mounting it and without administrator rights. Vehicle head units need it: the Ford
-Sync units are QNX6, the BMW MGU is ext4, and no filesystem type The Sleuth Kit
-supports can walk QNX6 at all, so those images were previously unreadable here.
-
-## The whole core surface
-
-Two files. Nothing existing was modified in either.
-
-**`vleapp.py`**, three edits totalling 8 lines:
-
-    choices=['fs', 'tar', 'zip', 'gz', 'file', 'raw']   # one enum entry
-    ...one sentence added to the -t help text...
-    elif extracttype == 'raw':                          # one dispatch branch
-        seeker = FileSeekerRaw(input_path, out_params.data_folder)
-
-**`scripts/search_files.py`**, one new class, 66 added lines, 0 changed:
-
-    class FileSeekerRaw(FileSeekerZip):
-
-Everything else in both pull requests is outside core: `scripts/vendor/` holds a
-verbatim copy of the reader, and `admin/scripts/check_vendored.py`, the
-`lint_changed.py` exclusion and the CI step are tooling.
-
-## Why it subclasses rather than walks
-
-The vendored reader exposes usable walkers (`Qnx6Walker`, `ExtWalker`, both with
-`listdir` / `entry` / `read_file`), so a seeker that walked the image lazily and
-staged only matched files was the obvious design and is not what this does.
-
-Deciding *which* partitions hold *which* filesystem, and which superblock
-generation is current, is not behind a callable seam in that tool; it is threaded
-through its command line flow. Reimplementing it here would copy non-trivial logic
-that then drifts from the vendored file, which is the exact failure the vendoring
-hash guard exists to catch.
-
-So `FileSeekerRaw` runs the vendored tool to produce a zip, then calls
-`FileSeekerZip.__init__` on it. Every staging, matching, disambiguation and
-timestamp decision stays on the path the zip seeker already runs on every zip
-input. The cost is an up-front extraction instead of lazy reads.
-
-**The seam to improve later** is exactly this: if the reader grows a callable
-"enumerate the volumes in this image" entry point, `FileSeekerRaw` can stop
-shelling out and stage lazily, and the class is the only thing that changes.
-
-## Porting it to another core
-
-The core part is small and generic. `FileSeekerRaw` references nothing
-VLEAPP-specific: it uses `logfunc`, `FileSeekerZip` and the vendored path, all of
-which exist or have equivalents in every core. Copying the class and the three
-`vleapp.py` edits is the whole job.
-
-What is worth deciding once, rather than five times:
-
-- **Where the vendored reader lives.** Here it is `scripts/vendor/`, resolved
-  relative to `search_files.py`. If the cores consolidate, this should be one
-  shared location, not five copies of a copy.
-- **Whether the staged zip is kept.** It goes to a temp directory and `cleanup()`
-  removes it. For a large volume the extraction is the slow part of the run, so
-  keeping it would make re-runs cheap. Deliberately not done yet, because the
-  report folder is shared and silently growing it by gigabytes is worse.
-- **Whether raw should be auto-detected** rather than selected with `-t`. The
-  reader can already tell whether an image holds anything it can read, so
-  detection is possible; it is a separate change and a shared one.
-
-## What was measured
-
-The same image was run three ways: through the vendor's own extracted file set,
-through a zip made by hand with the vendored reader, and through `-t raw`. All six
-artifacts common to the branches returned identical row counts, and two of the
-underlying stores were byte-identical by SHA-256 between the first two routes.
-
-`-t raw` found four QNX6 volumes on the test image, one more than the vendor's
-export carried extracted files for.
+# Raw image input
+
+For maintainers, and shared by iLEAPP, ALEAPP, RLEAPP, VLEAPP and DLEAPP.
+
+`-t raw` takes a disk image (`.img`, `.dd`, `.bin`, or any numbered `.001`
+segment of a split set) or an EnCase/EWF acquisition (`.E01` and its segments)
+and reads it in place: no mounting, no administrator rights, and no copy of the
+image or of its files anywhere but the files an artifact asks for. Its NTFS,
+FAT32, exFAT, ext2/3/4, HFS+, APFS, QNX6, QNX4, ETFS, EFS and QNX IFS volumes
+are searched directly.
+
+## Where the pieces are
+
+- `scripts/vendor/qnxprobe.py` and `scripts/vendor/ewfprobe.py` are the reader,
+  copied verbatim from their repositories and guarded by
+  `admin/scripts/check_vendored.py` (see `scripts/vendor/README.md`). Fix them
+  upstream and re-vendor; an edit here is reverted by the next sync.
+- `scripts/raw_image.py` holds everything of ours: `FileSeekerRaw`, the constants
+  the GUI uses to recognise an image by extension, and `split_image_sibling`.
+  **This file is byte-identical in all five cores.** Change it in one, land the
+  same bytes in the other four in the same round, and let the parity scanner in
+  leapps-org/leapps-parity confirm they match.
+- The entry point adds one `-t` choice, one dispatch branch and a `finally` that
+  calls `seeker.cleanup()` on every exit. The GUI maps the conventional image
+  extensions onto `raw` and lists them in the file dialog.
+- `admin/test/scripts/test_raw_image_seeker.py` checks staged bytes against
+  independent hash lists over the fixtures in `admin/test/data/raw_images/`,
+  including an E01 set and a split set built at test time.
+
+## How it reads
+
+`FileSeekerRaw` opens the image through the reader's `open_image()` (which joins
+a split set and reads an EWF set), asks `volumes()` for every volume the reader's
+own report would name, walks each readable volume once for its directory tree,
+and offers the run a member list in the shape the zip seeker offers: one name per
+file and one per directory (with a trailing slash), each prefixed by the volume's
+extraction name, `p<partition>_lba<start>[_<label>]` or `lba0` for a bare volume.
+`search()` matches a pattern against that list with the same `fnmatch` rules as
+the zip seeker and reads only the matches out of the image, so the cost of a run
+is the walk plus the files the artifacts wanted.
+
+Measured on a 238.5 GiB Windows acquisition (PC-MUS-001, 253,032 files): the
+walk took 9 seconds, a pattern resolves in under half a second, and DLEAPP's
+patterns select 694 MiB of the 103.3 GiB of live files. Staging every file to a
+temporary folder first, the design this replaced, would have written those 103 GiB
+before the first artifact ran.
+
+The run log names every volume with its filesystem and size, says which it
+cannot read and why, and warns when the image is shorter than the volumes its
+partition table describes, which is what a lone first segment of a split set
+looks like. A file that runs past the end of the image is named in the log and
+not staged, because a truncated database parses as a smaller one rather than as
+an error.
+
+## Times
+
+NTFS, ext, HFS+ and APFS store instants, and those reach the staged copy's
+mtime and the `FileInfo` the run records. FAT32 and exFAT store a wall-clock
+reading with no zone; the reader hands those back as text. The staged copy's
+mtime is set from that reading in the machine's local zone, exactly as the zip
+seeker sets it from a member's DOS stamp, and the `FileInfo` records no instant
+for it, so no report field carries a zone the evidence never had.
+
+## What it does not do
+
+- It reads live files. Deleted records the reader can recover on NTFS, FAT32
+  and exFAT are not staged.
+- It does not re-root a bare partition image. A raw image of an Android
+  `userdata` partition has `data/`, `media/` and `system/` at its root rather
+  than under `data/`, and an iOS Data volume has `mobile/` and `containers/`
+  rather than `private/var/`. Patterns that begin with `*/` match either way;
+  the few that spell out `private/var/` or `data/media/` do not match on a
+  bare partition image. Two measured consequences: on a bare Android `userdata`
+  image the storage-view dedupe recognises only the `data/data`, `data/user/N`
+  and `data_mirror` spellings, so a partition read at its own root is not
+  collapsed and per-app row counts multiply; on a bare iOS Data volume the
+  `iosfilesystemevents` family and `diagnosticlogdevents` find nothing. A
+  full-disk or full-`/data` image carries the expected root and neither happens.
+- F2FS, the filesystem most current Android userdata partitions use, is not
+  one the reader walks; such a volume is listed as not recognised.
+- An encrypted volume (Android file-based encryption, iOS data protection,
+  FileVault, BitLocker) reads, but its names or contents are ciphertext.
+
+## Comparing a raw run against a zip run
+
+The same extraction read as a zip and as a raw image of the same file tree does
+not produce byte-identical reports, and the differences are in the input format
+and in a few artifacts, not in what the seeker staged. A raw run over a full-disk
+or full-`/data` image stages every file an artifact asked for, so the run log's
+`Not staged` count is zero. The reports still differ in three ways worth knowing,
+none of them a file the seeker got wrong:
+
+- Timestamps. A zip member carries an MS-DOS stamp: two-second granularity, no
+  sub-second, no zone. A filesystem carries the real instant. So a column that
+  surfaces a staged file's own time reads a second or two apart between the two
+  routes, and the raw route is the more faithful of the two.
+- First-match order. A few artifacts read one file out of several equivalent
+  copies and take the first the seeker returns. The zip, tar and raw seekers each
+  list a directory in their own order, so which copy wins can differ. On Android
+  `emulatedSmeta` reads one user's `external.db`, `installedappsGass` labels its
+  source by the view it read, and `walstrings` numbers its rows by arrival. This
+  is a property of those artifacts and shows between the zip and tar routes too.
+- Companion files beside the zip. Some artifacts read a file a tool exported next
+  to the acquisition rather than inside it. iOS keychain artifacts read a
+  `<udid>_keychain.plist` sitting beside the zip on disk, found only when the
+  input is that zip. A raw image is the filesystem itself and carries no such
+  companion, so those artifacts report nothing from it.
+
+## Frozen builds
+
+The reader is imported as a module, so a PyInstaller build carries it like any
+other module; nothing is spawned. `test_builds.yml` runs the frozen CLI with
+`-t raw` on the NTFS fixture and requires the run log to show the walk. An
+earlier design that ran the reader as a subprocess through `sys.executable`
+could not work frozen, because in a bundle that is the tool itself.
+
+## The corpus validator
+
+`admin/scripts/validate_sample_data.py --run` runs a registry entry through
+`-t raw` when the entry carries `"input_type": "raw"`. A raw image is never
+guessed from its extension, because a registry can hold `.bin` files that are
+readable disk images and `.bin` files that are chip-level dumps no walker opens.