[Feature Request] Hash Verify -- opening multiple hash files

[a-raccoon]

One issue that I encounter regularly is trying to verify multiple hash files, all at once, which creates a huge drag with the harddisk attempting to read multiple files in parallel. There are a few solutions I would like to propose, if any of them would be of interest to implement.

• When a hash verify window is already open, then reuse it by amending additional hash file contents to it and continue processing in sequential order.
• Spawn multiple hash verify windows, like normal, but pause them until the previous windows are done.
• Add options for Joining (and Splitting) hash files, so I can join a few thousand together into one hash file, and then just verify it as normal. Joining and Splitting would have to take the *relative\folder\paths\ into consideration from each of the joined or split hash files.

Thoughts?

[tERyceNzAchE]

I'm in favour of option # 2.

[ZPNRG]

I would probably vote for option #2 where the hash checksum files are queued as they are executed.

WIth the Gurnec HashCheck project, I brought up the same issue back on 05-17-2017. See gurnec#29.

It should be noted that the fundamental issue applies to both creating and checking multiple hash checksum files at the same time, not just checking them,

Below is what I said back in 2017:

"Hi.

I made a comment in the thread for Issue #13 (Queue multiple file check/creation enhancement), but the bulk of the issue there seemed to surround having individual hash checksum files created for each file instead of queue-ing in the sense I am thinking about.

I would like to have just the simple queue-ing aspect. Many times, I want to check a number of hash checksum files (.sha1 or .sha256) that are large or are for large files, like Windows ISOs. If I just double-click them one after another and run them to have HashCheck check the hash checksum files, then the HD (spinning disks) is thrashed. So, I have to run them manually 1 by 1 after each finishes. It would be nice if HashCheck would queue the operations so that after a currently running hash task finishes (creating a hash checksum file or checking a hash checksum file), then it will process the next one that I had started, and keep creating or checking one by one until all of the operations I had started have completed.

To be clear, this would ideally work for creating multiple hash checksum files one after another or double-clicking on existing hash checksum files one after another to be checked. Either way, I would like to start the process for all of the hash checksum files I want created or checked, but then just have HashCheck process them serially 1 by 1 and not have them all going (or trying to work) in parallel (at the same time). Like I said, even fast spinning drives are hammered if the the hash checksums being created or checked are for large files or there are lots of decent sized files and two or more of these hash checksum files being created or checked are processing at the same time. Therefore, being able to start the processes (creating or checking) and have HashCheck queue them and only do one at a time would be nice.

Let me know if there are any questions.

Thank you for the work on HashCheck.

-ZPNRG"

[a-raccoon]

This is mainly why I proposed 3 competing ideas, as they're all compatible with each other and could be simultaneously implemented. The ability to Join multiple checksum files together would itself solve the issue of verifying multiple files, allowing the user to combine them before verifying the combined file. Otherwise, allowing the user to reuse or spawn-new verification windows, solves the issue of window spam, where checking 1000 hash files would spawn 1000 individual windows. Having them smartly pause and wait instead of checking in parallel would prevent 1000 files from being read and verified at the same time, wearing holes into your harddisk.

[ZPNRG]

@idrassi, have you thought about this feature request? Creating and/or verifying multiple digests at the same time in some type of queued manner is still one of my personal biggest needs for HashCheck due to how many digests I create and/or verify at times.

[ZPNRG]

@idrassi , glad to see you jump in a close some issues. Thank you very much for the huge update!

Would love to see some work done to address this issue (#5).

(Side note: Also glad to see you get the needed help with VeraCrypt recently.)

[idrassi]

@ZPNRG Thank you for the feedback.

Yes, remaining issues will be addressed. I try to balance all my activities. Next HashCheck version should come in 2 weeks. Now priority is get next VeraCrypt version released.

[nutpantz]

+1 this issue. i create a LOT of checksum files for my backups. and i verify the files every few months ( think 30 tb of files)
queuing the creation and verification of checksums is badly needed.( at least by me)

i vote for a window for each hashfile being queued and that window pausing until the previous hash job is done then switching the next job being processed in its own window.

this way if you have 20 hash job ( and each job could be 1 file or thousands) to create or verify, you can come back and quickly see how many jobs were completed, how many are left or if there are errors and in which job.
( having to scroll through 15 jobs with 5000 files in one window is a huge pain)
yes you may have 20 windows or more open, but windows itself will collate those windows and organize them in the taskbar when hidden so that is not really an issue anymore)

this is very similar to how "Multipar" works.
https://github.com/Yutaka-Sawada/MultiPar

and i can say that multipar is a delight to work with on large projects with thousands of files.
( but Parchive is an erasure code system that produces par files for checksum verification of data integrity, with the capability to perform data recovery operations that can repair or regenerate corrupted or missing data. NOT primary used for checksum hashes)

( and think you for the work on this app, i love it and have been using it for 10+ years)

[idrassi]

I have implemented the queueing part of this request in commit 92acf0e:.

HashCheck now uses a shared local queue so checksum creation, verification and Properties-tab hashing run one job at a time instead of competing for disk I/O. Separate dialogs are still kept, as suggested above, but jobs waiting behind another one now show a "Queued" state until they reach the front of the queue.

For CLI launches through HashCheckPackageHost.exe, I also added a per invocation /noqueue option to bypass serialization for that specific command.

The attached screenshot shows one verification job running while another one is waiting in the queue.

I'm preparing a new version that will include this change.

[ZPNRG]

@idrassi Thank you so much again for the newest release (I have already thoroughly appreciated some of the fixes and improvements) as well as this upcoming release. I will definitely be testing and using the queueing ability.

[christantoan]

I think it would be great if there's a button to force start the hash check.

[nutpantz]

I think it would be great if there's a button to force start the hash check.

i am in favor of being able to pause the current hash and start one that is farther down the que.
but force starting one so two are going at the same time? that should be in the settings.

thanks for this, i will start using and testing it right away.

[manuelrn]

@idrassi thank you so much for the two recent versions. I'm very glad you're continuing to support this great utility.

I have a suggestion:
In this latest version, you mention that you've implemented a queue so that if the user opens multiple tasks, the tool doesn't run them all at once and cause disk I/O saturation.

And I was wondering:
Would it be possible to implement it so that if the task is on a different disk, it does run?
In other words, so there’s one queue per disk, rather than a single global queue.

I mention this because, at least in my case, there are times when I have one file on two disks and I want to check if they’re identical. So I calculate the hash of both in their properties-tab to verify it, and I run both simultaneously because, since they’re on different disks, they won’t saturate or interfere with each other.
However, with this update, you would have to wait for one task to finish before the next one can run, even though that isn't necessary and they can run simultaneously since they involve different disks.

Although perhaps I've spoken too soon, and the utility already separates tasks by disk. I don't know, I haven't had the chance to test the new update yet.

Best regards and thank you very much!

[idrassi]

Thanks for the feedback. I see two main proposals here:

1. More queue management, for example "run this queued job now" or pausing the current job and promoting another one.
2. Queueing per physical disk instead of using one global HashCheck queue.

For queue management, I want to be careful not to make the behavior confusing. Today the queue is intentionally simple: jobs wait in FIFO order, each job keeps its own window and Pause means "pause this job" while keeping its place. If Pause also meant "yield the queue to another job", then we would have to define what happens when the paused job resumes: does it return to the front, go to the back, keep its old position, or compete again with other queued jobs? This also becomes harder because there is no central queue window: each job has its own dialog.

Internally, the current queue stores only the minimum shared state needed for FIFO serialization. Real queue management would require more shared state and commands between independent HashCheck windows: reordering, priority, yielding, resuming, stale job cleanup and UI state synchronization. That is doable but it is a big work.

The pragmatic queue management feature I would consider first is a "Run now" button shown only when a job is queued.
This would be an explicit per job override, similar in spirit to the CLI /noqueue option: the selected job leaves the queue and starts immediately, while the already running job continues. This can intentionally create concurrent disk I/O, so it should be a deliberate user action. It also doesn't disturb the order of the remaining queued jobs.

For per physical disk queueing, it indeed makes sens. For common use cases, it is possible get the underlying physical disk identifier and queue based on that instead of using one global queue. That would allow jobs on different disks to run concurrently while still preventing competing jobs on the same disk.

The complexity is in making it robust for all cases: jobs can include files from different drives, so how to handle such multi-drive per job cases?

A pragmatic approach can be:

• Use per physical disk queueing when all touched paths resolve cleanly to one or more local disk identifiers.
• If a job touches several disks, let it run only when all those disk identifiers are currently free.
• If a path can't be resolved reliably, put the job in a conservative fallback bucket, probably the current global queue.
• Keep /noqueue as the explicit override for users who intentionally want parallel work.

This gets complicated very quickly!

So my current direction is: keep the simple global queue as the safe default introduced in v2.6.1, consider "Run now" as the first queue management enhancement, and treat per physical disk scheduling as a separate improvement that should be implemented carefully.

[christantoan]

If a job touches several disks, let it run only when all those disk identifiers are currently free.

An idea would be to process files on disk identifiers that currently free first, parallelly if possible.

Don't you hate it when something that looks simple actually very complicated..? 😂