delete is too slow

[shahidcodes]

First of all thanks for this amazing tool.

Describe the bug
npkill finds the node_modules, I hit space to delete, it says deleting and takes like 2-3 min per folder.

To Reproduce
Steps to reproduce the behavior:

• Run npkill on macos
• Total size reported was > 20GB
• Hit space for few folders

Expected behavior
Delete should be fast

Screenshots

• OS: Macos
• Version: 0.11.3

[cs118651]

same situation

[zaldih]

Related with #121

I am aware that sometimes the deletion is too slow, but I don't quite understand why. Maybe the OS does some security check or something.

Does anyone have any ideas?

[alan910127]

@zaldih I also ran into this issue on Linux and looked into the source code. I see that rm -rf is being used to delete the node_modules directory.

According to online benchmarks (see references), the following methods are significantly faster than rm -rf, ranked by speed:

1. perl
2. rsync --delete
3. find -delete

I haven't tested this on MacOS, but since these tools are pretty standard on Linux, they may work similarly on MacOS. A possible approach could be:

import fs from "node:fs";
import path from "node:path";

function executable(exe: string): string | null {
    const envPath = process.env.PATH ?? "";
    const envExt = process.env.PATHEXT ?? "";
    const pathDirs = envPath.replace(/["]+/g, "").split(path.delimiter).filter(Boolean);
    const extensions = envExt.split(";");
    const candidates = pathDirs.flatMap((d) => extensions.map((ext) => path.join(d, exe + ext)));
    try {
        return candidates.filter(checkFileExists)[0] ?? null;
    } catch (e) {
        return null;
    }
}

function checkFileExists(filePath: string): boolean {
    try {
        // assume files under PATH entry directories are executable
        return fs.statSync(filePath).isFile();
    } catch {
        return false;
    }
}

if (executable("perl")) {
    console.log("perl -e 'for(<*>){((stat)[9]<(unlink))}'");
} else if (executable("rsync")) {
    const temp = fs.mkdtempSync("empty");
    console.log(`rsync -a --delete ${temp}/ node_modules/`);
} else if (executable("find")) {
    console.log("find node_modules -type f -delete");
} else {
    console.log("rm -rf node_modules");
}

I'm not entirely sure about the Perl syntax and would need to research it further.

References:

• A faster way to delete millions of files in a directory
• Which is the fastest method to delete files in Linux
• Checking if an executable exists in PATH using Node.js

[zaldih]

@alan910127 , very good research. Thank you for your time!

After investigating the resources you've shared, I've come up with these ideas:

• On Unix, use rsync. It may not be as fast as Perl, but I think its ease of use and popularity will make it work on most computers.
• If rsync doesn't exist, use rm as a fallback (warning the user that the deletion might be slower).
• On Windows, there seems to be a robocopy deletion technique that involves overwriting a directory with an empty one and then deleting it. This is to avoid traversing all the files. It would be used like this:

  mkdir empty
  robocopy /MIR empty node_modules > nul
  rmdir node_modules
  rmdir empty
  

• Another possibility would be to use the rimraf library, which seems to be very fast. However, I'd prefer to avoid using external libraries at all costs (especially "heavy" ones like these) to keep the tool as lightweight as possible.

In short, I'm going to do a lot of research and testing on these options. I'd love to see them in the next version. If you have any comments on this, I'd love to hear them!

[zaldih]

@alan910127 Hmm, this is very strange.
I have implemented the options and am running tests, and the results are confusing me.

Although theory suggests that other methods such as rsync or perl should be faster, they turn out to be much slower compared to rm -rf.
I will continue to investigate this. For now, I will leave my preliminary results below.

---

Unix

Directorio grande

• Ficheros: 277866
• Tamaño: 4,35GB

Benchmark implementado

• Método clásico (v0.12.2 - rm -rf): 9,90s
• Rsync: 14,82s
• Perl: 17,47s
• Find: 18,23s
• Rm-rf: 8,94s

Benchmark directo

❯ mkdir /tmp/empty
❯ time rsync -a --delete "/tmp/empty/" "node_modules2/"
0.87s user 10.95s system 56% cpu 20.922 total

❯ time rm -rf node_modules3
0.31s user 9.63s system 78% cpu 12.619 total

❯ time perl -e 'use File::Path qw(remove_tree); remove_tree("node_modules", {verbose => 0, safe => 0});'
4.69s user 13.36s system 78% cpu 22.895 total

Directorio pequeño

• Ficheros: 92622
• Tamaño: 1,50GB

Benchmark directo

❯ time rsync -a --delete "/tmp/empty/" "node_modules/"
0.20s user 3.10s system 95% cpu 3.444 total

❯ time rsync -a --delete --ignore-errors --whole-file --inplace --remove-source-files /tmp/empty/ node_modules/
0.22s user 3.12s system 95% cpu 3.502 total

❯ time rm -rf node_modules
0.13s user 3.03s system 95% cpu 3.321 total
_```_

[nathang21]

+1 i'm having the same problem, I left it running all day after selecting a bunch of folders to delete, and it seemingly is stuck.