dragonfly-client-rs

Modular compute nodes capable of scanning packages and sending results upstream to a control server, written in Rust.

Set up

This section goes over how to set up a client instance locally and via Docker.

Refer to the Environment variables section for information on what environment variables are necessary.

Local

Requirements

• Rust
• YARA

1. Set the appropriate environment variable pointing to the YARA installation

export YARA_LIBRARY_PATH='/path/to/yara/libs'

2. Build the binary with cargo

cargo build --release

3. Run the built binary

./target/release/dragonfly-client-rs

Docker

Requirements

• Docker Engine

1. Build and tag the image

docker build --tag vipyrsec/dragonfly-client-rs:latest .

2. Run the container

docker run --name dragonfly-client-rs vipyrsec/dragonfly-client-rs:latest

Docker Compose

Requirements

• Docker Engine
• Docker Compose

Run the service

docker compose up

How it works: Overview

The follow is a brief overview of how the client works. A more extensive writeup can be found towards the bottom of this page.

The client is comprised of a few discrete components, each running independently. These are the scanning threadpool, the loader thread, and the sender thread.

• The Scanning Threadpool - Downloads and scans the releases.
• The Loader Thread - This thread is responsible for requesting jobs from the API and submitting them to the threadpool.

Performance, efficiency, and optimization

The client aims to be highly configurable to suit a variety of host machines. The scanner processes one package and one distribution at a time. Compressed downloads, expanded archives, individual scan targets, archive entry counts, and distributions per package are bounded independently. The defaults target a 512 MiB background-worker container while preserving substantial headroom for compiled YARA rules, ZIP metadata, allocator overhead, and filesystem cache.

Within one package, identical file contents reuse successful YARA matches across distributions. Streaming XXH3-128 hashes and file sizes identify candidates; byte-for-byte comparison confirms equality before reuse. Rule filetype metadata is applied to each original path, preserving distribution scores and inspector links even when identical files have different names or extensions.

The cache is discarded after each package and holds at most DRAGONFLY_MAX_ARCHIVE_ENTRIES representatives and DRAGONFLY_MAX_EXPANDED_SIZE bytes on temporary disk (4096 files and 64 MiB by default), in addition to the current extracted distribution. Once full, new contents are scanned normally. Cache write errors log a warning and disable new entries without discarding successful scan results. Failed scans are never cached. The content_scan_cache log event reports scanned_files and reused_files for completed packages.

How it works: Detailed Breakdown

This section attempts to describe in detail how the client works under the hood, and how the various configuration parameters come into play.

The client can be broken down into a few discrete components: The scanner threads, the loader thread, the sender thread. We will first explore in detail the workings of each of these components in isolation and then how they all fit together.

The scanner thread(s) are what do most of the heavy lifting. They use bindings to the C YARA library, and most of this code can be found in scanner.rs. The way this program models PyPI data structure is as so: There are "packages" (or "releases") which is a name/version specifier combination. These "packages" are comprised of several "distributions" in the form of gzipped tarballs or wheels (which behave similarly to zip files, hence the use of the zip crate). Each distribution is comprised of a flat sequence of files (the hierarchical nature of the traditional file/folder system has been flatted for our use case). The main entry point interface to the scanner logic is via scan_all_distributions. This loops over the download URLs sequentially, stages each compressed distribution on temporary disk, validates its resource limits, and extracts it. Files are scanned individually from disk by YARA. Only the highest-scoring file and unique matched rules are retained for each distribution. After every distribution finishes, the client unions matched rule identifiers across the complete package and submits one package verdict to the API. Its score is the maximum independently calculated distribution score; evidence from mutually exclusive distributions is not added together. The inspector URL identifies the highest-scoring file in that distribution.

The client requests at most one job per configured worker, up to DRAGONFLY_BULK_SIZE, and scans those packages concurrently. Each package's distributions and files remain sequential. The default worker count follows the machine's available CPU parallelism; constrained deployments can set DRAGONFLY_THREADS=1 to guarantee sequential package processing. Empty and failed job requests are retried after DRAGONFLY_LOAD_DURATION seconds.

The client authenticates every Dragonfly API request with a Cloudflare Access service token. The source code of the YARA rules is compiled (very much like compiling regex) and stored in shared state. Then, the necessary threads are spawned. Once a threadpool task has finished scanning, it sends its results over the Dragonfly HTTP API.

Environment variables

Below are a list of environment variables that need to be configured, and what they do

Variable	Default	Description
DRAGONFLY_BASE_URL	https://dragonfly.vipyrsec.com	The base API URL for the mainframe server
DRAGONFLY_CF_ACCESS_CLIENT_ID		Environment-specific Cloudflare Access service-token client ID
DRAGONFLY_CF_ACCESS_CLIENT_SECRET		Environment-specific Cloudflare Access service-token client secret
DRAGONFLY_THREADS	Available parallelism / 1	Concurrent package workers; set to 1 for sequential constrained deployments
DRAGONFLY_LOAD_DURATION	60	Seconds to wait between each API job request
DRAGONFLY_BULK_SIZE	20	Upper bound for job request, also capped by the worker count
DRAGONFLY_MAX_ARCHIVE_ENTRIES	4096	Maximum number of entries in one archive
DRAGONFLY_MAX_DISTRIBUTIONS	32	Maximum number of distributions in one package
DRAGONFLY_MAX_DOWNLOAD_SIZE	33554432	Maximum compressed distribution size in bytes
DRAGONFLY_MAX_EXPANDED_SIZE	67108864	Maximum total expanded distribution size in bytes
DRAGONFLY_MAX_SCAN_SIZE	16777216	Maximum individual file size passed to YARA in bytes

Experimental cross-package reuse

Disabled by default. The staging experiment uses these settings:

Environment variable	Default	Meaning
DRAGONFLY_REUSE_CACHE_MODE	off	off, observe (rescan hits), or reuse
DRAGONFLY_REUSE_CACHE_ENTRIES	4096	Maximum entries per worker process
DRAGONFLY_REUSE_CACHE_BYTES	33554432	Maximum retained content and encoded-result bytes

The disposable cache belongs to the loaded rules snapshot and engine process. Every successful rules reload clears it, including reloads with an unchanged commit identifier. Engine/image changes and restarts start cold. No database, network cache, migration, or new dependency is involved. FIFO eviction bounds retained bytes and entry metadata; lookups compare exact bytes after hashing. Replicas do not share entries. Existing within-package deduplication remains.

Successful content results, including clean results, can be reused across package releases. Paths and Inspector locations are reconstructed for the current package. OpenGrep additionally keys by extension, bypasses content reuse for path-scoped rules or any rule options, dependency/validator context, or non-search/taint modes, and only admits explicitly scanned targets from complete, warning-free runs. Its timeout fallback groups do not populate this cache.

observe rescans every candidate and compares results. reuse rescans every hundredth hit; a mismatch disables the cache until a rules reload or restart and rejects jobs that consumed cached output. Fully fresh observation results are preserved. Reuse mode requires DRAGONFLY_THREADS=1 to make this invalidation atomic across the active job. Replica-level parallelism still uses independent caches. Cache I/O/decoding failures fall back to scanning. Scan failures are not cached.

Each job emits event="scan_reuse" with scanner, mode, rules commit (in its job span), candidate/reused files, reused bytes, engine-target files/bytes, measured engine wall time and cache overhead (microseconds), insertions, FIFO evictions, errors, validation samples and mismatches. Counts exclude existing within-package reuse. Engine-target counts describe submitted targets, not individual rules or fallback retry attempts. Reused bytes measure avoided engine input, not avoided downloads. Timings are wall time, not CPU billing or a claimed counterfactual.

Disable with DRAGONFLY_REUSE_CACHE_MODE=off and redeploy, or restore the previous image. Existing package results and schema are unchanged. Only staging should enable this experiment until its observation and reuse windows are reviewed.

Durable cache (staging experiment)

Set DRAGONFLY_REUSE_CACHE_DATABASE=true with reuse/observe mode to use Mainframe's optional /scan-cache API instead of process-local cross-job storage. This requires one scan thread and a Mainframe deployment with the reversible cache migration and SCAN_CACHE_ENABLED=true. Both flags default to disabled.

The key combines SHA-256 of file content, the actual rules corpus and its commit, the scanner/engine executable fingerprint, and OpenGrep's language extension. SHA-256 is calculated alongside XXH3 during the existing input hashing pass. No file contents are uploaded. Rules/engine changes miss the cache; unchanged workers can reuse database results after a restart. Findings are remapped to the current package and paths. Only successful, complete file scans are cached. A later failure in another file or distribution does not invalidate those completed file results; the failed package still follows normal retry handling. Failed file scans never produce durable clean entries.

Lookups/writes use batches of at most 128 files. Results are limited to 16 KiB/file and 512 KiB/write; temporary read results are capped at 16 MiB/job. A transport failure stops further cache calls for that job. Requests time out after 750 ms; new requests stop after two seconds of accumulated cache network time per job. Unavailable cache entries are scanned normally. Rule sources and match results are ordered deterministically; equivalent finding order never triggers invalidation. A sampled finding mismatch quarantines only that file hash/language in the current namespace, retaining the row as evidence until normal expiry. Hits and duplicate writes cannot clear quarantine. Other files remain reusable. Jobs that already consumed cached output are rejected; the fresh result is retained when no earlier cached output was used. Diagnostics include SHA-256, path, result counts and up to 16 cached/fresh matches. Failed quarantine requests remain blocked locally and retry before the next job's lookups. Local quarantine state is capped at 4,096 keys; exhausting that safety budget disables only this process's reuse. Generation revocation remains available on the server for broader incidents; a YARA sample mismatch no longer requests it.

The server enforces separate connection, rate, storage and expiry budgets. Database inserts are measured by scanner_cache_rows_inserted_total; the worker inserted_files counter remains specific to process-local storage. Existing reuse counters still measure actual avoided inputs. Hashing time is outside the cache-transport overhead counter; these counters do not establish CPU savings. Set DRAGONFLY_REUSE_CACHE_MODE=off to disable all cache requests. Setting only DRAGONFLY_REUSE_CACHE_DATABASE=false restores the original local cache mode.