You can use the Dockerfile to build the environment:
docker build -t promptfuzz .
docker run -it promptfuzz bash
Before you use this tool for a new project, you must have a automatic build script to build your project to prepare the required data (e.g., headers, link libraries, test data and etc.), like OSS-Fuzz. See Preparation.
We have prepared the build scripts for some popular open source libraries, you can refer to the data directory.
If you prefer to set up the environment locally instead of using Docker, you can follow the instructions below:
Requirements:
- Rust stable
- LLVM and Clang (built with compiler-rt)
- wllvm (installed by
pip3 install wllvm)
You can download llvm and clang from this link or install by llvm.sh.
Explicit dependency see Dockerfile.
If you prefer build llvm manually, you can build clang with compiler-rt and libcxx from source code following the config:
cmake -S llvm -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;lld" \
-DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;compiler-rt;" \
-DCMAKE_BUILD_TYPE=Release -DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON \
-DLIBCXXABI_ENABLE_SHARED=OFF -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
Run the script in the promptfuzz/data directory, to prepare the required data of this library.
After the build process is finished, the data of this library is stored under promptfuzz/output/build/.
CNTG use OPENAI API as the backend LLM service interfaces, and you need to set up the following environment variables to access your LLM service:
OPENAI_API_KEY: Your LLM service access key.OPENAI_MODEL_NAME: Your LLM service model name.OPENAI_PROXY_BASE(Optional): Your LLM service address.OPENAI_INPUT_PRICE(Optional): The cost of a single prompt.OPENAI_OUTPUT_PRICE(Optional): The cost of a single output.OPENAI_CONTEXT_LIMIT(Optional): The maximum number of tokens that can be generated by the LLM service.
You must set your API key and the model name in the environment variable:
user@ubuntu$ export OPENAI_API_KEY=$(your_key)
user@ubuntu$ export OPENAI_MODEL_NAME=$(your_model_name)
If you want to use a proxy server to access the LLM service, you can set the following environment variable:
user@ubuntu$ export OPENAI_PROXY_BASE=https://openai.proxy.com/v1
If you need to run CNTG on your local models, you should use vllm or other inference engines to deploy your LLM service first.
CNTG generates API sequences. There are several options that can be tuned in the configuration.
Typically, the only options that need to be actively set are -c and -r. The
-c option determines the number of cores to be used for sanitization. Enabling
the -r option will periodically re-check the correctness of the seed programs,
reducing false positives but also introducing some extra overhead.
For instance, the following command is sufficient to generate API sequences for zlib
cd /data/zlib
./build.sh
export OPENAI_MODEL_NAME="gpt-4o-mini-2024-07-18"
cargo run --bin fuzzer -- zlib -c $(nproc) -r
Library name: zlib, cJSON, cre2, lcms, sqlite3,libpng,
Once the API sequences are generated, you can fuse them into a single executable and collect coverage to benchmark the effectiveness of the generated API combinations.
Run the following command to record coverage-over-time to the seed metadata file
(located at output/<lib>/seed_metas.csv)
cargo run --bin harness -- <lib> record-coverage
You can run this command to fuse the programs into a single executable:
cargo run --bin harness -- zlib fuse-seeds
And then, you can collect the coverage of the fused program:
cargo run --bin harness -- zlib collect-coverage
To get a human-readable coverage report, you can use the report-cntg command. This command will first collect the coverage and then generate a report.
cargo run --bin harness -- zlib report-coverage