[NMCUR-421] Add inference-server PDF benchmark path

[praateekmahajan]

Description

Adds reusable benchmark utilities for starting Ray Serve or Dynamo inference servers and moves the existing NDD benchmark onto them. Shared inference-server CLI JSON arguments are validated through one reusable parse_json_object helper.

Adds NemotronParseHTTPClientStage to the production Nemotron-Parse stage package. It sends OpenAI-compatible multimodal page requests through Curator's existing AsyncOpenAIClient, including bounded concurrency and exponential-backoff retries, while preserving response order, raw completion usage, and finish metadata. Requests that still fail after retries raise the task, matching the in-process path and preventing silently partial PDF output.

NemotronParsePDFReader creates exactly one inference stage: the existing in-process stage when no endpoint is supplied, or the HTTP client stage when inference_server_endpoint is set. The composite and tutorial factory accept a direct inference_server_client_num_workers value and do not model inference-server replicas. In-process proc_size remains internally derived from AutoProcessor; the HTTP client stage retains the model-default (2048, 1664) metadata needed by postprocessing.

The in-process vLLM stage and HTTP client stage derive their generation settings from one canonical plain mapping, including a shared parameterized max_tokens default of 8192. The HTTP adapter only reshapes vLLM-specific fields into the OpenAI-compatible extra_body wire format.

The recommended tutorial entry point is now tutorials/interleaved/nemotron_parse_pdf/main.py: it detects Ray-visible GPUs, starts one Dynamo-vLLM replica per GPU, waits for health, runs the Ray Data pipeline, and stops the server. It does not expose --backend. The in-process fallback is inprocess.py, which owns --backend {vllm,hf}; shared argument and pipeline construction lives in pipeline_utils.py, so neither entry point imports the other. The docs include the exact Dynamo command and link the etcd/nats-server installer for source environments.

Server replicas default to get_available_cpu_gpu_resources()[1] and can be overridden with --num-replicas. The PDF benchmark exposes --inference-server-client-workers-per-replica, set explicitly to 4 in YAML, computes num_replicas * client_workers_per_replica, and passes that total to the composite as inference_server_client_num_workers. The HTTP stage warns below the validated starting point of 32 concurrent page requests per worker; the docs make clear that this was tested on 8 H100 GPUs and must be tuned on the target GPU and corpus.

The existing Xenna and Ray Data in-process workloads and throughput thresholds are retained. One Ray Data inference-server entry per server type covers Ray Serve and Dynamo, both with inference batch size 32. Dynamo uses its TCP request plane. The temporary global Slack disablement has been reverted, and both inference-server entries remain registered in benchmarking/4xGB200-64CPU.yaml.

For a startup-independent comparison between in-process and inference-server execution, the benchmark reports exactly three inference-stage metrics:

• inference_stage_pages_per_sec_per_gpu
• inference_stage_input_tokens_per_sec_per_gpu
• inference_stage_output_tokens_per_sec_per_gpu

They use the inference stage's task process-time sum normalized by stage parallelism and inference GPU count. Model/server startup is outside this stage timing. The benchmark harness continues to report full-script exec_time_s, which includes inference-server startup; time_taken_s covers pipeline.run(), and inference_server_startup_s reports server startup separately.

Usage

python benchmarking/scripts/nemotron_parse_pdf_benchmark.py \
  --benchmark-results-path=/tmp/results \
  --executor=ray_data \
  --inference-server-type=dynamo \
  --inference-server-client-workers-per-replica=4 \
  --inference-batch-size=32 \
  --manifest=/path/to/manifest.jsonl \
  --pdf-dir=/path/to/pdfs \
  --output-dir=/tmp/results/output \
  --model-path=/path/to/NVIDIA-Nemotron-Parse-v1.2 \
  --model-id=nvidia/NVIDIA-Nemotron-Parse-v1.2 \
  --backend=vllm

Four-way 8×H100 comparison

All four entries succeeded on PR head 12bccbdd and processed the same 5,794 pages. A terminal HTTP failure now fails its task rather than contributing a permanently-zero error metric.

Entry	inference_stage_pages_per_sec_per_gpu	Mean GPU power draw (W)
nemotron_parse_pdf_xenna	1.995	216.3
nemotron_parse_pdf_raydata	2.063	190.5
nemotron_parse_pdf_inference_server_ray_serve	3.929	234.9
nemotron_parse_pdf_inference_server_dynamo	3.968	250.0

The inference-stage metric excludes model/server startup. Mean GPU power draw is the arithmetic mean across all eight GPUs and all gpustats.csv samples over each complete benchmark entry. The run viewer also exposes the full-script exec_time_s values. Full results are available in the run viewer.

Validation

• Latest locally available Curator nightly (nightly-2026-09-07): 36 focused inference-stage, HTTP-client, server, and OpenAI-client tests passed.
• The broader PDF-stage file passed 47 tests and had six unrelated nightly-environment failures from the changed pypdfium2.save() API and missing cv2 extra.
• The reorganized main.py, inprocess.py, and benchmark CLIs all loaded successfully and rendered --help from the nightly container.
• EOS Dynamo batch-32 and batch-64 runs both completed all 5,794 requests without pre-shutdown errors. Batch 32 was retained because batch 64 improved aggregate throughput by only about 1.5% while increasing mean frontend request latency by about 41%.
• The final four-way EOS run completed all four entries successfully at identical page counts.
• Scoped repository pre-commit hooks passed for all new signed-off commits.
• No benchmark tests were added, per task direction.

Checklist

• I am familiar with the Contributing Guide.
• Existing tests cover the production-stage and HTTP-client changes.
• The tutorial and Fern documentation are up to date.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-inference-server-pdf-nmcur-421.docs.buildwithfern.com/nemo/curator

Here are the markdown pages you've updated:

• nemo/curator/curate-text/load-data/nemotron-parse-pdf
• nemo/curator/curate-text/synthetic/inference-server

[praateekmahajan]

/claude review
@greptileai review

[praateekmahajan]

/ok to test b901d27

[abhinavg4]

Majorly around the usage guide.

[abhinavg4]

Thanks

[sarahyurick]

Nit but do we need to expose this?

[praateekmahajan]

Looks like this was always there in the original main.py so got copied over, good catch removed it

[sarahyurick]

Can we keep the reference to this harmony issue if possible? So far I have only ever run into the tiktoken error on the GB200 benchmark.

[sarahyurick]

Thanks!