Skip to content

Commit 38b17f8

Browse files
joseph-isaacsclaude
joseph-isaacs
and
claude
authored
feat(bench): run the random access benchmark against S3 (#9412)
## Summary The random-access benchmark only ever read its data from local NVMe, which hides the cost that matters most for point lookups: how many bytes and round trips a format needs against an object store. This adds an S3 variant of the same benchmark so Vortex, Parquet, and Lance random access can be tracked against remote storage, both on `develop` and on demand for PRs. The synthetic Parquet files keep the parquet-rs default row group and page sizes; only the codec (zstd level 3, from #10157) is set. ## Changes - `random-access-bench` gains `--remote-data-dir s3://bucket/prefix/` and `--prepare-data`. Data is materialized locally as before, uploaded verbatim, and then opened through `object_store` (Vortex and Parquet) or Lance's own `s3://` provider. Remote measurements are named `...-tokio-s3` and tagged with `s3` storage so they form a separate series from the local-disk numbers. Arrow IPC has no object-store reader and is skipped for remote runs. - v3 ingest records have no storage field and their measurement ID hashes only commit, dataset, format and open mode, so S3 runs use an `-s3` dataset suffix (e.g. `taxi-s3/uniform`). This keeps them from overwriting the local-disk rows without a schema change; local-disk IDs are unchanged. - `vortex-bench` adds `RemoteDataDir`, which maps a local data path to its object key, plus `open_object_store` constructors for the Vortex and Parquet accessors. The Parquet accessor keeps its cached footer and re-opens through a boxed `AsyncFileReader` on each take, so the local and object-store paths share one read path. - CI: new `pr-bench-random-access-s3.yml` reusable workflow, wired into `pr-bench-dispatch.yml` behind the `action/bench-random-access-s3` label and `action/bench-all`. `pr-bench-runner.yml` learns `variant_id` and `remote_data_dir` inputs, uploads the data before the run and deletes it afterwards. `develop-bench.yml` gains a `Random Access (S3)` matrix entry that refreshes `s3://vortex-ci-benchmark-datasets/develop/random-access/` on every run. - `lance-bench` enables Lance's `aws` feature so `Dataset::open` accepts `s3://` URIs. - Docs: README section for running against S3 and the new label in the benchmarking guide. Unit tests cover the key and URI mapping of `RemoteDataDir` and the separate S3 ingest dataset. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01L627uApatcD62FGrQPutCL --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude <noreply@anthropic.com>
1 parent ca31e64 commit 38b17f8

16 files changed

Lines changed: 1241 additions & 89 deletions

File tree

‎.github/workflows/develop-bench.yml‎

Lines changed: 31 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ jobs:
8383
timeout-minutes: 120
8484
runs-on: >-
8585
${{ github.repository == 'vortex-data/vortex'
86-
&& format('runs-on={0}/runner=bench-dedicated/family=c8gd.metal-24xl/image=ubuntu24-full-arm64-pre-v2/extras=s3-cache/tag={1}', github.run_id, matrix.benchmark.id)
86+
&& format('runs-on={0}/runner=bench-dedicated/family=c8gd.metal-24xl/image=ubuntu24-full-arm64-pre-v2/extras=s3-cache/tag={1}{2}', github.run_id, matrix.benchmark.id, matrix.benchmark.variant_id)
8787
|| 'ubuntu-latest' }}
8888
strategy:
8989
fail-fast: false
@@ -93,6 +93,14 @@ jobs:
9393
name: Random Access
9494
build_args: "--features lance"
9595
v4_ingest: true
96+
# Same benchmark, reading the data from S3 instead of local NVMe.
97+
- id: random-access-bench
98+
variant_id: "-s3"
99+
name: Random Access (S3)
100+
build_args: "--features lance"
101+
v4_ingest: true
102+
remote_data_dir: >-
103+
s3://vortex-ci-benchmark-datasets/develop/random-access/
96104
- id: compress-bench
97105
name: Compression
98106
build_args: "--features lance"
@@ -135,6 +143,25 @@ jobs:
135143
extra_args: "--debuginfo-strip=false"
136144
parca_agent_version: "0.49.0"
137145

146+
- name: Setup AWS CLI
147+
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
148+
with:
149+
role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole
150+
aws-region: us-east-1
151+
152+
- name: Upload benchmark data to S3
153+
if: matrix.benchmark.remote_data_dir != null
154+
shell: bash
155+
env:
156+
AWS_REGION: "us-east-1"
157+
run: |
158+
set -Eeu -o pipefail -x
159+
160+
target/release_debug/${{ matrix.benchmark.id }} --prepare-data \
161+
--formats parquet,vortex,lance
162+
aws s3 rm --recursive "${{ matrix.benchmark.remote_data_dir }}"
163+
aws s3 cp --recursive vortex-bench/data "${{ matrix.benchmark.remote_data_dir }}"
164+
138165
- name: Setup benchmark environment
139166
run: sudo bash scripts/setup-benchmark.sh
140167

@@ -144,8 +171,10 @@ jobs:
144171
env:
145172
RUST_BACKTRACE: full
146173
FLAT_LAYOUT_INLINE_ARRAY_NODE: "1"
174+
AWS_REGION: "us-east-1"
147175
run: |
148-
python3 scripts/random-access-split.py --emit-ingest-records
176+
python3 scripts/random-access-split.py --emit-ingest-records \
177+
${{ matrix.benchmark.remote_data_dir && format('--remote-data-dir {0}', matrix.benchmark.remote_data_dir) || '' }}
149178
150179
- name: Run ${{ matrix.benchmark.name }} benchmark (per-dataset)
151180
if: matrix.benchmark.id == 'compress-bench'
@@ -165,12 +194,6 @@ jobs:
165194
run: |
166195
python3 scripts/string-split.py
167196
168-
- name: Setup AWS CLI
169-
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
170-
with:
171-
role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole
172-
aws-region: us-east-1
173-
174197
- name: Upload Benchmark Results
175198
shell: bash
176199
run: |

‎.github/workflows/pr-bench-dispatch.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,11 @@ jobs:
3535
uses: ./.github/workflows/pr-bench-random-access.yml
3636
secrets: inherit
3737

38+
all-random-access-s3-bench:
39+
needs: remove-all-label
40+
uses: ./.github/workflows/pr-bench-random-access-s3.yml
41+
secrets: inherit
42+
3843
all-compression-bench:
3944
needs: remove-all-label
4045
uses: ./.github/workflows/pr-bench-compress.yml
@@ -68,6 +73,22 @@ jobs:
6873
uses: ./.github/workflows/pr-bench-random-access.yml
6974
secrets: inherit
7075

76+
remove-random-access-s3-label:
77+
runs-on: ubuntu-latest
78+
timeout-minutes: 10
79+
if: github.event.label.name == 'action/bench-random-access-s3'
80+
steps:
81+
- uses: actions-ecosystem/action-remove-labels@2ce5d41b4b6aa8503e285553f75ed56e0a40bae0 # v1
82+
if: github.event.pull_request.head.repo.full_name == 'vortex-data/vortex'
83+
with:
84+
labels: action/bench-random-access-s3
85+
fail_on_error: true
86+
87+
random-access-s3-bench:
88+
needs: remove-random-access-s3-label
89+
uses: ./.github/workflows/pr-bench-random-access-s3.yml
90+
secrets: inherit
91+
7192
remove-compress-label:
7293
runs-on: ubuntu-latest
7394
timeout-minutes: 10

‎.github/workflows/pr-bench-random-access-s3.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Runs the random-access benchmark for a pull request, reading the data from S3.
2+
3+
name: PR Random Access S3 Benchmark
4+
5+
on:
6+
workflow_call: { }
7+
workflow_dispatch: { }
8+
9+
permissions:
10+
contents: read
11+
pull-requests: write # for commenting on PRs
12+
id-token: write # enables AWS-GitHub OIDC
13+
14+
jobs:
15+
bench:
16+
uses: ./.github/workflows/pr-bench-runner.yml
17+
secrets: inherit
18+
with:
19+
benchmark_id: random-access-bench
20+
benchmark_name: Random Access (S3)
21+
with_lance: true
22+
variant_id: "-s3"
23+
remote_data_dir: >-
24+
s3://vortex-ci-benchmark-datasets/${{ github.ref_name }}/${{ github.run_id }}/random-access/

‎.github/workflows/pr-bench-runner.yml‎

Lines changed: 47 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ name: PR Benchmark Runner
44

55
concurrency:
66
# The group causes runs to queue instead of running in parallel.
7-
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}-${{ inputs.benchmark_id }}
7+
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}-${{ inputs.benchmark_id }}${{ inputs.variant_id }}
88
# Don't cancel benchmarks that are already running, instead just queue them up.
99
cancel-in-progress: false
1010

@@ -21,6 +21,20 @@ on:
2121
required: false
2222
type: boolean
2323
default: false
24+
variant_id:
25+
description: >-
26+
Suffix distinguishing runs of the same benchmark, e.g. "-s3". Keeps the PR comment
27+
tag and the concurrency group of a variant separate from the default run.
28+
required: false
29+
type: string
30+
default: ""
31+
remote_data_dir:
32+
description: >-
33+
When set, the benchmark data is uploaded to this S3 prefix and read back from there
34+
instead of local disk. Only supported by random-access-bench.
35+
required: false
36+
type: string
37+
default: ""
2438

2539
permissions:
2640
contents: read
@@ -122,6 +136,26 @@ jobs:
122136
extra_args: "--debuginfo-strip=false"
123137
parca_agent_version: "0.49.0"
124138

139+
- name: Setup AWS CLI
140+
if: github.event.pull_request.head.repo.fork == false
141+
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
142+
with:
143+
role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole
144+
aws-region: us-east-1
145+
146+
- name: Upload benchmark data to S3
147+
if: inputs.remote_data_dir != '' && github.event.pull_request.head.repo.fork == false
148+
shell: bash
149+
env:
150+
AWS_REGION: "us-east-1"
151+
run: |
152+
set -Eeu -o pipefail -x
153+
154+
target/release_debug/${{ inputs.benchmark_id }} --prepare-data \
155+
--formats ${{ inputs.with_lance && 'parquet,vortex,lance' || 'parquet,vortex' }}
156+
aws s3 rm --recursive "${{ inputs.remote_data_dir }}"
157+
aws s3 cp --recursive vortex-bench/data "${{ inputs.remote_data_dir }}"
158+
125159
- name: Setup benchmark environment
126160
run: sudo bash scripts/setup-benchmark.sh
127161

@@ -131,8 +165,10 @@ jobs:
131165
env:
132166
RUST_BACKTRACE: full
133167
FLAT_LAYOUT_INLINE_ARRAY_NODE: "1"
168+
AWS_REGION: "us-east-1"
134169
run: |
135-
python3 scripts/random-access-split.py
170+
python3 scripts/random-access-split.py \
171+
${{ inputs.remote_data_dir != '' && format('--remote-data-dir {0}', inputs.remote_data_dir) || '' }}
136172
137173
- name: Run ${{ inputs.benchmark_name }} benchmark (per-dataset)
138174
if: inputs.benchmark_id == 'compress-bench'
@@ -152,13 +188,6 @@ jobs:
152188
run: |
153189
python3 scripts/string-split.py
154190
155-
- name: Setup AWS CLI
156-
if: github.event.pull_request.head.repo.fork == false
157-
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
158-
with:
159-
role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole
160-
aws-region: us-east-1
161-
162191
- name: Install uv
163192
uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6
164193
with:
@@ -181,7 +210,7 @@ jobs:
181210
uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3
182211
with:
183212
file-path: comment.md
184-
comment-tag: bench-pr-comment-${{ inputs.benchmark_id }}
213+
comment-tag: bench-pr-comment-${{ inputs.benchmark_id }}${{ inputs.variant_id }}
185214

186215
- name: Comment PR on failure
187216
if: failure() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false
@@ -191,4 +220,11 @@ jobs:
191220
# BENCHMARK FAILED
192221
193222
Benchmark `${{ inputs.benchmark_name }}` failed! Check the [workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details.
194-
comment-tag: bench-pr-comment-${{ inputs.benchmark_id }}
223+
comment-tag: bench-pr-comment-${{ inputs.benchmark_id }}${{ inputs.variant_id }}
224+
225+
- name: Delete benchmark data from S3
226+
if: always() && inputs.remote_data_dir != '' && github.event.pull_request.head.repo.fork == false
227+
shell: bash
228+
env:
229+
AWS_REGION: "us-east-1"
230+
run: aws s3 rm --recursive "${{ inputs.remote_data_dir }}"

0 commit comments

Comments
 (0)