Skip to content

Commit 85569a2

Browse files
authored
duckdb: don't use tokio for local reads (#9175)
Compat:: while creating a filesystem makes duckdb integration use Tokio. Tokio is bad on 64/128 core machines because it spawns every read into a blocking pool. On 128 core contention eats 48% of runtime Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
1 parent 9e84097 commit 85569a2

1 file changed

Lines changed: 6 additions & 0 deletions

File tree

‎vortex-duckdb/src/multi_file.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,12 @@ use crate::duckdb::ExtractedValue;
2929
static REGISTRY: LazyLock<Registry> = LazyLock::new(Registry::new);
3030

3131
fn resolve_filesystem(base_url: &Url) -> VortexResult<FileSystemRef> {
32+
// Compat makes us use tokio which is very bad for local reads on
33+
// high-core machines because reads go into blocking pool
34+
if base_url.scheme() == "file" {
35+
return Ok(Arc::new(ObjectStoreFileSystem::local(RUNTIME.handle())));
36+
}
37+
3238
// `base_url` has its path cleared by the caller, so the resolved path is empty and only the
3339
// store matters here. Going through the shared registry means DuckDB resolves the same set of
3440
// schemes as the Python and Java bindings, including the OpenDAL-backed ones when the

0 commit comments

Comments
 (0)