Skip to content

README.md

Latest commit

 

History

History
41 lines (31 loc) · 2.67 KB

README.md

File metadata and controls

41 lines (31 loc) · 2.67 KB
Raw

Engine documentation

LakeBench engines provide a consistent benchmark interface while preserving each runtime's native storage, authentication, telemetry, and pricing behavior.

Engine Install Primary runtime Documentation Coverage
Generic Spark lakebench[spark] Local or Spark runtime Spark Report
Fabric Spark Runtime-provided Spark Microsoft Fabric Fabric Spark —
Synapse Spark Runtime-provided Spark Azure Synapse Synapse Spark —
HDInsight Spark Runtime-provided Spark Azure HDInsight HDInsight Spark —
Databricks Spark lakebench[databricks_spark] Databricks on AWS, Azure, or GCP Databricks Spark —
Databricks SQL Warehouse lakebench[databricks_sql_warehouse] Databricks SQL Databricks SQL Warehouse —
Fabric Data Warehouse lakebench[fabric_data_warehouse] Microsoft Fabric Fabric Data Warehouse —
DuckDB lakebench[duckdb] Local or notebook runtime DuckDB Report
Polars lakebench[polars] Local or notebook runtime Polars Report
Daft lakebench[daft] Local or notebook runtime Daft Report
Sail lakebench[sail] Local or notebook runtime Sail Report

All engines write benchmark results through the same shared result contract. A benchmark accepts the engine instance, so the benchmark workflow remains the same after engine construction:

from lakebench.benchmarks import TPCH

benchmark = TPCH(
    engine=engine,
    scenario_name="sf10",
    scale_factor=10,
    input_folder_uri="path-or-cloud-uri/tpch/sf10",
)
benchmark.run()

Shared pricing contract

Engines backed by vCore compute can accept either cost_per_vcore_hour or cost_per_hour; the inputs are mutually exclusive. A total hourly rate should include every cost component the user wants represented in estimated_retail_job_cost.

Automatic estimates use public list prices and are not invoice calculations. They do not account for negotiated discounts, commitments, credits, taxes, or other private billing adjustments.

Adding an engine

Contributor guidance for custom engines remains in the root README. Internal helpers such as DeltaRs do not have standalone engine pages.