Skip to content

feat(cloud): add dbt-sf-to-bq-translator skill - #221

Closed
prabhaarya wants to merge 1 commit into
google:mainfrom
prabhaarya:feat/add-dbt-sf-to-bq-translator
Closed

prabhaarya wants to merge 1 commit into
google:mainfrom
prabhaarya:feat/add-dbt-sf-to-bq-translator

Conversation

@prabhaarya

Copy link
Copy Markdown

Adds the dbt-sf-to-bq-translator skill under skills/cloud/dbt-sf-to-bq-translator/.
This skill provides autonomous guidance and automation for translating Snowflake dbt SQL models into standardized, compilation-ready Google Cloud BigQuery SQL.

Key Capabilities

  • Jinja Macro Masking & Restoration: Safely converts dbt Jinja constructs (ref, source, complex control blocks) into standard SQL-compliant placeholders before sending to BigQuery Translation Service, restoring them post-translation to preserve full dbt DAG lineage.
  • AST-Based Config Transformation: Parses and sanitizes {{ config(...) }} blocks, stripping Snowflake-specific parameters (such as transient, copy_grants, and secure) and sanitizing pre/post hooks.
  • GoogleSQL Standardization: Enforces mandatory Google copyright headers, standardized JSON extraction (CAST(JSON_EXTRACT_SCALAR(...) AS TYPE)), explicit type casting, and deduplication patterns using QUALIFY rn = 1.
  • Namespace Resolution: Resolves hardcoded two-part and three-part Snowflake table references back into native dbt {{ ref(...) }} and {{ source(...) }} macros using discovered project models and schema definitions.
  • Bulk Translation CLI: Provides a standalone CLI script (scripts/bulk_translate_via_gcloud.py) for batch migration workflows.

Changes Included

  • skills/cloud/dbt-sf-to-bq-translator/:
    • SKILL.md: Main agent skill definition with structured triggers and step-by-step migration instructions.
    • README.md: Skill overview, prerequisite GCP permissions, and CLI usage.
    • references/dbt_migration_patterns.md: Before-and-after dialect translation pattern guide.
    • scripts/dbt_translator.py: Core translation engine, AST config parser, regex sanitizers, and GCS workflow runner.
    • scripts/bulk_translate_via_gcloud.py: Bulk translation command-line entrypoint.
    • tests/test_dbt_translator.py: Complete unit test suite for namespace resolution, macro audit, config transformation, and sanitization.
    • assets/: Workflow architecture and processing diagrams.
  • README.md: Registered DBT Snowflake to BigQuery Translator under the Databases and analytics section.
  • index.json: Catalog entry registered in alphabetical order pointing to the raw SKILL.md entrypoint.

Verification

  • Ran unit test suite (tests/test_dbt_translator.py): all 15 tests passed.
  • Verified that all scripts use generic placeholders with no hardcoded filesystem paths or customer references.
  • Verified index.json formatting and deterministic sorting against repository standards.

@kumar-dhanagopal

Copy link
Copy Markdown
Collaborator

Hello @prabhaarya , thank you for this contribution! As explained in https://github.com/google/skills/blob/main/CONTRIBUTING.md#how-you-can-help, we're not accepting PRs for this repo currently. Your suggestion is valuable though, and we'd like to make sure that we internally triage it. Please feel free to post it as an issue in https://github.com/google/skills/issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants