Repository navigation
feat: Add SQL schema discovery library for PostgreSQL - #7081
Open
patelchaitany wants to merge 2 commits into
Open
patelchaitany wants to merge 2 commits into
patelchaitany wants to merge 2 commits into
Conversation
Introduces feast.schema_discovery, which introspects a PostgreSQL database from an ad-hoc connection URL. It requires neither a feature_store.yaml nor a registered data source, so a user can explore a database before defining anything in Feast. SchemaDiscoveryService exposes list_schemas(), list_tables() and describe_table(). Column discovery deliberately combines two sources: - Feast's own DataSource supplies column types. Its get_table_column_names_and_types() reports canonical catalog spellings such as "character varying", which is what pg_type_to_feast_value_type keys on. SQLAlchemy reports "VARCHAR(255)" for the same column, which maps to UNKNOWN. - SQLAlchemy's Inspector supplies schema and table navigation, plus the nullability and primary-key metadata that a cursor description omits. Table and schema names reach the database through string interpolation in DataSource.get_table_query_string(), so identifiers are validated against a strict unquoted-identifier pattern before any connection is opened. That pattern is anchored with \Z rather than $, because $ also matches immediately before a trailing newline and would admit "users\n". Errors subclass FeastError and carry http_status_code(), so the REST layer added later can map them without a translation table. v1 accepts PostgreSQL only; every other dialect is refused by allowlist. Signed-off-by: Chaitany Patel <patelchaitany93@gmail.com>
Exercises SchemaDiscoveryService against a real PostgreSQL 16 instance
via testcontainers, following the PostgresContainer pattern already used
elsewhere in the integration suite.
Covers schema listing with system namespaces excluded, table and view
enumeration, column type mapping across the common PostgreSQL types,
primary-key and nullability reporting, describing a view, and the
not-found and connection-failure paths.
Includes a regression guard asserting that no column in the fixture
table maps to UNKNOWN. That assertion fails if column types ever start
coming from SQLAlchemy's type spellings ("VARCHAR(255)") rather than the
catalog spellings the Feast type mappers expect.
These tests require Docker and are marked with pytest.mark.integration.
Signed-off-by: Chaitany Patel <patelchaitany93@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Adds
feast.schema_discoveryso users can inspect a PostgreSQL database from a connection URL before creating afeature_store.yamlor registering a data source.SchemaDiscoveryServicelists schemas, lists tables and views, and describes columns with SQL types, nullability, primary-key metadata, and mapped FeastValueTypevalues. It combines Feast's existing PostgreSQL type discovery with SQLAlchemy metadata inspection, validates identifiers before connecting, and returns structured errors without exposing passwords.This change provides the Python library. A REST endpoint is outside its scope.
Checks
mypy feast/schema_discoverypass.git commit -s).Testing Strategy
Misc
@haoxu0, please review the PostgreSQL schema-discovery library and its proposed API.