feat: connector capability markers (location shape/identity/record-version) - #752
Merged
Conversation
Introduce LocationShape and connector-level capability markers (location_shape, location_identity, emits_record_version, supports_recursion) on the registry entry dataclasses, with defaults that preserve today's fsspec blob behavior for unannotated connectors. Mark the fsspec location leaves (remote_url, recursive) x-runtime-eligible in the shared base config so every fsspec connector's JSON schema carries it, and set entry-level markers on the s3/azure/gcs/box/dropbox/sftp source and destination entries. Sources emit a record version except sftp. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
location_shape None means opted-out, so consumers fall back rather than mistake an unannotated connector for an explicit fsspec declaration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tions Annotate the sql-table cohort's registry entries (the entry platform-api picks: source-preferred) with location_shape=SQL_TABLE, the equality-only location_identity, supports_recursion=False, and x-runtime-eligible on the non-credential location fields so platform-api's derivation reproduces the hand-seeded capability table. Clean matches: postgres, snowflake, teradata, couchbase, vastdb, motherduck, kdbai, ibm_watsonx_s3, databricks_volume_delta_tables. Table-name markers live on the connector-specific IndexerConfig subclass (not the shared SQL base) so vastdb, whose identity is bucket/schema, is not over-marked. mongodb and singlestore carry shape/identity/emits/recursion but their hand-table runtime path references a section the picked source entry cannot derive (indexer-hosted database/collection; dest-only table_name); flagged for platform-api-side reconciliation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…initions Annotate the search-index cohort with location_shape=SEARCH_INDEX, the index/collection identity, supports_recursion=False, and x-runtime-eligible on the mutable index/collection leaf (never the host/url credentials). Clean matches: pinecone, weaviate-cloud, qdrant-cloud, chroma, milvus, vectara. elasticsearch emits a record version. Collection leaves are marked on the shared weaviate/qdrant uploader base configs (only the cloud variants are hand-seeded). elasticsearch, opensearch, and astradb keep index_name/keyspace/collection_name on the source indexer config, so their derived runtime path is indexer_config.* while the hand table labels them connector_config.*/uploader_config.*; flagged for platform-api-side reconciliation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…itions Annotate the api-folder cohort with location_shape=API_FOLDER, the folder-path identity, and x-runtime-eligible on the path/scope leaves. onedrive, sharepoint, and google_drive have real folder trees (supports_recursion=True) and emit a record version; confluence/jira/salesforce/outlook/zendesk/gitlab are flat (supports_recursion=False). sharepoint's recursive is inherited from the onedrive indexer config; its overridden path leaf is marked directly. slack keeps its channels list on the source indexer config, so the derived runtime path is indexer_config.channels while the hand table labels it connector_config.channels; flagged for platform-api-side reconciliation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…initions Annotate the remaining cohort: local/redis/neo4j (location_shape=OTHER, equality identity, no recursion) and the destination-only delta_table (fsspec-url; table_uri is an s3 URL, so recursion stays on). x-runtime-eligible marks the non-credential location leaves. airtable, kafka-cloud, and databricks_volumes carry shape/identity/emits/ recursion, but their hand-table runtime path cannot be reproduced: airtable's base_id/table_id are parsed out of indexer list_of_paths (no such fields), kafka's field is bootstrap_server (singular) with the topic on the indexer, and databricks_volumes keeps catalog/schema/volume on the indexer config. Flagged for platform-api-side reconciliation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…le markers Several registry entries pointed location_identity at a census-guessed settings section while their x-runtime-eligible markers lived on the real field. Make each entry's identity reference the same fields/sections the markers actually annotate: indexer_config.* for source location leaves, uploader_config.* for destination leaves, connector_config.* only for fixed connection-level identity fields. Also splits the opensearch and singlestore dual-role connectors so the source identity resolves onto the indexer config and the destination identity onto the uploader config, marking the destination location leaves runtime-eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… common write targets Twelve commonly used write-target connectors had an unannotated destination registry entry (location_shape=None). Annotate each with the shape of its cohort and a location_identity that names its real write location, marking the write-location leaves x-runtime-eligible: - sql-table: postgres/snowflake/teradata/vastdb -> uploader_config.table_name (+ connection database/schema/bucket fixed identity); mongodb -> uploader_config.database+collection; couchbase writes to its connection-level bucket/scope/collection (no uploader location field). - search-index: elasticsearch -> connector_config.hosts + uploader_config.index_name; astradb -> uploader_config.keyspace+collection_name. - other: kafka-cloud -> connector_config.bootstrap_server + uploader_config.topic; databricks_volumes -> uploader_config.catalog/schema/volume; local -> uploader_config.output_dir. - fsspec-url: onedrive -> uploader_config.remote_url. table_name eligibility now lives on the shared SQLUploaderConfig, so the singlestore uploader redeclaration added earlier is dropped. Preserves the PART A invariant: every identity leaf is a real field on its section, eligible unless it is a fixed connection-level identity field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…markers Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 53 files
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
Address valid cubic findings where a marker contradicted the connector's actual runtime behavior. Only unambiguous, test-green corrections are applied; identity leaves stay real fields on their named section. Record-version emission (metadata.version is a real changing revision token at runtime): - opensearch source: inherits _version into metadata.version (matches the elasticsearch sibling) - confluence source: page version.number, increments on edit - salesforce source: SystemModstamp, changes on modification - outlook source: message change_key, changes on modification Recursion: - outlook source: has a real recursive field and _list_messages descends child folders, so supports_recursion=True location_identity completions (real fields that select the physical target but were omitted, so distinct targets could collide): - kafka cloud source/destination: +connector_config.port (broker endpoint is host:port, assembled at runtime) - redis destination: +connector_config.port - motherduck destination: +connector_config.table (write target table) - vectara destination: +connector_config.corpus_key (actual write target; corpus_name is None when only the key is given) - onedrive source/destination: +connector_config.user_pname (selects the drive) - vastdb/singlestore source: +indexer_config.table_name, with the x-runtime-eligible override matching the postgres/snowflake/teradata SQL-source pattern Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve conflicts in CHANGELOG.md and __version__.py: main advanced to 1.8.0 (PLU-511 ACL digest), so the connector capability-markers enhancement moves to 1.8.1 at the top of the changelog and the version is bumped to 1.8.1. All of main's intervening entries (1.8.0 down through 1.7.11) are preserved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Conflicts were both version-bump collisions: main released 1.9.0 while this branch had bumped 1.8.0 -> 1.8.1. Rebase the branch's patch bump onto the new base (1.9.1) and keep both CHANGELOG entries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only CHANGELOG.md conflicted: main added a `Fixes` entry under the same unreleased 1.9.1 heading this branch already uses for the connector capability-markers enhancement. 1.9.1 is not tagged yet, so both entries belong to that release — keep the branch's `Enhancements` section and append main's `Fixes` section under it. No version change is needed; both sides are already at 1.9.1. main's release fix (hatchling<1.32 pin, pypa/gh-action-pypi-publish v1.14.2, workflow_dispatch trigger) merged cleanly and is byte-identical to origin/main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The main merge folded this branch's changelog entry into the already released 1.9.1 section, so check-version failed on a version already committed to main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would require human review. Adds capability markers/location-identity annotations across 40+ connectors that will drive downstream parameterization; correctness of these annotations requires per-connector domain verification, and the diff is truncated.
Re-trigger cubic
Resolved conflicts: - CHANGELOG.md: kept both, 1.10.0 section above main's 1.9.3/1.9.2 entries - unstructured_ingest/__version__.py: kept 1.10.0 (already ahead of main's 1.9.3) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Shadow auto-approve: would require human review. Adds capability markers with non-None defaults (supports_recursion=True, emits_record_version=False, location_identity paths) that could alter downstream behavior for all connectors, contradicting the 'no behavior change' claim; per-connector annotations are product decisions needing human sign-off.
Re-trigger cubic
Conflicts: - unstructured_ingest/__version__.py, CHANGELOG.md: main released 1.10.0 through 1.10.4, so this branch's unreleased 1.10.0 entry moves to its own 1.11.0 section above main's history. - unstructured_ingest/processes/connectors/sharepoint.py: keep main's Teams targeting (site description note, team_id/channels indexer fields) alongside this branch's x-runtime-eligible markers on site and path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PKcn1ksBdeb7AMD74xZC6o
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Connector capability markers (location shape / identity / record-version)
Adds capability markers to the connector registry so the control plane (platform-api / platform-etl) can derive per-connector parameterization behavior instead of hardcoding it.
What's here
LocationShapeenum + kw-only markers onSourceRegistryEntry/DestinationRegistryEntry:location_shape,location_identity,supports_recursion,emits_record_versionNone(unannotated) so consumers fall back to their hand-seeded table for connectors not yet annotated — no behavior change until each entry is explicitly markedPart of the parameterized-connectors generalization (consumed by platform-api #1210/its follow-up and platform-etl).
🤖 Generated with Claude Code