Raise Fivetran Tasks and Warnings when a resync is needed by mcrauwel · Pull Request #100 · planetscale/fivetran-source

mcrauwel · 2026-10-26T10:40:44Z

Uses the Fivetran Partner SDK user alerts so customers get an actionable message instead of a generic sync failure when a resync or intervention is needed. No proto bump needed: the pinned SDK version already has Task and Warning in UpdateResponse.

Changes

Tasks (blocking, the sync stops; Fivetran clears them after the next successful sync)

Shard no longer exists (shard provided in VGTID, ..., not found in the ... keyspace), for example after a reshard once the old shards are deleted. New lib.IsShardNotFoundInVGTIDError.
Schema change: the two VStream schema-incompatibility errors we already match (IsVStreamSchemaIncompatibilityError) now raise a Task instead of a bare FailedPrecondition.
Progress is checkpointed before the Task. After sending a Task, Update returns nil; if sending the Task fails it falls back to codes.FailedPrecondition.

Warnings (non-blocking, dismissible)

Stale legacy shards: before Fixing substring keyspace match issue. #91 the show vitess_shards like "%<keyspace>%" query could save shards of another keyspace (e.g. prod_main_sharded/-40 as a shard of prod_main). vtgate rejects these before reading any rows, so no data was affected. StateFromRequest now drops saved shard entries containing / that aren't in ListShards, logs them, and sends one Warning per sync.
Binlog expiry: the existing automatic cursor reset now records the table and shard, and one combined Warning is sent per sync (flushed before a Task if one is raised; not sent if the sync fails).

Customer-facing text

Task, missing shard: Re-sync required: shard "<shard>" no longer exists in PlanetScale keyspace "<keyspace>" (for example after a reshard), so incremental replication cannot resume for the tables in this keyspace. To fix this, go to the connection's Settings tab in Fivetran and click "Resync all data".
Task, schema change: Re-sync required: a schema change on table "<table>" in PlanetScale keyspace "<keyspace>" (for example a column added in the middle of the table, or a dropped or renamed column) prevents incremental replication from continuing. To fix this, go to the connection's Settings tab in Fivetran and click "Resync all data".
Warning, stale shards: PlanetScale removed stale shard references left in this connection's sync state by an older connector version (keyspace <ks>: <shards>). These references never contained data. No action or re-sync is needed.
Warning, binlog expiry: PlanetScale binary logs expired before Fivetran could read the latest changes for the following table(s): <ks.table (shard s)>, .... An automatic historical re-sync has been initiated. No action is needed.

The binlog Warning lists at most 20 table/shard entries and the stale-shards Warning at most 20 shards (across keyspaces), each followed by "and N more". Re-sync is connection-level only for this connector today.

Tests

New tests for every Task and Warning path (alerts_test.go, types_test.go, sync_test.go); TestVStreamSchemaIncompatibilityReturnsFailedPrecondition is now ...RaisesResyncTask. go vet ./cmd/... ./lib/... and go test ./cmd/... ./lib/... pass.

Open questions for Fivetran

If the connector sends a Task and also returns a gRPC error, does Fivetran retry or alert twice?
Are checkpoints sent right before a Task persisted?
What exactly does error_code: BINLOG_EXPIRATION_ERROR (fix(binlog-error): trigger the resync from connector whenever binlog expire error occurs to make MAR free #73) trigger: a MAR-free re-sync, a truncate or soft delete, a dashboard notice? Does it need to be on the final checkpoint?
Can the same mechanism be used for reshards and schema changes?
Does Fivetran already show something when a re-sync starts automatically after binlog expiry? If so, the binlog Warning is redundant.

fivetran-ankitsilla · 2026-10-29T11:54:28Z

+	if len(b.items) == 0 {
+		return ""
+	}
+	return "PlanetScale binary logs expired before Fivetran could read the latest changes for the following table(s): " +


We can change the message here

return "PlanetScale binary logs expired before Fivetran could read the latest changes for the following table(s): " +
joinLimited(b.items) + ". " +
"An automatic historical re-sync has been initiated. No action is needed."

good call, took your wording as is

fixed in 1b96e7c

fivetran-ankitsilla · 2026-10-29T16:24:15Z

+
+func joinLimited(items []string) string {
+	if len(items) <= maxAlertItems {
+		return strings.Join(items, ", ")


Minor question: was the intention to cap the total number of stale shards at 20? Currently, all shards for one keyspace are joined into a single item before joinLimited is called, so the limit applies to keyspaces rather than shards. For example, one keyspace with 25 stale shards would display all 25. This is probably not a blocker given typical shard counts, but the behavior differs from the PR description.

you're right, the cap was on keyspaces since the shards got joined per keyspace first

StaleShardsWarningMessage now caps the total number of shards across keyspaces, so one keyspace with 25 shards shows 20 and "and 5 more"

fixed in 1b96e7c

fivetran-ankitsilla · 2026-10-29T16:39:35Z

+	// Send at most one combined Warning for every table/shard whose cursor was
+	// reset because binlogs expired during this sync (no-op if none, or if it
+	// was already flushed before raising a Task).
+	if fErr := logger.FlushBinlogExpirationWarning(); fErr != nil {


This Warning is flushed even when s.handle() returns an error. For example, binlog expiry may be recorded, historical sync may subsequently fail, and the connector would still send “No action is required” before returning a gRPC error. Should we only flush the success/no-action Warning when err == nil, or use different wording when the sync is failing?

agreed, Handle now only flushes the binlog Warning when the sync succeeds

the trade-off is the affected tables are only kept in memory, so a failed sync won't show the Warning at all

the re-sync still recovers next sync though: either the reset cursor was already checkpointed, or binlogs are still expired and it gets reset again

the flush before a Task stays as it was

fixed in 1b96e7c

fivetran-ankitsilla · 2026-10-29T16:54:34Z

@mcrauwel Have added few comments, please check!

Also, I have answered all the open questions in the slack thread.
Adding here for reference as well:

If the connector sends a Task and also returns a gRPC error, does Fivetran retry or alert twice?
If the Task is successfully delivered first, Fivetran immediately stops processing the Update stream by raising FailureWithTask. It does not subsequently process the returned gRPC error, so it should neither create a second alert nor retry because of that gRPC error. Nevertheless, after sending a Task, the connector should terminate normally and avoid returning another error because the Task is already the terminal outcome.

Are checkpoints sent right before a Task persisted?
Yes checkpoints sent before task are persisted

What exactly does error_code: BINLOG_EXPIRATION_ERROR (fix(binlog-error): trigger the resync from connector whenever binlog expire error occurs to make MAR free #73) trigger: a MAR-free re-sync, a truncate or soft delete, a dashboard notice? Does it need to be on the final checkpoint?
PR #73: BINLOG_EXPIRATION_ERROR does not itself trigger a historical re-sync. Historical recovery is triggered by clearing the expired cursor. The connector then attempts to read from the beginning during the same sync. If the cleared cursor is checkpointed and recovery continues in a later sync, that sync will also start from the cleared cursor.
I checked with the team and found out that the handling regarding BINLOG_EXPIRATION_ERROR for making it MAR free was not taken up. So this marker doesn't make the resync MAR free.
Also FYI: as per the current rules, MAR is not charged for the same data (Primary Key) getting resynced. Only for the newer rows or new PrimaryKey , MAR is charged.
The marker does not itself issue a truncate or soft delete, nor does it create a dashboard Task or Warning. It must be included in checkpointed state for Fivetran to receive it but there is no handling added as of today for this marker.

Can the same mechanism be used for reshards and schema changes?
No, the cursor-reset mechanism could technically be used to start historical recovery for reshards or incompatible schema changes, but BINLOG_EXPIRATION_ERROR should not be reused because it specifically identifies binlog expiry.
For the current reshard case, sending a Task and asking the customer to initiate a re-sync is the safer approach.

Does Fivetran already show something when a re-sync starts automatically after binlog expiry? If so, the binlog Warning is redundant.
No additional customer-facing notice appears to be generated from BINLOG_EXPIRATION_ERROR. We found no handling for this marker in the codebase, and PR #73 itself does not send a Task or Warning. Fivetran may still show the normal historical-sync status, but it does not appear to explain that binlog expiration caused it. Therefore, the proposed binlog Warning present in the PR is not redundant and is useful for giving the customer that context.

…ed sync - Binlog expiry Warning now ends with "An automatic historical re-sync has been initiated. No action is needed." - Stale-shards Warning caps the total number of shards listed (20), across keyspaces, instead of the number of keyspaces. - Only flush the binlog expiry Warning when the sync succeeds, so a failing sync never shows "No action is needed". The historical re-sync still recovers on the next sync. Signed-off-by: Matthias Crauwels <matthias.crauwels@planetscale.com>

Raise Fivetran Tasks and Warnings when a resync is needed

50b1a87

mcrauwel force-pushed the mcrauwel/fivetran-resync-tasks branch from 2ec06f4 to 50b1a87 Compare October 26, 2026 13:44

mcrauwel marked this pull request as ready for review October 26, 2026 13:58

fivetran-ankitsilla reviewed Oct 29, 2026

View reviewed changes

mcrauwel and others added 2 commits October 30, 2026 11:42

Merge branch 'main' into mcrauwel/fivetran-resync-tasks

cf79ca4

Navigation Menu

Repository navigation

Raise Fivetran Tasks and Warnings when a resync is needed - #100

Raise Fivetran Tasks and Warnings when a resync is needed#100
mcrauwel wants to merge 3 commits into
mainfrom
mcrauwel/fivetran-resync-tasks

mcrauwel commented Oct 26, 2026 •

edited

Loading

Uh oh!

fivetran-ankitsilla Oct 29, 2026

Uh oh!

mcrauwel Oct 30, 2026

Uh oh!

fivetran-ankitsilla Oct 29, 2026

Uh oh!

mcrauwel Oct 30, 2026

Uh oh!

fivetran-ankitsilla Oct 29, 2026

Uh oh!

mcrauwel Oct 30, 2026

Uh oh!

fivetran-ankitsilla commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mcrauwel commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Customer-facing text

Tests

Open questions for Fivetran

Uh oh!

fivetran-ankitsilla Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

mcrauwel Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

fivetran-ankitsilla Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

mcrauwel Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

fivetran-ankitsilla Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

mcrauwel Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

fivetran-ankitsilla commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mcrauwel commented Oct 26, 2026 •

edited

Loading