Add sync_tables helper for multi-server base_schemas sync#6
Add sync_tables helper for multi-server base_schemas sync#6
Conversation
|
@arturoptophys, which are the base_schemas tables we should sync from (server3)? Then |
|
@lecriste one-way S2-S1 -> MouseScoreSheet_WaterRestriction, MouseScoreSheet, Session, SessionScoreSheet. Here order is important so SessionScoreSheet need to be last. // Here there is no need to sync S1 entries into mouse_ar pipeline. both-ways -> Surgery |
|
|
||
| rows = src_table.fetch(as_dict=True) | ||
| before = len(tgt_table) | ||
| with tgt_table.connection.transaction: |
There was a problem hiding this comment.
I think a single insert is anyway wrapped in a transaction. The questions is should there be a try-except to continue with other tables if one errors or is this intended to fail completely if something goes wrong?
There was a problem hiding this comment.
OK, removed: 67187e6
The with ... .transaction: would be useful in case DataJoint internally splits the call into multiple statements (e.g. master + part tables, or batch-splitting on large payloads), but it's not the case for the flat tables of base_schemas.
There was a problem hiding this comment.
In the current behaviour (propagate and abort), tables are processed in FK order: if Mouse fails, continuing on to Session would hit FK errors anyway.
Do you think it's better to fail hard early on any table with dependents?
Thanks @arturoptophys! So the steps will be:
Can you confirm? |
This PR introduces
base_schemas/scripts/sync.pywith async_tables()function that copies rows of base_schemas tables between two DataJoint servers usingdj.Instance(DataJoint 2.2+). Callers pass two connection-config dicts and a list of table classes; the helper builds the instances and usesFreeTable(full_table_name)to resolve tables on each side. Supports per-table restrictions for incremental syncs, wraps each insert in a target-side transaction, and is idempotent viaskip_duplicates=True.Changes
base_schemas/scripts/sync.py— new helper module.tests/test_sync.py— 6 unit tests (mocked, no live MySQL), all passing.pyproject.toml— bumpeddatajoint>=2.2.Test plan
pytest tests/test_sync.py— 6/6 pass