Hi Chai team,
I am trying to reproduce temporal training/test splits for structure-prediction models and noticed an ambiguity in Chai's mmCIF metadata handling that may also be relevant to benchmark contamination.
In:
chai_lab/data/parsing/structure/all_atom_entity_data.py
StructureMetadata.release_date is populated from:
release_date=_safe_get(
structure.info,
"_pdbx_database_status.recvd_initial_deposition_date"
)
However, _pdbx_database_status.recvd_initial_deposition_date is the initial PDB deposition date, rather than the initial public PDB release date, which is typically represented by the earliest _pdbx_audit_revision_history.revision_date.
Could you clarify whether the reported Chai-1 training-data cutoff was applied using:
the PDB initial deposition date, or
the PDB initial public release date?
Since the full Chai-1 training-data construction pipeline is not part of the public chai-lab repository, I do not want to infer the actual training semantics solely from the public parser code.
The distinction could be important for temporal benchmark validity.
If Chai-1's training set was selected using deposition date, then a structure that was deposited before the training cutoff but publicly released after it could potentially have been available for training. A benchmark selecting "post-training" structures according to release date could therefore unintentionally include structures that were already eligible for the Chai-1 training set.
This raises a possible leakage/contamination concern for external temporal benchmarks such as FoldBench, as well as any Chai-1 internal post-cutoff benchmark, depending on which PDB date field was used to construct those evaluation sets.
I am not claiming that such entries were necessarily included in Chai-1's final training dataset. Rather, I would like to clarify the inclusion criterion so that benchmark authors can correctly determine whether a target is genuinely post-training.
Could you also clarify:
What exact PDB date field was used to construct the Chai-1 structural training dataset?
What date field was used for Chai-1's own post-cutoff evaluation sets?
If deposition date was used for training, were benchmark targets explicitly checked to ensure their deposition dates were also after the cutoff?
If StructureMetadata.release_date intentionally represents deposition date, would it be worth renaming or documenting this field to avoid confusion?
This seems particularly relevant because benchmarking papers often refer generically to a model's "training cutoff" without specifying whether it means deposition or public release.
Thanks for any clarification.
Hi Chai team,
I am trying to reproduce temporal training/test splits for structure-prediction models and noticed an ambiguity in Chai's mmCIF metadata handling that may also be relevant to benchmark contamination.
In:
chai_lab/data/parsing/structure/all_atom_entity_data.py
StructureMetadata.release_date is populated from:
release_date=_safe_get(
structure.info,
"_pdbx_database_status.recvd_initial_deposition_date"
)
However, _pdbx_database_status.recvd_initial_deposition_date is the initial PDB deposition date, rather than the initial public PDB release date, which is typically represented by the earliest _pdbx_audit_revision_history.revision_date.
Could you clarify whether the reported Chai-1 training-data cutoff was applied using:
the PDB initial deposition date, or
the PDB initial public release date?
Since the full Chai-1 training-data construction pipeline is not part of the public chai-lab repository, I do not want to infer the actual training semantics solely from the public parser code.
The distinction could be important for temporal benchmark validity.
If Chai-1's training set was selected using deposition date, then a structure that was deposited before the training cutoff but publicly released after it could potentially have been available for training. A benchmark selecting "post-training" structures according to release date could therefore unintentionally include structures that were already eligible for the Chai-1 training set.
This raises a possible leakage/contamination concern for external temporal benchmarks such as FoldBench, as well as any Chai-1 internal post-cutoff benchmark, depending on which PDB date field was used to construct those evaluation sets.
I am not claiming that such entries were necessarily included in Chai-1's final training dataset. Rather, I would like to clarify the inclusion criterion so that benchmark authors can correctly determine whether a target is genuinely post-training.
Could you also clarify:
What exact PDB date field was used to construct the Chai-1 structural training dataset?
What date field was used for Chai-1's own post-cutoff evaluation sets?
If deposition date was used for training, were benchmark targets explicitly checked to ensure their deposition dates were also after the cutoff?
If StructureMetadata.release_date intentionally represents deposition date, would it be worth renaming or documenting this field to avoid confusion?
This seems particularly relevant because benchmarking papers often refer generically to a model's "training cutoff" without specifying whether it means deposition or public release.
Thanks for any clarification.