Lutaml integration - #175
Draft
andrew2net wants to merge 64 commits into
Draft
Conversation
![hound[bot]](https://avatars.githubusercontent.com/in/3598?s=60&v=4){kind=small}
There was a problem hiding this comment.
Some files could not be reviewed due to errors:
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unre...
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unrecognized cop Performance/Count found in .rubocop.yml, unrecognized cop Performance/Detect found in .rubocop.yml, unrecognized cop Performance/FlatMap found in .rubocop.yml, unrecognized cop Performance/ReverseEach found in .rubocop.yml, unrecognized cop Performance/Size found in .rubocop.yml, unrecognized cop Performance/StringReplacement found in .rubocop.yml
There was a problem hiding this comment.
Some files could not be reviewed due to errors:
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unre...
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unrecognized cop Performance/Count found in .rubocop.yml, unrecognized cop Performance/Detect found in .rubocop.yml, unrecognized cop Performance/FlatMap found in .rubocop.yml, unrecognized cop Performance/ReverseEach found in .rubocop.yml, unrecognized cop Performance/Size found in .rubocop.yml, unrecognized cop Performance/StringReplacement found in .rubocop.yml
There was a problem hiding this comment.
Some files could not be reviewed due to errors:
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unre...
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unrecognized cop Performance/Count found in .rubocop.yml, unrecognized cop Performance/Detect found in .rubocop.yml, unrecognized cop Performance/FlatMap found in .rubocop.yml, unrecognized cop Performance/ReverseEach found in .rubocop.yml, unrecognized cop Performance/Size found in .rubocop.yml, unrecognized cop Performance/StringReplacement found in .rubocop.yml
There was a problem hiding this comment.
Some files could not be reviewed due to errors:
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unre...
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unrecognized cop Performance/Count found in .rubocop.yml, unrecognized cop Performance/Detect found in .rubocop.yml, unrecognized cop Performance/FlatMap found in .rubocop.yml, unrecognized cop Performance/ReverseEach found in .rubocop.yml, unrecognized cop Performance/Size found in .rubocop.yml, unrecognized cop Performance/StringReplacement found in .rubocop.yml
The openssl 3.3.0 causes OpenSSL::SSL::SSLError error Some small bugfixes have also been done
…; update scraper to use except method for params; modify bibliography_spec to enable tests and adjust expectations; fix scraper_spec to enable isoref test; update VCRs
…iso dependency version to 1.15.8
…djust VCR re_record_interval to 7 days
…pubid Add handling for Pubid::Core::Identifier objects in HitCollection#find and create_pubid methods. This allows the code to work with Pubid::Core::Identifier instances directly and use Relaton::Index binary search for better performance when possible. Using ID keys for conrol of index ID scructure. Cache parsed index data in tests to avoid re-parsing the zip/YAML every test, improving test performance.
* update Gemfile to use GH version of relaton-bib & lutaml-model 0.8.0 * fix: update document identifier to use pubid for consistency in file writing * feat: add model ItemData to Bibdata and Bibitem classes; update schema version mapping in Ext class * Update VCRs * refactor: update ItemBase class to inherit from Lutaml::Model::Serializable and remove unnecessary attributes * chore: remove stale commented-out attribute mutation in Contributor Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add key_value mapping for attributes in Ext class * chore: remove unused create_relation method from ItemData class * fix Gemfile --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…for 2.1.0 release
Replace the ICS-page scraper-driven DataFetcher (and its persistent queue/threaded model) with a streaming consumer of `iso_deliverables_metadata.jsonl` and `iso_technical_committees.jsonl`, plus a new DataParser that converts one record into an Iso::ItemData. Adds `iso-open-data` (incremental, gated on upstream Last-Modified) and `iso-open-data-all` (full refresh) source modes; Scraper is retained only as a fallback for `Bibliography.get` lookups missing from the curated index. Refreshes fixtures, cassettes, and docs accordingly.
Build a publicationDate index alongside the reference index so DataParser can stamp each emitted relation's bibitem with a `published` date when the related document is itself present in the Open Data feed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Align spec fixtures and assertions with the Open Data ingest output: relation types use obsoletedBy/updatedBy, abstracts are flattened, place uses <city>, and the corrected-date case is skipped since Open Data does not expose it.
Open Data emits stub records with a `Withdrawn` reference prefix for abandoned projects (publicationDate: null, stage *.98). The previous "Withdrawn" → "ISO" rewrite produced strings like "ISO 1701/Add 1" that pubid-iso can't parse, leaving String docids in the index and crashing `index.save` whenever a Pubid entry sorted ahead of them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picks up the `<date type="published">` elements that Open Data ingest now attaches to related bibitems, and re-records the HTTP cassettes that drifted in the meantime. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ISO's Azure Blob endpoint intermittently returns HTTP 403 AuthorizationFailure, causing the scheduled crawler to fail. Retry up to 4 times (30/60/120/240s) before raising.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Amendments, corrigendums and supplements carry no year on their own pubid identifier — the year lives on the underlying standard. When a publication-date filter is active, `get` routes through `find_match_by_date` -> `year_in_range?`, which bails on `return false if year.zero?`. For these references `hit_year` was 0, so they were excluded from every date-filtered query regardless of the filter year. This surfaced via Metanorma, which derives a publication-date filter from the citing document's `:copyright-year:`/`:created-date:` attributes: `ISO 32000-2:2020/DAM 1` resolved fine with `relaton fetch` but came back "Not found" when cited from such a document. Fall back to `hit.pubid.root.year`, which walks the full base chain (handling nested supplement/corrigendum-of-amendment), so the year of the underlying standard is used when the reference has none of its own.
The year-filter fix lets amendments reach `fetch_and_check_date`. When the index references a data file that 404s, `hit.item` returns an item with no docidentifier and the method crashed on `ret.docidentifier.first.content`. Guard against it so the candidate is skipped and the lookup degrades to "not found" instead of raising.
metanorma-iso#1562 introduced the Supplement ISO deliverable (PubID `ISO …/Suppl N:YYYY`). relaton-iso's doctype model did not recognise `supplement`, so parsing or citing a Supplement bibliographic item failed validation against `TYPES`. - Add `supplement` to `Doctype::TYPES`. - Map the Open Data `supplementType` code `Suppl` to `supplement` in `DataParser::SUPPLEMENT_DOCTYPES`, matching pubid-iso's canonical abbreviation. - Cover the `Suppl` (and previously-untested `Add`) mappings in the data_parser spec. Fixes #185 Refs metanorma/metanorma-iso#1562
* Add `extract` to the ISO doctype TYPES enum (#188) ISO's `extract` deliverable (PubID `ISO …/Ext N:YYYY`) is now emitted as a doctype by metanorma-iso and rendered by pubid-iso (`Pubid::Iso::Identifier::Extract`), but relaton-iso did not recognise `extract`, so parsing or citing an ISO extract failed validation against `Doctype::TYPES`. This is the exact parallel of #186 (supplement). - lib/relaton/iso/model/doctype.rb — add `extract` to the TYPES allow-list. - lib/relaton/iso/data_parser.rb — map Open Data `supplementType` code `Ext` to `extract` in SUPPLEMENT_DOCTYPES. - spec/relaton/iso/data_parser_spec.rb — assert the `Ext => extract` mapping. Refs metanorma/metanorma-iso#1562 * Require pubid-iso ~> 1.15.20 for editionless amendment URN fix pubid-iso 1.15.20 (pubid/pubid#80) renders the URN for amendments and corrigenda whose base document has no edition instead of raising NoEditionError. Without it, serializing a working-draft amendment such as ISO/IEC 23008-1/WD Amd 1 (via to_most_recent_reference -> deep_clone) crashes, breaking bibliography_spec.rb's "fetch WD Amd" example.
…190) * Strip year for undated ISO citations under a publication-date cutoff A publication-date cutoff (publication_date_before/after) must only bound which edition is selected, not force the year to be retained. Previously the mere presence of a cutoff made `get_all` true, so `get` returned the dated item and skipped `to_most_recent_reference`, rendering an undated citation such as `ISO 10303-41` as `ISO 10303-41:2022`. Move relation date-filtering out of the `get_all` branch so it still runs under a cutoff, and drop the two publication_date_* disjuncts from `get_all`. The year is now retained only for a genuinely dated citation (explicit year, keep_year, or all_parts); otherwise the identifier is stripped to its undated form. Adds regression specs for the yearless-query-plus-cutoff case and its keep_year opt-in. * Rewind withdrawn status to the cutoff for as-of-date references A standard withdrawn by a successor published after the cutoff was, as of the cutoff, still the current (published) edition. relaton stores only the current stage with no history, so a now-withdrawn standard came back as stage 95.99 (withdrawn) even when the citing document predates the withdrawal. In filter_item_by_date, when every obsoletedBy successor postdates the cutoff, rewind the item's stage 95.99 -> 60.60 before the (existing) relation stripping removes those successors. A withdrawal with no obsoletedBy successor, or one that predates the cutoff, is left untouched. Refs metanorma/metanorma-standoc#941
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.