Eli/extend parser with rubrics - #3108
Open
eli-miriam wants to merge 20 commits into
Open
Conversation
Currently it is not possible to apply migrations to coding system release databases. This is for two reasons. Firstly the `database_ready` method of the "versioning" app model, returns False if any migrations are pending and thus the coding system database connections are not added to django's collection of available database connections. This commit changes this behaviour such that it only returns False if there is a pending migration for any app other than one of the coding systems (which is closer to the intent stated in its original docstring). Secondly, even with this fix,`update_coding_system_database_connections` is not called by the default django `migrate` command, so trying to pass anything other than the default database to the --database arg fails. This commit includes a custom management command which updates the database connections, then iterates over every release database, applying either a specified migration or all pending ones (mimicking the behaviour of the built-in migrate command). 0
Adds new model classes with fields as per final model. Preserves some fields on the Concept model that will ultimately move to the newly-introduced ConceptEdition model class to avoid data loss.
Data migration to move data from the old Concept fields into their replacement counterparts on the ConceptEdition model. Creates an Edition for the previously-loaded 2019 edition with appropriate metadata.
Removes transitional fields now that the data migration has moved their data to its new home.
```
import json
from pathlib import Path
fixture_path = Path(
"coding_systems/icd10/fixtures/icd10.icd10_test_20200101.json"
)
fixture = json.load(fixture_path.open())
editions = [
{
"model": "icd10.edition",
"pk": "test",
"fields": {"version": 1, "year": 1900, "source_description": "test fixture"},
}
]
concepts = [f for f in fixture if f["model"] == "icd10.concept"]
concepteditions = [
{
"model": "icd10.conceptedition",
"pk": i,
"fields": {k: v for k, v in c["fields"] if k in ["kind", "term"]}
| {"edition_id": "test", "concept_id": c["pk"]},
}
for i, c in enumerate(concepts, 1)
]
concepts = [
c | {"fields": {fk: fv for fk, fv in c["fields"].items() if fk == "parent"}}
for c in concepts
]
fixture_new = editions + concepts + concepteditions
json.dump(fixture_new, fixture_path.open("w"))
```
- Implemented a new parser for WHO ICD-10 ClaML XML to extract codes and their relationships. - Added a data downloader to fetch and extract ICD-10 ClaML ZIP files for 2016 and 2019 editions. - Updated import_data function to handle new data sources and and the new data model
…arse is now a "pure" claml parser, and at a later point in the pipeline for the 2016 release we will apply the NHS-specific changes - Moved code normalization into the claml_parser rather than the importer - Fixed a bug where some 5 character codes (e.g. M0000) had there 3-character root (M00) as their parent instead of M000 - Also return the place modifiers from the parsed claml so we can use these in future processing - Update data_downloader so that if a release dir isn't passed it defaults to ./coding_systems/icd10/data for local dev work
This is limited to applying the place of occurrence modifier to all 3 char codes in the range W00-Y34 (except Y06 and Y07) as a 4th character. This overlaps with a handful of actual 4 character codes in that chapter and we assert that the overlap is exactly as expected.
All differences must be accounted for or it will complain
All differences must be accounted for or it will complain
rw251
reviewed
Jun 10, 2026
| "note", | ||
| "exclusion", | ||
| "inclusion", | ||
| "preferredlong", |
Contributor
There was a problem hiding this comment.
Think the l is capitalised. But actually, maybe easiest to just remove preferred and preferredLong as the code skips if it one of these so will never match this set.
Contributor
There was a problem hiding this comment.
Or, better yet, when we rebase we can just use the RubricKind class from the model
rw251
force-pushed
the
rw/add-icd10-data
branch
3 times, most recently
from
June 12, 2026 10:09
03c954a to
d25fcd0
Compare
rw251
force-pushed
the
rw/add-icd10-data
branch
4 times, most recently
from
June 26, 2026 09:43
8ae15a1 to
3bd3894
Compare
Jongmassey
force-pushed
the
rw/add-icd10-data
branch
2 times, most recently
from
July 15, 2026 13:15
454a77f to
914e4b2
Compare
Jongmassey
force-pushed
the
icd-10-multi-editions
branch
from
July 16, 2026 16:30
fc8b873 to
d92135c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extend the ClaML parser to handle rubrics, and to handle codes like
A52.7where it has a dagger usage denoted inFragmentsections of its code but not in its top-level element.Currently the data importer does not do anything with the newly added rubrics.