Repository navigation
fix(dataprep): rebuild deces-2026-m09 from INSEE (temporary workaround) - #111
Merged
Merged
Conversation
The data.gouv resource titled deces-2026-m09.txt serves deces-2025-m09.txt (same sha1 8eb03b35, deaths of September 2025). The sync keys files on the URL name, which is already in storage, so September 2026 was never ingested and prod stayed on 2026-m08. INSEE publishes the right month as CSV. A new datagouv-insee-overrides step, run before the data.gouv sync, rebuilds the listed months in the data.gouv fixed-width layout, recodes them to latin1 like the sync does, and stores them under the data.gouv name, so FILES_TO_PROCESS picks them up unchanged. - the conversion reproduces data.gouv deces-2026-m08.txt byte for byte, and the stored deces-2026-m08.txt.gz once recoded - the step fails if the record count differs or if the archive mostly holds deaths of another month than the file name says - it is skipped once the file is in storage, so the remote worker's own full-check is a no-op - the list and the awk script live outside tagfiles.version: removing the entry later does not change DATAPREP_VERSION
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The data.gouv resource titled
deces-2026-m09.txt(2026-10-05) points to…/20261005-122110/deces-2025-m09.txt: byte-identical to the September 2025 file (sha18eb03b35…). The data.gouv sync keys files on the URL name, already in storage, so September 2026 was never ingested; prod still serveslastDataset: 2026-m08. Reported upstream: https://www.data.gouv.fr/fr/datasets/fichier-des-personnes-decedees/discussions/?discussion_id=6ac496085fc76144658f417dINSEE serves the right month (
Deces_2026_M09.zip, CSV, 51 437 records, 42 666 deaths in 2026-09).What
Temporary workaround, scoped by
packages/deces-dataprep/insee-overrides.list:datagouv-insee-overridesstep, prerequisite ofdatagouv-to-storage: for each listed month absent from storage, download the INSEE archive, convert it withinsee-csv-to-txt.awk(awk, no new dependency), recode to latin1 as the sync does, gzip, push under the data.gouv namefull-check)tagfiles.version: dropping the entry once data.gouv is fixed does not changeDATAPREP_VERSIONValidation
Deces_2026_M08.csvconverted == data.gouvdeces-2026-m08.txtbyte for byte (sha10d2a9fb7…), and == storeddeces-2026-m08.txt.gzonce recoded to latin1deces-2026-m09.txt.gz(51 437 records), skips on second run, fails on a month mismatchExpected effect
deces-2026-m09.txt.gzentersFILES_TO_PROCESS→ new data hash (and newDATAPREP_VERSION, Makefile is tagged) → full rebuild and prod release on the nextdataprep-monthlyrun.