feat(analyzer): Add Healthcare identifiers recognizer - #2159
feat(analyzer): Add Healthcare identifiers recognizer#2159bhargavikalicheti wants to merge 19 commits into
Conversation
|
Hi @SharonHart @omri374 , No rush - gentle reminder on PR whenever you get a chance. Thank you! |
There was a problem hiding this comment.
Pull request overview
Adds a set of conservative, disabled-by-default US healthcare identifier recognizers to Presidio Analyzer, aiming to detect common healthcare administrative IDs only when appropriate workflow context is present (to reduce false positives in general alphanumeric/ID-like text).
Changes:
- Introduces new US healthcare admin ID recognizers (claim, prior auth, prescription, referral, provider tax ID) plus a health insurance member ID recognizer, all requiring nearby context.
- Wires the new recognizers into predefined recognizer exports and default registry YAML (disabled by default).
- Adds unit tests, supported-entities documentation entries, and changelog notes for the new entities/recognizers.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| presidio-analyzer/tests/test_us_healthcare_admin_recognizers.py | New tests for healthcare admin ID recognizers (positive/negative context + metadata). |
| presidio-analyzer/tests/test_us_health_insurance_member_id_recognizer.py | New tests for health insurance member ID recognizer detection behavior and metadata. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/us/us_healthcare_admin_recognizers.py | Adds context-required pattern recognizer base + concrete admin ID recognizers. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/us/us_health_insurance_member_id_recognizer.py | Adds context-required member/subscriber ID recognizer with negative-context pruning. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/us/init.py | Exports the new US healthcare recognizers from the US package. |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/init.py | Exposes the new recognizers via the top-level predefined_recognizers import surface. |
| presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml | Registers the recognizers as predefined + disabled-by-default with country_code: us. |
| docs/supported_entities.md | Documents the new supported entity types and brief descriptions. |
| CHANGELOG.md | Notes new analyzer recognizers under Unreleased. |
omri374
left a comment
There was a problem hiding this comment.
Thanks! Please add references to be able to trace where the regex pattern is coming from, and see the comment around context management.
|
@SharonHart @omri374 Hi! Just a friendly follow up on my PR whenever you have a chance. I'd appreciate a review when your schedule allows. Please let me know if there are any changes you'd like me to make. Thanks! |
|
@omri374 @SharonHart - Hi! just checking if you are okay with this and ready to merge please? |
Added various disabled-by-default recognizers for US and South African IDs, including health insurance member IDs, claim numbers, and UUID detection. Introduced NoOpNlpEngine for standalone recognizers.
omri374
left a comment
There was a problem hiding this comment.
Thanks! Left a few comments, hopefully all are easy to change and clear.
| "Claim number BCBSM1234567 was denied", | ||
| ], | ||
| ) | ||
| def test_when_member_id_lacks_insurance_context_then_below_threshold( |
There was a problem hiding this comment.
Other gaps worth covering: lowercase and mixed case inputs; multiple IDs in one text; trailing punctuation; negative pattern cases for the admin recognizers (PA-12345 too short, PA-1234567890123 too long)
|
Hi @omri374, thank you for the detailed review. I’ve addressed the feedback: |
|
Thank you @bhargavikalicheti. Could you please fix the small conflict on the analyzer yaml? |
done @omri374. Thanks. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (1)
CHANGELOG.md:10
- CONTRIBUTING.md explicitly asks contributors not to update CHANGELOG.md in PRs (changelog entries are generated during release). Please drop these new bullet points from CHANGELOG.md to avoid merge conflicts.
- Added a disabled-by-default US health insurance member ID (`US_HEALTH_INSURANCE_MEMBER_ID`) recognizer requiring healthcare or insurance context.
- Added disabled-by-default US healthcare administrative ID recognizers for claim numbers, prior authorization numbers, prescription numbers, provider tax IDs, and referral numbers.
| enabled: false | ||
| country_code: us | ||
|
|
||
| - name: UsHealthInsuranceMemberIdRecognizer |
There was a problem hiding this comment.
Enabling these recognizers via YAML silently wipes their score thresholds (applies to all six new entries)
Enabling any of these six recognizers via this config which is the only supported path, since they ship enabled: false — erases the score thresholds the constructors set. RecognizerListLoader.get() runs recognizer.score_thresholds = normalize_score_thresholds(conf.get("score_thresholds")) (recognizers_loader_utils.py:435), which turns the absent key into {} and overwrites the {'US_CLAIM_NUMBER': 0.6}-style defaults from __init__.
Verified with a real AnalyzerEngine: after flipping enabled: true, "Tracking number CLM456789123 is active" → US_CLAIM_NUMBER at 0.1, and "v2patch10build7" → US_HEALTH_INSURANCE_MEMBER_ID at 0.1 — exactly the false positives the PR's tests assert are suppressed (those tests use direct instantiation + add_recognizer, so they never hit this).
Two possible fixes: declare score_thresholds explicitly on these six yaml entries, or make the loader only assign when the conf actually provides thresholds. The second also fixes this for user-supplied configs that omit the key.
| r"(?<=\b(?:(?:(?:billing|rendering|healthcare)\s+provider|" | ||
| r"provider\s+organization|provider)\s+(?:tax\s*(?:id|number|" | ||
| r"identification\s+number)|tin|ein)|billing\s+provider)" | ||
| r"(?:\s*:\s*|\s+))" + VALID_EIN_PREFIX + r"-\d{7}\b", |
There was a problem hiding this comment.
TIN labelled pattern misses TIN# / EIN No. forms
This lookbehind only allows : or whitespace after the label ((?:\s*:\s*|\s+)), while every sibling recognizer also accepts a #|no\.?|number|id separator. So "Provider TIN# 12-3456789" and "Billing provider EIN No. 12-3456789" miss the labelled 0.35 pattern.
Change Description
Adds conservative, context-aware US healthcare identifier recognizers to Presidio Analyzer.
New disabled-by-default recognizers:
US_HEALTH_INSURANCE_MEMBER_IDUS_PRIOR_AUTHORIZATION_NUMBERUS_CLAIM_NUMBERUS_PRESCRIPTION_NUMBERUS_REFERRAL_NUMBERUS_PROVIDER_TAX_IDThese recognizers combine plausible identifier patterns with healthcare/insurance context enhancement and entity-specific score thresholds to reduce false positives. Pattern-only matches remain available to callers that explicitly lower the analysis threshold, such as structured-data workflows.
Issue reference
Fixes Feature Request: Healthcare Recognizer for Common Healthcare Identifiers (Member ID, Claims, Prior Authorization, etc.)
#2136
Checklist