Skip to content

feat(analyzer): Add Healthcare identifiers recognizer - #2159

Open
bhargavikalicheti wants to merge 19 commits into
data-privacy-stack:mainfrom
bhargavikalicheti:feature/us-healthcare-recognizer
Open

feat(analyzer): Add Healthcare identifiers recognizer#2159
bhargavikalicheti wants to merge 19 commits into
data-privacy-stack:mainfrom
bhargavikalicheti:feature/us-healthcare-recognizer

Conversation

@bhargavikalicheti

@bhargavikalicheti bhargavikalicheti commented Jul 12, 2026

Copy link
Copy Markdown

Change Description

Adds conservative, context-aware US healthcare identifier recognizers to Presidio Analyzer.

New disabled-by-default recognizers:

  • US_HEALTH_INSURANCE_MEMBER_ID
  • US_PRIOR_AUTHORIZATION_NUMBER
  • US_CLAIM_NUMBER
  • US_PRESCRIPTION_NUMBER
  • US_REFERRAL_NUMBER
  • US_PROVIDER_TAX_ID

These recognizers combine plausible identifier patterns with healthcare/insurance context enhancement and entity-specific score thresholds to reduce false positives. Pattern-only matches remain available to callers that explicitly lower the analysis threshold, such as structured-data workflows.

Issue reference

Fixes Feature Request: Healthcare Recognizer for Common Healthcare Identifiers (Member ID, Claims, Prior Authorization, etc.)
#2136

Checklist

  • I have reviewed the contribution guidelines
  • I agree to follow this project's Code of Conduct
  • I confirm that I have the right to submit this contribution and that it does not knowingly contain proprietary or confidential code.
  • My code includes unit tests
  • All unit tests and lint checks pass locally
  • My PR contains documentation updates / additions if required

@bhargavikalicheti

Copy link
Copy Markdown
Author

Hi @SharonHart @omri374 , No rush - gentle reminder on PR whenever you get a chance. Thank you!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a set of conservative, disabled-by-default US healthcare identifier recognizers to Presidio Analyzer, aiming to detect common healthcare administrative IDs only when appropriate workflow context is present (to reduce false positives in general alphanumeric/ID-like text).

Changes:

  • Introduces new US healthcare admin ID recognizers (claim, prior auth, prescription, referral, provider tax ID) plus a health insurance member ID recognizer, all requiring nearby context.
  • Wires the new recognizers into predefined recognizer exports and default registry YAML (disabled by default).
  • Adds unit tests, supported-entities documentation entries, and changelog notes for the new entities/recognizers.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
presidio-analyzer/tests/test_us_healthcare_admin_recognizers.py New tests for healthcare admin ID recognizers (positive/negative context + metadata).
presidio-analyzer/tests/test_us_health_insurance_member_id_recognizer.py New tests for health insurance member ID recognizer detection behavior and metadata.
presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/us/us_healthcare_admin_recognizers.py Adds context-required pattern recognizer base + concrete admin ID recognizers.
presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/us/us_health_insurance_member_id_recognizer.py Adds context-required member/subscriber ID recognizer with negative-context pruning.
presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/us/init.py Exports the new US healthcare recognizers from the US package.
presidio-analyzer/presidio_analyzer/predefined_recognizers/init.py Exposes the new recognizers via the top-level predefined_recognizers import surface.
presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml Registers the recognizers as predefined + disabled-by-default with country_code: us.
docs/supported_entities.md Documents the new supported entity types and brief descriptions.
CHANGELOG.md Notes new analyzer recognizers under Unreleased.

@omri374 omri374 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Please add references to be able to trace where the regex pattern is coming from, and see the comment around context management.

@bhargavikalicheti

Copy link
Copy Markdown
Author

@SharonHart @omri374 Hi! Just a friendly follow up on my PR whenever you have a chance. I'd appreciate a review when your schedule allows. Please let me know if there are any changes you'd like me to make. Thanks!

@bhargavikalicheti

Copy link
Copy Markdown
Author

@omri374 @SharonHart - Hi! just checking if you are okay with this and ready to merge please?

omri374 added 2 commits August 4, 2026 11:25
Added various disabled-by-default recognizers for US and South African IDs, including health insurance member IDs, claim numbers, and UUID detection. Introduced NoOpNlpEngine for standalone recognizers.

@omri374 omri374 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Left a few comments, hopefully all are easy to change and clear.

Comment thread presidio-analyzer/tests/test_us_health_insurance_member_id_recognizer.py Outdated
"Claim number BCBSM1234567 was denied",
],
)
def test_when_member_id_lacks_insurance_context_then_below_threshold(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other gaps worth covering: lowercase and mixed case inputs; multiple IDs in one text; trailing punctuation; negative pattern cases for the admin recognizers (PA-12345 too short, PA-1234567890123 too long)

Comment thread presidio-analyzer/tests/mocks/nlp_engine_mock.py Outdated
Comment thread docs/supported_entities.md Outdated
@bhargavikalicheti

Copy link
Copy Markdown
Author

Hi @omri374, thank you for the detailed review. I’ve addressed the feedback:
clarified the identifier sources and pattern rationale
reduced broad member ID confidence
anchored administrative IDs to appropriate labels
restricted Provider EIN matching to valid IRS prefixes
replaced exact floating-point assertions with pytest.approx
expanded positive, negative, boundary, casing, punctuation, and multiple-ID tests
replaced the custom NLP mock with the existing spaCy fixture
removed the healthcare administrative base class
corrected the documentation and PR description to describe context enhancement and entity thresholds, without claiming negative-context checks
The PR is ready for another review whenever you have a chance. Thank you!

@omri374

omri374 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Thank you @bhargavikalicheti. Could you please fix the small conflict on the analyzer yaml?

@bhargavikalicheti

Copy link
Copy Markdown
Author

  • A conversation must be resolved before this pull request can be merged. At least 1 approving review is required by reviewers with write access.
  • All comments must be resolved.

done @omri374. Thanks.

Comment thread presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

CHANGELOG.md:10

  • CONTRIBUTING.md explicitly asks contributors not to update CHANGELOG.md in PRs (changelog entries are generated during release). Please drop these new bullet points from CHANGELOG.md to avoid merge conflicts.
- Added a disabled-by-default US health insurance member ID (`US_HEALTH_INSURANCE_MEMBER_ID`) recognizer requiring healthcare or insurance context.
- Added disabled-by-default US healthcare administrative ID recognizers for claim numbers, prior authorization numbers, prescription numbers, provider tax IDs, and referral numbers.

enabled: false
country_code: us

- name: UsHealthInsuranceMemberIdRecognizer

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enabling these recognizers via YAML silently wipes their score thresholds (applies to all six new entries)

Enabling any of these six recognizers via this config which is the only supported path, since they ship enabled: false — erases the score thresholds the constructors set. RecognizerListLoader.get() runs recognizer.score_thresholds = normalize_score_thresholds(conf.get("score_thresholds")) (recognizers_loader_utils.py:435), which turns the absent key into {} and overwrites the {'US_CLAIM_NUMBER': 0.6}-style defaults from __init__.

Verified with a real AnalyzerEngine: after flipping enabled: true, "Tracking number CLM456789123 is active"US_CLAIM_NUMBER at 0.1, and "v2patch10build7"US_HEALTH_INSURANCE_MEMBER_ID at 0.1 — exactly the false positives the PR's tests assert are suppressed (those tests use direct instantiation + add_recognizer, so they never hit this).

Two possible fixes: declare score_thresholds explicitly on these six yaml entries, or make the loader only assign when the conf actually provides thresholds. The second also fixes this for user-supplied configs that omit the key.

r"(?<=\b(?:(?:(?:billing|rendering|healthcare)\s+provider|"
r"provider\s+organization|provider)\s+(?:tax\s*(?:id|number|"
r"identification\s+number)|tin|ein)|billing\s+provider)"
r"(?:\s*:\s*|\s+))" + VALID_EIN_PREFIX + r"-\d{7}\b",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIN labelled pattern misses TIN# / EIN No. forms

This lookbehind only allows : or whitespace after the label ((?:\s*:\s*|\s+)), while every sibling recognizer also accepts a #|no\.?|number|id separator. So "Provider TIN# 12-3456789" and "Billing provider EIN No. 12-3456789" miss the labelled 0.35 pattern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants