-
Notifications
You must be signed in to change notification settings - Fork 1.3k
feat(analyzer): Add Healthcare identifiers recognizer #2159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
bhargavikalicheti
wants to merge
19
commits into
data-privacy-stack:main
Choose a base branch
from
bhargavikalicheti:feature/us-healthcare-recognizer
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
370d464
Healthcare recognizers
bhargavikalicheti a5ecee2
Merge branch 'main' into feature/us-healthcare-recognizer
SharonHart c67e7a3
fix(analyzer): use thresholds for healthcare recognizers
bhargavikalicheti 589c043
docs(analyzer): cite healthcare identifier sources
bhargavikalicheti 36d96de
Merge branch 'main' into feature/us-healthcare-recognizer
omri374 5c3640b
Merge branch 'main' into feature/us-healthcare-recognizer
omri374 cef2f77
Update CHANGELOG with new recognizers and features
omri374 d3ae87c
docs(analyzer): clarify member ID references
bhargavikalicheti 522e3ec
fix(analyzer): lower member ID base confidence
bhargavikalicheti 7ebf31d
fix(analyzer): lower member ID base confidence
bhargavikalicheti 2aa4fb2
fix(analyzer): anchor healthcare IDs on labels
bhargavikalicheti 9fa1d57
fix(analyzer): validate provider EIN prefixes
bhargavikalicheti 2ed83f3
test(analyzer): use approximate context score
bhargavikalicheti 8ed30de
test(analyzer): cover healthcare ID edge cases
bhargavikalicheti d2fb76b
test(analyzer): use spaCy for healthcare context
bhargavikalicheti a41f7e7
refactor(analyzer): flatten healthcare recognizers
bhargavikalicheti f2d4dad
docs(analyzer): clarify healthcare detection
bhargavikalicheti 68c775b
Merge branch 'main' into feature/us-healthcare-recognizer
bhargavikalicheti b52181a
Updating the yaml with changes that couldve lost during merge
bhargavikalicheti File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
75 changes: 75 additions & 0 deletions
75
...er/predefined_recognizers/country_specific/us/us_health_insurance_member_id_recognizer.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| """Recognizer for US health insurance member identifiers.""" | ||
|
|
||
| from typing import Dict, List, Optional | ||
|
|
||
| from presidio_analyzer import Pattern, PatternRecognizer | ||
|
|
||
|
|
||
| class UsHealthInsuranceMemberIdRecognizer(PatternRecognizer): | ||
| """Recognize US health insurance member/subscriber IDs with context. | ||
|
|
||
| US health insurance member identifiers are payer-specific and do not have a | ||
| single universal checksum or format. To avoid broad matching of generic | ||
| alphanumeric IDs, this recognizer requires both: | ||
| - a plausible alphanumeric member ID pattern, and | ||
| - nearby healthcare/insurance context. | ||
|
|
||
| CMS consumer guidance explicitly labels the payer-assigned member number on | ||
| a sample insurance card. Medicaid T-MSIS defines MEMBER-ID as the value shown | ||
| on the insurance carrier's card and permits up to 20 characters. These | ||
| sources establish the identifier and upper bound, not a universal syntax; | ||
| the default regex is therefore a conservative, replaceable heuristic. | ||
| Presidio applies ``re.IGNORECASE`` through its default global regex flags, | ||
| so the uppercase character classes also match lowercase and mixed-case IDs. | ||
|
|
||
| CMS card reference: https://www.cms.gov/files/document/11818-sample-insurance-card-english.pdf | ||
| Medicaid data reference: https://www.medicaid.gov/tmsis/dataguide/v4/data-elements/tpl003036/ | ||
|
|
||
| :param patterns: List of patterns to be used by this recognizer | ||
| :param context: List of context words which increase detection confidence | ||
| :param supported_language: Language this recognizer supports | ||
| :param supported_entity: The entity this recognizer can detect | ||
| :param score_thresholds: Optional default and entity-specific score thresholds | ||
| """ | ||
|
|
||
| COUNTRY_CODE = "us" | ||
|
|
||
| PATTERNS = [ | ||
| Pattern( | ||
|
omri374 marked this conversation as resolved.
|
||
| "Health insurance member ID (weak)", | ||
| r"\b(?=[A-Z0-9-]{6,20}\b)(?=[A-Z0-9-]*[A-Z])" | ||
| r"(?=[A-Z0-9-]*\d)[A-Z]{1,5}-?[A-Z0-9]{5,14}\b", | ||
| 0.1, | ||
| ), | ||
| ] | ||
|
|
||
| CONTEXT = [ | ||
| "member", | ||
| "subscriber", | ||
| "insurance", | ||
| "policy", | ||
| ] | ||
|
|
||
| def __init__( | ||
| self, | ||
| patterns: Optional[List[Pattern]] = None, | ||
| context: Optional[List[str]] = None, | ||
| supported_language: str = "en", | ||
| supported_entity: str = "US_HEALTH_INSURANCE_MEMBER_ID", | ||
| name: Optional[str] = None, | ||
| score_thresholds: Optional[Dict[str, float]] = None, | ||
| ): | ||
| patterns = patterns if patterns else self.PATTERNS | ||
| context = context if context else self.CONTEXT | ||
| super().__init__( | ||
| supported_entity=supported_entity, | ||
| patterns=patterns, | ||
| context=context, | ||
| supported_language=supported_language, | ||
| name=name, | ||
| ) | ||
| self.score_thresholds = ( | ||
| score_thresholds | ||
| if score_thresholds is not None | ||
| else {supported_entity: 0.4} | ||
| ) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Enabling these recognizers via YAML silently wipes their score thresholds (applies to all six new entries)
Enabling any of these six recognizers via this config which is the only supported path, since they ship
enabled: false— erases the score thresholds the constructors set.RecognizerListLoader.get()runsrecognizer.score_thresholds = normalize_score_thresholds(conf.get("score_thresholds"))(recognizers_loader_utils.py:435), which turns the absent key into{}and overwrites the{'US_CLAIM_NUMBER': 0.6}-style defaults from__init__.Verified with a real
AnalyzerEngine: after flippingenabled: true,"Tracking number CLM456789123 is active"→US_CLAIM_NUMBERat 0.1, and"v2patch10build7"→US_HEALTH_INSURANCE_MEMBER_IDat 0.1 — exactly the false positives the PR's tests assert are suppressed (those tests use direct instantiation +add_recognizer, so they never hit this).Two possible fixes: declare
score_thresholdsexplicitly on these six yaml entries, or make the loader only assign when the conf actually provides thresholds. The second also fixes this for user-supplied configs that omit the key.