[spencer-banko]

Requested by Brody Lussier

Closes #30. This PR now covers both halves of the ticket: the registry lookups (below, by @spencer-banko) and the storage, hourly job, rerun, person page and Approvals queue (see "Storage, job and admin screens" further down).

Branch: pre-check-volunteers-against-public-registries → main

Problem

Background clearance is fully manual. An admin runs a check somewhere else and types in the outcome. Fingerprinting rules out full automation, but searching public sex-offender registries for a volunteer's name can be automated. To do that we need:

• one lookup function that returns a shared result type, so a vendor API can replace it later without touching the admin side
• sources that respect each registry's terms of use and robots.txt
• matching that uses date of birth, so a common name doesn't flag everyone who has it
• no names or results in logs, no stored result pages, no real people's records in fixtures

Solution

checkRegistries(subjects) in src/lib/screening/index.ts checks a batch of volunteers against every source. For each source it returns one of three outcomes per volunteer:

Outcome	Meaning
no_match	Searched, nothing found
possible_match	One or more hits, each with a link to verify
unavailable	Couldn't check (manual_only, missing_dob or source_error), with a link to search by hand

Every result carries source, sourceLabel and checkedAt. The function only looks things up. It never throws, never touches clearance status and never logs names or results.

Sources: what each one allows (checked 2026-09-30)

Source	robots.txt	Terms of use	Decision
NSOPW (nsopw.gov)	Disallow: /search/ and Disallow: /node/*	Conditions of Use: users "agree to not perform any actions on this Website, such as automated searching or attempts to bypass security mechanisms". You must accept the conditions before searching, and results load with JavaScript.	Automated access is forbidden. Never fetched. Always returns unavailable / manual_only with a link to https://www.nsopw.gov/search-0
Georgia GBI (state.sor.gbi.ga.gov)	None (404)	The search UI needs an "I agree" click, and its conditions say nothing about automated access. gbi.georgia.gov publicly offers "Download a file of the latest Georgia sexual offender registry data".	Use the official CSV download (https://state.sor.gbi.ga.gov/SORT_PUBLIC/sor.csv), which avoids scraping entirely.

Implementation details

File	What it does
src/types/screening.ts	The shared result type: RegistryCheckResult, RegistryHit, RegistrySubject, and the source, outcome and reason enums. This is the contract with the storage and UI half.
src/lib/screening/index.ts	checkRegistries(subjects, sources?) → Map<userId, RegistryCheckResult[]>, one result per source in source order. A source that throws becomes source_error for every subject it was checking, and the other sources still run.
src/lib/screening/source.ts	The ScreeningSource interface (id, label, manualSearchUrl, check(subjects, checkedAt)). A vendor API is one new implementation of it.
src/lib/screening/sources/gbi.ts	Details below.
src/lib/screening/sources/nsopw.ts	Manual-only. No network call.
src/lib/screening/match.ts	Name normalization and matching, plus the birth-year check.
src/lib/csv.ts	Adds parseCsv (RFC 4180) next to the existing toCsv.
src/constants/limits.ts	Adds SCREENING_BIRTH_YEAR_TOLERANCE = 1 and SCREENING_FETCH_TIMEOUT_MS = 8000.
scripts/screening-check.ts + npm run screening:check	Assertion checks (see Tests).

How the GBI source works:

• It downloads the CSV once per call with plain fetch, an 8-second timeout, a descriptive User-Agent and no retries. The whole batch shares that one download.
• It indexes the batch by last-name key and scans the file once, keeping only rows that match.
• The CSV is never written anywhere, and the rest of it is discarded after the call.
• Each hit stores only the printed name, the birth year, the city/county and the GBI search URL. The CSV has no link to individual records.
• It refuses to report no_match from a bad download. If the expected columns are missing or the file has fewer than 1,000 entries, the result is source_error. The real file has about 25,600.
• Volunteers with no date of birth get missing_dob, and if nobody in the batch has one, the file isn't downloaded.

Name matching:

• Accents are folded, apostrophes and periods dropped (O'Brien → OBRIEN), hyphens split, and suffixes (JR, SR, II–V) removed.
• First names must agree on their first word. Middle names are ignored.
• Last names match on the whole name or on any part of a compound name that is at least 3 letters long.

Decisions made

• GBI through its official CSV, not scraping the search page. GBI offers the file publicly, so there are no ASP.NET postbacks, no terms gate and no HTML parser. It's one request for any number of volunteers. The tradeoff is that the file has a year of birth only and no link to each record.
• Hits count within ±1 birth year. GBI publishes a birth year and NSOPW an age, not a date. One year of slack covers age rounding and data-entry slips. A blank registry year counts as a match, because it can't rule anyone out.
• No date of birth → "couldn't check" (missing_dob). We never flag on a name alone. Google sign-ups can have no DOB, and the admin gets the manual link instead.
• A hit links to the GBI search page and includes the city/county. The CSV has no URL for each record, so the location helps the admin find the right entry.
• A bad or truncated download gives "couldn't check", never "no match". A false "no match" is the worst failure here.
• The function takes a batch of volunteers. Each hourly run downloads the GBI file once. A vendor API can loop over the batch behind the same signature.
• Logging: only the source ID and a short error, such as [screening] gbi_ga failed: download failed: 503. Never names, hits or outcomes.
• Location follows the sprint doc: src/lib/screening/. The shared types are in src/types/screening.ts, next to the other shared types.

Tests

The repo has no test runner (see docs/sprints/sprint-3.md), so this adds scripts/screening-check.ts, a script of node:assert checks. Every name in it is invented, and no fixture comes from real data.

npm run screening:check            # 23 checks against a made-up registry file, no network
npm run screening:check -- --live  # plus one check that downloads the real GBI file (made-up names only)

The 23 offline checks cover:

• CSV parsing: quotes, doubled quotes, commas, CRLF, line breaks inside quotes, and a toCsv round trip.
• Names: accents, apostrophes, hyphens and suffixes. Middle names are ignored. Different first or last names don't match, compound last names match either way, and short particles alone don't match.
• Birth year:
  • same year and ±1 match; 2 or more years off doesn't
  • a Jan 1 birthday stored as midnight Eastern keeps its year
  • a blank registry year still matches
• GBI source:
  • a possible match lists every hit within a year, with location and link
  • the same name 3 or more years off → no_match
  • a name not in the file → no_match
  • no DOB → missing_dob, with the file never downloaded
  • several volunteers share one download
  • changed columns, a truncated file or a failed download → source_error
• NSOPW: always manual_only, and fetch is never called (stubbed to throw).
• checkRegistries:
  • one result per source per volunteer, in source order
  • a broken source doesn't stop the others
  • an empty batch calls no sources

The live check downloads the real file and confirms the made-up names come back no_match. The download, parse and match took about 2.0 s, under the 8 s timeout. The [screening] … failed: lines in the output come from the failure-path checks and are expected.

Also run locally: npm run format:check, npm run lint, npm run typecheck and npm run build. All pass.

Not verified:

• A possible match against real registry data. That would mean searching for a real person, so possible matches are only checked against the made-up file.
• The lookup running inside the hourly job and the admin screens. Those belong to the storage and UI half.

Storage, job and admin screens

Before: a registry result existed only in memory, and an admin waiting on a volunteer's clearance had nothing to look at. Approvals listed only clearances an admin had already marked submitted.

After: every volunteer with a sign-up pending on clearance gets checked by the hourly job. The admin sees each registry's result on the person page and in a new Waiting on clearance list in Approvals. Clearance status is never changed.

File	What it does
src/db/models/screeningCheck.ts, src/db/actions/screeningCheck.ts	One ScreeningCheck per volunteer: the latest RegistryCheckResult per source, plus rerunRequestedAt/By. Separate collection, like Clearance. Results are kept after the volunteer stops being pending.
src/services/screening.ts	runDue() (hourly) checks volunteers pending on clearance who were never checked, hit source_error, or had missing_dob and now have a DOB, plus queued reruns. requestRerun() queues a rerun and writes a screening.rerun_requested audit entry.
src/services/jobs.ts	Adds screening to runAll.
src/app/api/v1/admin/people/[id]/screening/route.ts	POST, admin only: queue a rerun. Lookups never run inside a request.
src/services/admin.ts, src/types/api.ts	PersonDetail.screening and AdminApprovals.awaitingClearance.
src/app/admin/people/[id]/page.tsx, src/components/admin/ScreeningResults.tsx	"Registry pre-check" card: each source's outcome, every hit's link, manual-search links, and a Rerun button that shows "Queued" until the job runs.
src/app/admin/page.tsx	New "Waiting on clearance" list next to "Clearances to review", e.g. Waiting on: clearance · GBI: No match · NSOPW: Search by hand.

A rerun requested while the job is running stays queued for the next run, so it is never silently dropped.

Review fixes to the lookups

• A GBI download whose length doesn't match Content-Length, or whose rows don't match the header width, is now source_error, not no_match.
• parseCsv throws on an unclosed quote.
• An invalid DOB gives missing_dob.
• A repeated userId in a batch gets one result per source.
• Only ScreeningSourceError messages are logged. Any other exception is logged as unexpected error, so a future source can't leak a name.
• console.log in scripts/screening-check.ts is left as is, matching scripts/seed.ts and scripts/jobs.ts.

Also merged main, which resolves the package.json conflict with the staging scripts.

Ran

npm run format:check, npm run lint, npm run typecheck, npm run build and npm run screening:check (29 checks, 6 new). All pass.

Not verified: the job, rerun and admin screens against a real database. No MongoDB was reachable where this was built. To check: sign up a made-up volunteer (with and without a DOB) for an event that needs clearance, run npm run jobs, then open Approvals and the person page, click Rerun, and run the jobs again.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GG4LpiuNziAYJVp9WJMbtK

---

Generated by Claude Code

[greptile-apps]

Disclaimer: This comment was AI-generated and is not necessarily completely accurate. Please take code comments with a grain of salt.

Confidence Score: 2/5

[Medium risk] Adds screening registry lookup and matching logic.

The PR should not merge until incomplete CSV data cannot yield a clean result and duplicate subject IDs are handled; the explicit console-log requirement also needs to be satisfied.

Findings

1.  Partial downloads can appear complete ▶
2.  Unclosed quotes hide registry entries ▶
3.  Duplicate IDs produce duplicate results ▶
4.  Invalid DOB appears cleared ▶
5.  Source errors can expose names ▶
6.  Test output violates logging directive ▶

Summary

The PR adds a shared registry-result contract, a batched GBI CSV lookup, a manual-only NSOPW result, and an assertion script. The screening job and UI are explicitly out of scope. The main concerns are false no_match results from incomplete CSV processing and duplicate results for repeated user IDs. Checklist areas needing improvement are input validation and console-log usage. Review score: 70/100.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Batch of subjects] --> B[checkRegistries]
  B --> C[GBI CSV download and matching]
  B --> D[NSOPW manual-only result]
  C --> E[Results keyed by userId]
  D --> E

Loading

_{Reviews (1) · Last reviewed commit: "Add screening pre-check registry lookups"}

[netlify]

✅ Deploy Preview for pink-stem-staging ready!

Name	Link
🔨 Latest commit	41831b0
🔍 Latest deploy log	https://app.netlify.com/projects/pink-stem-staging/deploys/6ad657044a3a28c0c20bd68f
😎 Deploy Preview	https://deploy-preview-34--pink-stem-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.