Speed up CI test runs (parallelism + caching)

[camd]

Summary

Reduce wall-clock time of the CI unit-test jobs by parallelizing execution, giving the heavy jobs more cores, and trimming repeated setup cost. No test files are modified — changes are limited to CI config and test-runner settings.

Measured results

Overall CI wall-clock is gated by the slowest parallel job (python-tests-general): 925s → 416s, −55%, ~8.5 min saved per run. All jobs pass. Master numbers are medians over recent runs (low variance).

Job	Master (2 vCPU, serial)	+ xdist (2 vCPU)	+ 4 vCPU (this PR)	vs master
python-tests-general	925s	661s	416s	−55%
python-tests-perf	598s	488s	349s	−42%
python-tests-frontend	495s	391s	294s	−41%
python-tests-telemetry	317s	279s	237s	−25%
javascript-tests	64s*	—	40s	−37%
javascript-lint	(bundled above)	—	18s (parallel)	new

* the old JS job bundled lint + markdownlint + tests serially.

Cost note: a large machine bills ~2× credits/minute, but each job now runs in roughly half the wall-clock — so vs today's master the 4-vCPU jobs are both faster and slightly cheaper in total credits.

Changes

JavaScript (javascript-tests)

• Run on a 4-vCPU box (resource_class: large) and let Jest's worker pool balance files dynamically (--maxWorkers=4). Container sharding was measured and rejected: it partitions files statically by path (18.5s vs 5.1s imbalance when the two heavy perfherder files collided on one shard) and re-pays the ~40s pnpm install per container, for a suite whose tests total ~41s. One bigger box wins on both wall-clock and cost, and Jest's in-process pool already balances dynamically.
• Split linting into its own javascript-lint job so test results are no longer gated behind pnpm lint + markdownlint; the two now run concurrently.

Python (python-tests-*)

• Add pytest-xdist and run each of the four marker jobs with -n auto --dist load (multi-process instead of serial, no extra Docker-stack startups).
• Bump the four jobs to resource_class: large (4 vCPU) so -n auto gets more workers — the default machine class is 2 vCPU.
• --dist load (not loadscope): profiling the two biggest general-marker files showed loadscope pinned the 52s test_perf_data_adapters.py to one worker (53.7s on -n 4), while load spread its 32 independent parametrized tests across workers for ~2× (26.3s), identical pass counts. An audit confirmed every module/session-scoped fixture in tests/ is read-only setup, so per-worker re-creation under load is safe.
• DB isolation is automatic: pytest-django creates a separate test database per worker (test_treeherder_gw0, …), so fixtures that assume specific row IDs remain valid.

Caching

• Reusable setup-pnpm command caches the pnpm store (keyed on pnpm-lock.yaml).
• Reusable install-tox command caches the pip/tox install across the four Python jobs.

Verification

• All 8 CircleCI checks pass on this PR; durations above are from the actual PR runs vs master medians.
• Locally: Jest on 4 workers green with coverage; pytest -n auto --dist load pass counts match the serial run with per-worker databases created as expected.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.97%. Comparing base (6bc1bb6) to head (3139bce).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff            @@
##           master    #9636    +/-   ##
========================================
  Coverage   82.97%   82.97%            
========================================
  Files         618      618            
  Lines       35798    35798            
  Branches     3273     3216    -57     
========================================
  Hits        29705    29705            
- Misses       5723     5943   +220     
+ Partials      370      150   -220     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.