You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Speed up PHPUnit suite and CI setup: trim duplicate tests, shared helpers, faster job setup - #7392
Follow-up to #7302. Speeds up the PHPUnit suite and its CI jobs. The suite loses duplicate tests and tests that couldn't fail, copy-pasted tests become data providers, and repeated setup moves into shared helpers. There are two commits; reviewing them one at a time is easiest.
1. CI: cut per-job setup time
Each unit-test job spent ~25s in PHPUnit and ~100s in setup. This PR cuts that setup time:
Composer: a plain actions/cache keyed on composer.lock replaces ramsey/composer-install. Its cache key hashes every composer.json in the workspace (**/composer.json), which took ~10s per job once the mu-plugins-ext Jetpack and Parse.ly copies are in place.
Subversion is installed only when the WordPress test library isn't cached. sjinks/setup-wordpress-test-library needs svn only on a cache miss. Nightly installs it up front; any other version tries without it first and, if that fails, installs svn and retries. This saves ~10s per job.
Submodules are cloned in parallel (submodule.fetchJobs, set via GIT_CONFIG_* on the checkout step). That took a local clone from 38s to 25s.
mu-plugins-ext files are hard-linked with rsync --link-dest instead of copying ~500 MB (~10s per job). The resulting file set is identical, including the .dockerignore excludes and --delete.
Coverage now excludes vip-integrations/, which holds the 3,127 third-party plugin PHP files from mu-plugins-ext. They made up most of the ~47s "Generating code coverage report in Clover XML format" step in each of the 4 coverage jobs, and the longest jobs are the coverage ones. Codecov already ignored them because they aren't in this repo.
Measured on CI. Two green runs of this PR (1, 2) compared with the two most recent green develop runs that have the same 44-job matrix (1, 2):
develop
this PR
Change
Runner time per run (sum of the 44 unit-test jobs)
108 / 117 min
77 / 81 min
−30%
Average job
147 / 160 s
105 / 111 s
about −45 s
Coverage jobs (4)
232 / 221 s
124 / 135 s
−43%
Slowest job (sets how long the run takes)
259 / 233 s
185 / 145 s
−74 s or more
Per-job averages for each step, from the job logs:
Step
develop
this PR
PHPUnit run
27.4 s
13.7 s
Clover coverage report (coverage jobs only)
43 s
1 s
Subversion install
~12 s in every job
nightly jobs only
Composer install
13 s
4.8 s
Copy mu-plugins-ext into the tree
12 s
3.6 s
Checkout with submodules
16.8 s
12.9 s
WordPress test library: every job found it in cache on the first attempt, so the Subversion install-and-retry fallback never ran.
Noise: cloning mu-plugins-ext varies by 10–20 s between runs, with or without this PR, so "Prepare source code" varies too.
One small regression: PHP setup went from about 3.3 s to 5.7 s. It may now pay for the package-list refresh that the removed Subversion install used to run first.
Not included: time waiting for a runner.
2. PHPUnit: trim and speed up the suite
WP 7.1.3, same machine, best of two runs, default order:
develop
this PR
Tests
2048
1886 (1897 after rebasing onto #7386 and restoring cases found in review)
PHPUnit time, single-site
14.0s
5.7s
PHPUnit time, multisite
15.2s
6.8s
Test methods run in a separate process
19
13
Lines in tests/
37.3k
~30k
Global
VIP_Test_Listener restores $_SERVER and clears Constant_Mocker around every test. That removes about 70 copied clear() calls and several $_SERVER backups, and fixes leaks between classes under --order-by=random.
tests/bootstrap.php lowers the bcrypt cost everywhere. Before, the test library only lowered it inside a test, so users created in wpSetUpBeforeClass() cost ~45ms each.
New shared helpers in tests/utils/:
run_php() runs PHP in a child process without booting WordPress.
http_response() builds an HTTP response array.
Captures_Errors asserts a return value and a trigger_error() message in one test.
Two Site Details helpers.
Private copies of the reflection helpers now use tests/utils/utils.php.
Speed
No real sleeps. The default-threshold remote-request test passes timeout: 0, and a separate assertion checks the default 1s timeout.
Process isolation is gone where it wasn't needed: the stats bootstrap test (it already runs its own child process), request block, Enterprise Search configure tests, 3 Search migration tests (ElasticPress auth header injected through a filter instead of a real ES_SHIELD define), and support-user link denials. Where isolation is still needed it covers fewer processes: the Jetpack compatibility cases share 1 process instead of 4, and the connector-controls cases need 6 instead of 13, grouped by the real constant value.
The file ACL tests no longer copy the 6.9 MB fixture image into uploads. They never removed those copies, so each run leaked about 69 MB.
Users, posts, attachments and subsites that tests only read are created once per class. Per-test rest_api_init calls and Search boots that the tests didn't use are gone.
Removed or merged
Exact duplicates; tests that only exercised WordPress core, PHP or a mock; and copy-paste groups merged into providers. For example, the per-integration is_loaded, version-folder, *_CONFIGS and "inactive when unavailable" tests now live in tests/integrations/test-integration-loading.php, and the old per-integration files are deleted.
tests/lib/utils/context.php is renamed to test-class-context.php. PHPUnit only runs test-*.php files, so its tests never ran before.
Tests that couldn't fail, now fixed.
Queue rate-limit tests looped over true values instead of post IDs.
SettingsHealthJob __in_progress couldn't fail.
The bulk_chunk_size override test used the default value.
test_replicate_deletes_to_other_index_versions never reached the replicated delete.
VIP_PHPMailer::isPermittedPath was never exercised, because tests use MockPHPMailer.
The HyperDB "already defined" constants row returned early for another reason.
A Queue test passed only when an earlier test had left a Search singleton behind.
Pre-review checklist
Please make sure the items below have been covered before requesting a review:
This change works and has been tested locally or in Codespaces (or has an appropriate fallback). The full suite passes locally on WP 7.1.3, single-site and multisite, in random and default order. It also passes in the Parse.ly filter_enabled mode, and on WP 6.5 single-site and multisite with Jetpack skipped. npm run test:smoke passes too.
This change works and has been tested on a sandbox. Not applicable; tests and CI only.
This change has relevant unit tests (if applicable).
This change uses a rollout method to ease with deployment (if applicable - especially for large scale actions that require writes). Not applicable.
This change has relevant documentation additions / updates (if applicable). docs/testing.md describes the listener behavior and the shared helpers.
Pre-deploy checklist
VIP staff: Ensure any alerts added/updated conform to internal standards (see internal documentation). Not applicable.
Steps to Test
Check that the full CI matrix passes on this PR, and that job durations are in line with the measured numbers above.
- Install Composer dependencies with a plain cache keyed on composer.lock.
ramsey/composer-install hashes every composer.json in the workspace,
which takes ~10s per job once the mu-plugins-ext copies are in place.
- Only install Subversion when the WordPress test library isn't cached
(always for nightly); otherwise the first attempt succeeds without it.
- Clone submodules in parallel (submodule.fetchJobs).
- Hard-link the mu-plugins-ext files with rsync --link-dest instead of
copying ~500 MB.
- Exclude vip-integrations/ (third-party plugin copies from
mu-plugins-ext) from coverage, which makes up most of the ~47s Clover
report step in coverage jobs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove duplicate tests and tests that couldn't fail, merge copy-pasted
tests into data providers, extract shared helpers, and cut avoidable
slow work. 2048 -> 1886 tests; PHPUnit time 14.0s -> 5.7s single-site
and 15.2s -> 6.8s multisite (WP 7.1.3).
- Listener restores $_SERVER and clears Constant_Mocker around every
test; bootstrap lowers the bcrypt cost globally so users created in
wpSetUpBeforeClass() are cheap. Drop the per-file copies.
- New shared helpers in tests/utils: run_php(), http_response(),
Captures_Errors, site-details helpers.
- Fewer process-isolated tests (19 -> 13 methods); no real sleeps; no
6.9 MB fixture uploads in the ACL tests; shared fixtures per class.
- Rename tests/lib/utils/context.php to test-class-context.php so its
tests actually run.
- Fix tests that could pass regardless of production behavior, and an
order-dependent Queue test.
- Point test:smoke at the kept update_core test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- test_default_purge_permissions keyed its cases by role with a null key
for the logged-out case, which PHP 8.5 deprecates (and PHPUnit turns
into an error). Use [ role, allowed ] pairs.
- run_php() read the child's stdout to EOF before stderr, so a child
that filled the stderr pipe would hang the test run. Send stdout and
stderr to temporary files instead of pipes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preserve false results from earlier add_post_metadata filters
tests/performance/test-do-pings.php:13
The merged provider drops the distinct case where an earlier add_post_metadata filter has already returned false. The test now claims that non-_encloseme keys preserve the earlier value, but only verifies true; a regression that turns false back into true would pass. Keep the former false-value case in this provider.
The merged provider only passed true from earlier filters for other meta keys, so a filter that forced true would pass. Restore the false row the old tests had.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Jetpack sync queue filters: restore the "not set" rows. A filter that
returned empty values early would pass every remaining row.
- Pendo track event: send a bool and an int through the event
properties again. flatten_properties_to_strings() keeps them as-is
(like the base is_multisite/is_vip_user properties), and no remaining
test caught casting them to strings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #7302. Speeds up the PHPUnit suite and its CI jobs. The suite loses duplicate tests and tests that couldn't fail, copy-pasted tests become data providers, and repeated setup moves into shared helpers. There are two commits; reviewing them one at a time is easiest.
1. CI: cut per-job setup time
Each unit-test job spent ~25s in PHPUnit and ~100s in setup. This PR cuts that setup time:
actions/cachekeyed oncomposer.lockreplacesramsey/composer-install. Its cache key hashes everycomposer.jsonin the workspace (**/composer.json), which took ~10s per job once the mu-plugins-ext Jetpack and Parse.ly copies are in place.sjinks/setup-wordpress-test-libraryneedssvnonly on a cache miss. Nightly installs it up front; any other version tries without it first and, if that fails, installssvnand retries. This saves ~10s per job.submodule.fetchJobs, set viaGIT_CONFIG_*on the checkout step). That took a local clone from 38s to 25s.rsync --link-destinstead of copying ~500 MB (~10s per job). The resulting file set is identical, including the.dockerignoreexcludes and--delete.vip-integrations/, which holds the 3,127 third-party plugin PHP files from mu-plugins-ext. They made up most of the ~47s "Generating code coverage report in Clover XML format" step in each of the 4 coverage jobs, and the longest jobs are the coverage ones. Codecov already ignored them because they aren't in this repo.Measured on CI. Two green runs of this PR (1, 2) compared with the two most recent green develop runs that have the same 44-job matrix (1, 2):
Per-job averages for each step, from the job logs:
2. PHPUnit: trim and speed up the suite
WP 7.1.3, same machine, best of two runs, default order:
tests/Global
VIP_Test_Listenerrestores$_SERVERand clearsConstant_Mockeraround every test. That removes about 70 copiedclear()calls and several$_SERVERbackups, and fixes leaks between classes under--order-by=random.tests/bootstrap.phplowers the bcrypt cost everywhere. Before, the test library only lowered it inside a test, so users created inwpSetUpBeforeClass()cost ~45ms each.tests/utils/:run_php()runs PHP in a child process without booting WordPress.http_response()builds an HTTP response array.Captures_Errorsasserts a return value and atrigger_error()message in one test.tests/utils/utils.php.Speed
timeout: 0, and a separate assertion checks the default 1s timeout.ES_SHIELDdefine), and support-user link denials. Where isolation is still needed it covers fewer processes: the Jetpack compatibility cases share 1 process instead of 4, and the connector-controls cases need 6 instead of 13, grouped by the real constant value.rest_api_initcalls and Search boots that the tests didn't use are gone.Removed or merged
is_loaded, version-folder,*_CONFIGSand "inactive when unavailable" tests now live intests/integrations/test-integration-loading.php, and the old per-integration files are deleted.tests/lib/utils/context.phpis renamed totest-class-context.php. PHPUnit only runstest-*.phpfiles, so its tests never ran before.Tests that couldn't fail, now fixed.
truevalues instead of post IDs.SettingsHealthJob __in_progresscouldn't fail.bulk_chunk_sizeoverride test used the default value.test_replicate_deletes_to_other_index_versionsnever reached the replicated delete.VIP_PHPMailer::isPermittedPathwas never exercised, because tests useMockPHPMailer.Searchsingleton behind.Pre-review checklist
Please make sure the items below have been covered before requesting a review:
filter_enabledmode, and on WP 6.5 single-site and multisite with Jetpack skipped.npm run test:smokepasses too.docs/testing.mddescribes the listener behavior and the shared helpers.Pre-deploy checklist
Steps to Test
CI=1 ./bin/test.sh --wp 7.1.3 --order-by=randomandCI=1 ./bin/test.sh --wp 7.1.3 --multisite 1 --order-by=random.npm run test:smoke.🤖 Generated with Claude Code