Testing 6.6.0-rc1 on 5506 tests suite

[kborowinski]

So I have this rather big tests suite for my PS modules at work. About ~5500 tests. I gave it a go on a 6.6.0-rc1 release and initially saw a sea of red 😬.

But 99% was due to one documented breaking change:

-ForEach $null/@() throws by default (Run.FailOnNullOrEmptyForEach).

So this one change fixed it for me:

if ((Get-Module -Name Pester).Version -ge [Version]'6.0.0') {
    $pesterConfiguration.Run.FailOnNullOrEmptyForEach = $false
}

or even a simpler way:

if ($pesterConfiguration.Run.PSObject.Properties['FailOnNullOrEmptyForEach']) {
    $pesterConfiguration.Run.FailOnNullOrEmptyForEach = $false
}

The other problem I had was due to a new behavior of Should -HaveCount on dictionary types:

#This was failing
$template.Foreground | Should -HaveCount 1

#This fixed the issue
$template.Foreground.Keys | Should -HaveCount 2

So far so good. I'll run more tests Monday morning, since I couldn't run all the integration tests yet.

[nohwnd]

Thanks, that is great news. The error could suggest also the global option.

[kborowinski]

Yeah, that's a good idea. Other option would be to set the global to $false by default, but I guess you had reasons to make it $true?

[fflaten]

The previous behavior (false) would silently skip tests when you either had a discovery-failure or typo.

They're hard to identify, so feels better to let the user accept the risk of null/empty data-driven tests when needed. Ideally at a block/test level to avoid issues above in the future, but global for quick migration. 🙂

[kborowinski]

Quick update. I've ran complete test suite at work this morning, on 6.0.0-rc2, including integration, and all's green! 😍👍

[kborowinski]

This is sick! 😍 Great job @nohwnd and @fflaten

top:
5512 tests on Pester 6.0.0-rc4 in parallel @ PS 7.6.3 in 26.31s

vs.

bottom:
5356 tests on Pester 6.0.0-rc4 @ PS 5.1 in 90.07s

[kborowinski]

@nohwnd This is on a different computer:

Parallel:
Pester 6.0.0-rc4 @ PS 7.6.3 in 33.01s

vs

Serial:
Pester 6.0.0-rc4 @ PS 7.6.3 in 88.12s

[nohwnd]

nice, looks like you would benefit from sharding this as well , the run caps basically on the longest file. We run parallel on file level, so if the longest file split in half it would in theory run in 19s and then maybe if the other longest file split in half it would also run in 19s.

But that is hard for us to figure out for you, since we don't know anything about your before alls, and if they are safe to run in parallel from the same file, and we also discovery in 1 sub runspace, so we would have to offload to yet another 2 sub runspaces or something.

But maybe can pilot it with something like #pester:shard directive, that you can place in the middle of your file. And figure it out from there.

[kborowinski]

Is sharding implemented in rc4? Didn't have much time to test it properly. Each test file has synthetic tests followed by slow integration tests. I placed the #pester:shard before each integration test block but there was no difference in speed at all.

Edit:
Ah... you're just thinking out loud about #pester:shard 🤭

[nohwnd]

yes, that was just thinking out loud. You can now do it manually, by splitting the file, and duplicating the setups, not really elegant, but reasonable if you are chasing your build taking 30s or less. Hard for Pester to do automatically for you, as you are balancing the overhead of more runspaces, we don't have historical data of what helped, and don't know your expectations - e.g. is 30s already fast enough.

[kborowinski]

Yeah, makes sense. The thing with the current test architecture is that it's not that easy to split them up without substantial re-engeenering. The performance gain from the parallel execution is already good. So I'll keep as is for the time being.