Skip to content

Commit 0085fd5

Browse files
CHORE: Flaky Setup - Gate benchmark and baseline steps on job success (#724)
### Work Item / Issue Reference > ADO Work Item: Fixed [AB#47384](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/47384) ------------------------------------------------------------------- ### Summary the benchmark and baseline steps keep running after SQL Server setup has already failed, so a dead job still burns 20 minutes benchmarking against a database that does not exist. build 167929, `macOS x86_64 SQL2025`: Build + start SQL Server (Colima boot & SQL setup...) failed 15.6 min Install mssql_py_core from NuGet wheels skipped Uninstall system ODBC Driver before running tests skipped Run pytest with coverage skipped Download and restore AdventureWorks2022 database succeeded 0.4 min Download macOS baseline from main succeeded 0.0 min Prepare macOS baseline file succeeded 0.0 min Run performance benchmarks on macOS SQL2025 succeededWithIssues 20.2 min pytest was correctly skipped because it relies on the default step condition. the four that ran override it: condition: or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')) that only asks which SQL version the leg is. writing an explicit condition replaces the implicit `succeeded()` rather than adding to it, so nothing upstream is checked. `Publish macOS benchmark baseline` already had this right with `and(succeeded(), ...)`. each condition now starts with `succeeded()`. the windows leg has the same four steps with the same pattern, and its SQL setup steps carry no `continueOnError`, so a failure there propagates the same way. eight sites in total across `pytestonwindows` and `PytestOnMacOS`. the benchmark step also carries `timeoutInMinutes: 20` and `continueOnError: true`, so when it times out the job reports `succeededWithIssues` instead of failing. the macOS SQL2022 leg in that same build finished `succeededWithIssues` for exactly that reason. this does not change that behaviour, it just stops the step running when there is nothing to benchmark against. related to [AB#47311](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/47311). that one raised the macOS job timeout to 90 minutes, this one stops a failed setup from eating 20 minutes of the budget. --------- Co-authored-by: Gaurav Sharma <223556219+Copilot@users.noreply.github.com>
1 parent 3af2ed4 commit 0085fd5

1 file changed

Lines changed: 14 additions & 14 deletions

File tree

‎eng/pipelines/pr-validation-pipeline.yml‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -238,23 +238,23 @@ jobs:
238238
- script: |
239239
python -m pytest -v --junitxml=test-results-localdb.xml --cov=. --cov-report=xml:coverage-localdb.xml --capture=tee-sys --cache-clear
240240
displayName: 'Run tests with coverage on LocalDB'
241-
condition: eq(variables['sqlVersion'], 'LocalDB')
241+
condition: and(succeeded(), eq(variables['sqlVersion'], 'LocalDB'))
242242
env:
243243
DB_CONNECTION_STRING: 'Server=(localdb)\MSSQLLocalDB;Database=TestDB;Uid=testuser;Pwd=$(DB_PASSWORD);TrustServerCertificate=yes'
244244
245245
# Run tests for SQL Server 2022
246246
- script: |
247247
python -m pytest -v --junitxml=test-results-sql2022.xml --cov=. --cov-report=xml:coverage-sql2022.xml --capture=tee-sys --cache-clear
248248
displayName: 'Run tests with coverage on SQL Server 2022'
249-
condition: eq(variables['sqlVersion'], 'SQL2022')
249+
condition: and(succeeded(), eq(variables['sqlVersion'], 'SQL2022'))
250250
env:
251251
DB_CONNECTION_STRING: 'Server=localhost;Database=TestDB;Uid=testuser;Pwd=$(DB_PASSWORD);TrustServerCertificate=yes'
252252
253253
# Run tests for SQL Server 2025
254254
- script: |
255255
python -m pytest -v --junitxml=test-results-sql2025.xml --cov=. --cov-report=xml:coverage-sql2025.xml --capture=tee-sys --cache-clear
256256
displayName: 'Run tests with coverage on SQL Server 2025'
257-
condition: eq(variables['sqlVersion'], 'SQL2025')
257+
condition: and(succeeded(), eq(variables['sqlVersion'], 'SQL2025'))
258258
env:
259259
DB_CONNECTION_STRING: 'Server=localhost;Database=TestDB;Uid=testuser;Pwd=$(DB_PASSWORD);TrustServerCertificate=yes'
260260
@@ -292,7 +292,7 @@ jobs:
292292
exit 1
293293
}
294294
displayName: 'Download and restore AdventureWorks2022 database'
295-
condition: or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025'))
295+
condition: and(succeeded(), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
296296
env:
297297
DB_PASSWORD: $(DB_PASSWORD)
298298
@@ -307,7 +307,7 @@ jobs:
307307
artifact: 'perf-baseline-$(sqlVersion)'
308308
path: $(Build.SourcesDirectory)
309309
displayName: 'Download baseline from main'
310-
condition: and(ne(variables['Build.SourceBranch'], 'refs/heads/main'), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
310+
condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/main'), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
311311
continueOnError: true
312312

313313
# Rename downloaded baseline so the script finds it (artifact may be in a subfolder)
@@ -320,7 +320,7 @@ jobs:
320320
Write-Host "No baseline file downloaded (first run or artifact missing)"
321321
}
322322
displayName: 'Prepare baseline file'
323-
condition: and(ne(variables['Build.SourceBranch'], 'refs/heads/main'), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
323+
condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/main'), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
324324
continueOnError: true
325325
326326
# Run performance benchmarks on SQL Server 2022
@@ -412,7 +412,7 @@ jobs:
412412
Write-Host "`nRunning performance benchmarks..."
413413
python benchmarks/perf-benchmarking.py --baseline benchmark_baseline.json --json benchmark_results.json
414414
displayName: 'Run performance benchmarks on SQL Server 2022/2025'
415-
condition: or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025'))
415+
condition: and(succeeded(), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
416416
continueOnError: true
417417
env:
418418
DB_CONNECTION_STRING: 'Server=localhost;Database=AdventureWorks2022;Uid=sa;Pwd=$(DB_PASSWORD);TrustServerCertificate=yes'
@@ -672,7 +672,7 @@ jobs:
672672
rm -f /tmp/AdventureWorks2022.bak
673673
docker exec sqlserver rm -f /tmp/AdventureWorks2022.bak || true
674674
displayName: 'Download and restore AdventureWorks2022 database on macOS'
675-
condition: or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025'))
675+
condition: and(succeeded(), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
676676
continueOnError: true
677677
env:
678678
DB_PASSWORD: $(DB_PASSWORD)
@@ -688,7 +688,7 @@ jobs:
688688
artifact: 'perf-baseline-macOS-$(sqlVersion)'
689689
path: $(Build.SourcesDirectory)
690690
displayName: 'Download macOS baseline from main'
691-
condition: and(ne(variables['Build.SourceBranch'], 'refs/heads/main'), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
691+
condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/main'), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
692692
continueOnError: true
693693

694694
- script: |
@@ -700,7 +700,7 @@ jobs:
700700
echo "No baseline file downloaded (first run or artifact missing)"
701701
fi
702702
displayName: 'Prepare macOS baseline file'
703-
condition: and(ne(variables['Build.SourceBranch'], 'refs/heads/main'), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
703+
condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/main'), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
704704
continueOnError: true
705705
706706
# Run performance benchmarks on macOS
@@ -714,7 +714,7 @@ jobs:
714714
echo "Running performance benchmarks..."
715715
python benchmarks/perf-benchmarking.py --baseline benchmark_baseline.json --json benchmark_results.json
716716
displayName: 'Run performance benchmarks on macOS $(sqlVersion)'
717-
condition: or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025'))
717+
condition: and(succeeded(), or(eq(variables['sqlVersion'], 'SQL2022'), eq(variables['sqlVersion'], 'SQL2025')))
718718
timeoutInMinutes: 20
719719
continueOnError: true
720720
env:
@@ -963,7 +963,7 @@ jobs:
963963
"
964964
fi
965965
displayName: 'Run pytest with coverage in $(distroName) container'
966-
condition: or(eq(variables['useAzureSQL'], 'false'), and(eq(variables['useAzureSQL'], 'true'), ne(variables['AZURE_CONNECTION_STRING'], '')))
966+
condition: and(succeeded(), or(eq(variables['useAzureSQL'], 'false'), and(eq(variables['useAzureSQL'], 'true'), ne(variables['AZURE_CONNECTION_STRING'], ''))))
967967
env:
968968
DB_PASSWORD: $(DB_PASSWORD)
969969
@@ -995,7 +995,7 @@ jobs:
995995
docker exec sqlserver-$(distroName) rm -f /tmp/AdventureWorks2022.bak || true
996996
fi
997997
displayName: 'Download and restore AdventureWorks2022 database in $(distroName)'
998-
condition: and(eq(variables['distroName'], 'Ubuntu'), eq(variables['useAzureSQL'], 'false'))
998+
condition: and(succeeded(), eq(variables['distroName'], 'Ubuntu'), eq(variables['useAzureSQL'], 'false'))
999999
continueOnError: true
10001000
env:
10011001
DB_PASSWORD: $(DB_PASSWORD)
@@ -1052,7 +1052,7 @@ jobs:
10521052
echo "Skipping performance benchmarks on $(distroName) (only runs on Ubuntu with local SQL Server)"
10531053
fi
10541054
displayName: 'Run performance benchmarks in $(distroName) container'
1055-
condition: and(eq(variables['distroName'], 'Ubuntu'), eq(variables['useAzureSQL'], 'false'))
1055+
condition: and(succeeded(), eq(variables['distroName'], 'Ubuntu'), eq(variables['useAzureSQL'], 'false'))
10561056
continueOnError: true
10571057
env:
10581058
DB_PASSWORD: $(DB_PASSWORD)

0 commit comments

Comments
 (0)