Skip to content

Commit 47c2d0b

Browse files
authored
CHORE: Flaky Setup + Optimization - Parallelize setup steps and reduce macOS CI job time (#723)
### Work Item / Issue Reference > [AB#47311](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/47311) ------------------------------------------------------------------- ### Summary This pull request optimizes and streamlines the macOS test job in the Azure DevOps pipeline. The main improvements focus on reducing setup time, increasing reliability, and overlapping independent setup steps to speed up the overall process. Notably, the setup for Colima and SQL Server now runs in parallel with Python dependency installation and build steps, and a pip package cache is introduced to avoid redundant downloads between runs. **Pipeline performance and reliability improvements:** * Increased the `timeoutInMinutes` for the `PytestOnMacOS` job to 90 minutes to accommodate occasional long setup times and prevent premature job termination. * Added a pip package cache using the `Cache@2` task, significantly reducing dependency installation time by persisting wheels between runs. **Job step reorganization and parallelization:** * Combined and refactored Homebrew installations into a single step to avoid global lock delays and unnecessary updates, installing `cmake`, `docker`, and `colima` together. * Merged the Colima VM boot, SQL Server container setup, Python dependency installation, and pybind build into a single script step, running container setup in the background while building Python dependencies and extensions in the foreground. This parallelization hides setup latency and speeds up the job. * Improved logging and error handling for the container setup, ensuring failures are surfaced clearly and setup logs are displayed on error.
1 parent 33e286f commit 47c2d0b

1 file changed

Lines changed: 95 additions & 49 deletions

File tree

‎eng/pipelines/pr-validation-pipeline.yml‎

Lines changed: 95 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -461,9 +461,21 @@ jobs:
461461

462462
- job: PytestOnMacOS
463463
displayName: 'macOS x86_64'
464+
# Colima + SQL Server container setup averages ~12.5 min but has a long tail
465+
# (setup has been observed at 17-39 min). The ADO default job timeout is 60 min,
466+
# which the setup tail plus tests plus the 20-min benchmark step can exceed,
467+
# getting the job killed mid-step. Give the job enough headroom for the tail.
468+
timeoutInMinutes: 90
464469
pool:
465470
vmImage: 'macos-latest'
466471

472+
variables:
473+
# pip's on-disk download/wheel cache. Persisted across runs by the Cache@2
474+
# task below so `pip install -r requirements.txt` reuses wheels instead of
475+
# re-downloading them on every run. Exposed to script steps as the
476+
# $PIP_CACHE_DIR environment variable, which pip honors automatically.
477+
PIP_CACHE_DIR: $(Pipeline.Workspace)/.pip-cache
478+
467479
strategy:
468480
matrix:
469481
SQL2022:
@@ -482,63 +494,97 @@ jobs:
482494
addToPath: true
483495
displayName: 'Use Python $(pythonVersion) on macOS'
484496

485-
- script: |
486-
brew update
487-
# Uninstall existing CMake to avoid tap conflicts
497+
- task: Cache@2
498+
inputs:
499+
key: 'pip | "$(Agent.OS)" | "$(pythonVersion)" | requirements.txt'
500+
restoreKeys: |
501+
pip | "$(Agent.OS)" | "$(pythonVersion)"
502+
pip | "$(Agent.OS)"
503+
path: $(PIP_CACHE_DIR)
504+
displayName: 'Cache pip packages'
505+
506+
- script: |
507+
# Single serialized Homebrew transaction. brew holds a global lock, so
508+
# installing cmake, docker, and colima together is both faster than three
509+
# separate `brew install` calls and lets the next step overlap the Colima
510+
# VM boot + SQL setup with the Python build.
511+
# No `brew update` here: the hosted macOS runner ships an up-to-date
512+
# Homebrew and a full formula re-sync is ~1-3 min of pure setup overhead.
513+
# Uninstall the preinstalled CMake first to avoid tap conflicts.
488514
brew uninstall cmake --ignore-dependencies || echo "CMake not installed or already removed"
489-
# Install CMake from homebrew/core
490-
brew install cmake
491-
displayName: 'Install CMake'
492-
493-
- script: |
494-
brew update
495-
brew install docker colima
496-
497-
# Start Colima with extra resources
498-
colima start --cpu 4 --memory 8 --disk 50
515+
brew install cmake docker colima
516+
displayName: 'Install build & container tooling (Homebrew)'
517+
518+
- script: |
519+
set -e
520+
# Overlap the ENTIRE container-side setup (Colima VM boot -> image pull ->
521+
# container start -> SQL readiness) with the CPU-bound Python setup (pip
522+
# install + pybind build). The two chains are independent until pytest, so
523+
# we run the container chain in the background and the build in the
524+
# foreground, then wait for the container before continuing.
525+
#
526+
# On the Intel macOS-15 hosted runner the linux/amd64 SQL Server image runs
527+
# natively (no emulation); the dominant container costs are the Colima VM
528+
# boot (~2-4 min) and the ~2 GB image pull, both of which are hidden behind
529+
# the build here.
530+
#
531+
# NOTE: this deliberately merges what used to be separate steps (start
532+
# Colima, pull+start SQL, install Python deps, build pybind). Splitting them
533+
# back out restores the previous per-step timing breakdown.
534+
setup_sql() {
535+
echo "[sql] Starting Colima VM..."
536+
colima start --cpu 4 --memory 8 --disk 50
537+
docker context use colima >/dev/null || true
538+
docker version
539+
540+
echo "[sql] Pulling SQL Server image..."
541+
docker pull "$(sqlServerImage)"
542+
543+
echo "[sql] Starting SQL Server container..."
544+
docker run \
545+
--name sqlserver \
546+
-e ACCEPT_EULA=Y \
547+
-e MSSQL_SA_PASSWORD="${DB_PASSWORD}" \
548+
-p 1433:1433 \
549+
-d "$(sqlServerImage)"
550+
551+
echo "[sql] Waiting for SQL Server to accept connections..."
552+
for i in {1..30}; do
553+
if docker exec sqlserver \
554+
/opt/mssql-tools18/bin/sqlcmd \
555+
-S localhost -U SA -P "$DB_PASSWORD" \
556+
-C -Q "SELECT 1"; then
557+
echo "[sql] SQL Server is ready."
558+
return 0
559+
fi
560+
sleep 2
561+
done
562+
echo "[sql] SQL Server did not become ready in time." >&2
563+
return 1
564+
}
499565
500-
# Optional: set Docker context (usually automatic)
501-
docker context use colima >/dev/null || true
566+
echo "Starting container setup (Colima + SQL Server) in the background..."
567+
setup_sql > /tmp/sql_setup.log 2>&1 &
568+
SQL_PID=$!
502569
503-
# Confirm Docker is operational
504-
docker version
505-
docker ps
506-
displayName: 'Install and start Colima-based Docker'
570+
echo "Installing Python dependencies (overlapped with container setup)..."
571+
python -m pip install --upgrade pip
572+
pip install -r requirements.txt
507573
508-
- script: |
509-
# Pull and run SQL Server container
510-
docker pull $(sqlServerImage)
511-
docker run \
512-
--name sqlserver \
513-
-e ACCEPT_EULA=Y \
514-
-e MSSQL_SA_PASSWORD="${DB_PASSWORD}" \
515-
-p 1433:1433 \
516-
-d $(sqlServerImage)
574+
echo "Building pybind bindings (.so) (overlapped with container setup)..."
575+
( cd mssql_python/pybind && ./build.sh )
517576
518-
# Starting SQL Server container…
519-
for i in {1..30}; do
520-
docker exec sqlserver \
521-
/opt/mssql-tools18/bin/sqlcmd \
522-
-S localhost \
523-
-U SA \
524-
-P "$DB_PASSWORD" \
525-
-C -Q "SELECT 1" && break
526-
sleep 2
527-
done
528-
displayName: 'Pull & start SQL Server (Docker)'
577+
echo "Waiting for container setup (Colima + SQL Server) to finish..."
578+
if ! wait "$SQL_PID"; then
579+
echo "Container setup failed:"
580+
cat /tmp/sql_setup.log
581+
exit 1
582+
fi
583+
cat /tmp/sql_setup.log
584+
displayName: 'Build + start SQL Server (Colima boot & SQL setup overlapped with build)'
529585
env:
530586
DB_PASSWORD: $(DB_PASSWORD)
531587
532-
- script: |
533-
python -m pip install --upgrade pip
534-
pip install -r requirements.txt
535-
displayName: 'Install Python dependencies'
536-
537-
- script: |
538-
cd mssql_python/pybind
539-
./build.sh
540-
displayName: 'Build pybind bindings (.so)'
541-
542588
- template: steps/install-mssql-py-core.yml
543589
parameters:
544590
platform: unix

0 commit comments

Comments
 (0)