Skip to content

Commit ae798a3

Browse files
jahnvi480Copilot
andauthored
FIX: Enforce bundled native compatibility in Conda packages (#781)
### Work Item / Issue Reference > [AB#47315](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/47315) ### Summary This pull request introduces significant improvements to the Conda packaging and validation workflow for `mssql-python`, clarifies the installation and release process for Conda candidates, and strengthens platform compatibility and auditing. The most important changes are grouped below. **Conda Packaging and Platform Compatibility** * The Conda recipe now requires `MSSQL_PYTHON_VERSION` to be explicitly set, preventing accidental builds with the wrong code version. The combined Conda package now includes both the code and ODBC payload, eliminating the need for a separate ODBC package as required by pip installations. * The Linux `__glibc` dependency is raised to >=2.34 to ensure compatibility with the full native payload, including the bulk-copy core. * On macOS, the build script now explicitly matches `macosx` wheels to avoid accidentally using Linux wheels in cross-build scenarios. * For Windows ARM64, the Conda requirements add a `python_abi` pin for Python 3.12 to ensure ABI compatibility, and the documentation clarifies that dependencies resolve from the `defaults` channel without automatic terms acceptance. **Validation and Auditing Enhancements** * The Conda build scripts are updated to never silently drop the bulk-copy core: builds now fail if the required native extension is missing or incompatible, ensuring no silent reduction in functionality. * The audit logic now applies PE machine-type validation to both `win-64` and `win-arm64` packages, and always uses the actual target subdir for checks. * The build environment no longer sets or relies on automatic ToS acceptance for Windows ARM64, and the related environment variable is removed for clarity and compliance. **Testing and CI Improvements** * The CI pipeline now provisions the `testuser` login with `VIEW SERVER PERFORMANCE STATE` permission, enabling pool eviction tests to observe physical connections in SQL Server 2022 and 2025 jobs. **Documentation Updates** * The main `README.md` and a new `conda/README.md` clarify the temporary nature of the combined Conda candidate, its installation instructions, platform requirements, and that these changes do not constitute a public release or certification. **Validation Logic Hardening** * The Python validation logic now includes an additional check to ensure the required native extension for bulk copy (`mssql_py_core`) is present and loaded from the correct environment, with improved error messaging and isolation. These changes collectively improve the reliability, clarity, and platform compliance of the Conda packaging and validation process for `mssql-python`. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent ca5d73f commit ae798a3

35 files changed

Lines changed: 4918 additions & 2872 deletions

‎OneBranchPipelines/conda-build-pipeline.yml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
# =========================================================================================
22
# OneBranch STANDALONE Conda Build Pipeline (mssql-python)
33
# =========================================================================================
4-
# Builds the SELF-CONTAINED mssql-python conda package (which vendors the ODBC Driver 18
5-
# payload -- there is NO separate companion package) for every conda subdir, WITHOUT
4+
# Builds the SELF-CONTAINED mssql-python conda package (binding + ODBC Driver 18 + RS,
5+
# with NO separate Conda companion packages) for every conda subdir, WITHOUT
66
# rebuilding the wheels. It consumes the wheels already produced by the wheel build
7-
# pipeline (definition 2199): the mssql-python wheels (drop_Consolidate_ConsolidateArtifacts)
8-
# and the mssql-python-odbc wheels (drop_ConsolidateOdbc_ConsolidateArtifacts), then
7+
# pipeline (definition 2199): binding wheels plus source-pinned RS wheels/manifests/receipt
8+
# (drop_Consolidate_ConsolidateArtifacts) and the mssql-python-odbc wheels
9+
# (drop_ConsolidateOdbc_ConsolidateArtifacts), then
910
# repackages + validates them into conda packages on the matching platform agent.
1011
#
1112
# WHY PER-OS (not one job for all OSs): unlike the odbc wheels (a pure data re-tag),
@@ -306,7 +307,7 @@ extends:
306307
ob_outputDirectory: '$(Build.ArtifactStagingDirectory)'
307308
# The live Encrypt=yes TLS gate is NOT run in the build pipeline: it needs a
308309
# reachable SQL Server + a secret connection string (release/integration infra),
309-
# and the masking-immune static RUNPATH audit (audit_bundled_binaries.py) already
310+
# and the masking-immune static RUNPATH audit (eng.conda_tools elf) already
310311
# guards the OpenSSL layout here. It moves to the release pipeline, where the
311312
# secret is always present so the gate is unconditional -- no queue-time toggle.
312313
# The minimal-base ldd reachability gate stays opt-in because it is only VALID on

‎OneBranchPipelines/jobs/consolidate-artifacts-job.yml‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -125,12 +125,16 @@ jobs:
125125
displayName: 'Consolidate Windows symbols (optional)'
126126
continueOnError: true
127127
128-
# Include the companion dependency version for release validation.
128+
# Stage the actual RS payload using THIS producer source's immutable pins.
129+
# Downstream Conda jobs consume this artifact, never their checkout's NuGet pin.
129130
- bash: |
130131
set -e
131-
cp $(Build.SourcesDirectory)/eng/versions/mssql-python-rs.version $(ob_outputDirectory)/dist/
132-
echo "mssql-python-rs version: $(cat $(ob_outputDirectory)/dist/mssql-python-rs.version)"
133-
displayName: 'Include mssql-python-rs version'
132+
python3 "$(Build.SourcesDirectory)/eng/scripts/download_mssql_python_rs_wheels.py" \
133+
--output-dir "$(ob_outputDirectory)/rs-wheels"
134+
cp "$(Build.SourcesDirectory)/eng/versions/mssql-python-rs.version" "$(ob_outputDirectory)/dist/"
135+
cp "$(Build.SourcesDirectory)/eng/versions/mssql-python-rs-nuget.version" "$(ob_outputDirectory)/dist/"
136+
echo "mssql-python-rs version: $(cat "$(ob_outputDirectory)/dist/mssql-python-rs.version")"
137+
displayName: 'Stage source-pinned mssql-python-rs wheels and transport receipt'
134138
135139
# Verify consolidation
136140
- bash: |

‎OneBranchPipelines/jobs/consolidate-conda-artifacts-job.yml‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Consolidate Conda Artifacts Job Template
2-
# Collects the per-platform self-contained mssql-python conda packages (which vendor
3-
# the ODBC payload) that each build leg staged under conda/<subdir>/ and gathers
4-
# them into a single conda/ tree for the release pipeline to publish.
2+
# Collects the per-platform combined binding/ODBC/RS packages staged under
3+
# conda/<subdir>/ and preserves their shared producer RS receipt in the final
4+
# conda/ tree. Package counts exclude this provenance sidecar.
55
#
66
# BEST-EFFORT (build pipeline): conda is a downstream repackage of the ESRP-signed
77
# wheels and must NEVER block the primary wheel deliverable. An empty set WARNS here;
@@ -22,7 +22,7 @@
2222
# TOTAL (PyPI parity minus musllinux) = 28
2323
# win-arm64 is cross-built for py3.12-3.14 only (py3.10/3.11 have no win-arm64 deps
2424
# on Anaconda defaults); its runtime import is skipped on the x64 host, so its arch
25-
# is enforced by the PE-machine assert (assert_pe_machine.py). musllinux (no conda
25+
# is enforced by the PE audit (eng.conda_tools pe). musllinux (no conda
2626
# musl subdir) is intentionally NOT conda-built. This job is BEST-EFFORT and never
2727
# hard-fails on a short set; the release pipeline's conda-release-step enforces the
2828
# hard gate (required subdirs present + complete Python matrix) before publish.
@@ -92,6 +92,18 @@ jobs:
9292
found=1
9393
done < <(find "$(Pipeline.Workspace)/all-artifacts" -type f \( -name 'mssql-python*.conda' -o -name 'mssql-python*.tar.bz2' \))
9494
95+
receipt="$(ob_outputDirectory)/conda/rs-transport.json"
96+
while IFS= read -r source; do
97+
if [ -f "$receipt" ]; then
98+
cmp -s "$source" "$receipt" || {
99+
echo "ERROR: Conda legs disagree on their selected RS transport receipt." >&2
100+
exit 1
101+
}
102+
else
103+
cp "$source" "$receipt"
104+
fi
105+
done < <(find "$(Pipeline.Workspace)/all-artifacts" -type f -name 'rs-transport.json')
106+
95107
echo ""
96108
echo "Consolidated conda tree:"
97109
find "$(ob_outputDirectory)/conda" -type f | sort

0 commit comments

Comments
 (0)