Repository navigation
Inconsistent retrieval of CP1252 encoded data in VARCHAR columns - Windows vs. Linux #468
Description
Describe the bug
When retrieving data from a VARCHAR column containing byte value 173 using default encoding settings, the Windows driver returns a bytes value, while the Linux driver returns a str value.
I suspect this is happening because, under Windows, the driver is attempting to decode the data in a VARCHAR column using utf-8 encoding, and a single byte value 173, which is a valid value in various code pages, including the default Windows CP-1252, does not map to a valid character under utf-8 encoding. When the decode fails, the driver passes the data through, unencoded, as a bytes value. Under Linux, some other decoding behavior is going on that produces a str.
To reproduce
con = mssql_python.connect(... connect string ...)
c = con.cursor()
c.autocommit = True
c.execute("create table tab(col varchar(10) collate SQL_Latin1_General_CP1_CI_AS)")
# Insert a value containing a byte 173
c.execute("insert into tab values('\xad')")
c.execute("select * from tab")
print(c.fetchall())Windows output:
[(b'\xad',)]
Linux output:
[('\xad',)]
Expected behavior
When run under Windows and Linux I would expect this program to produce the same results - ideally, a string containing '\xad'.
Interestingly, when the setdecoding() option is used to specify encoding "cp1252", the output under both Windows and Linux is the same:
con = mssql_python.connect(... connect string ...)
con.setdecoding(mssql_python.SQL_CHAR, "cp1252")
...
produces output:
[('\xad',)]
Further technical details
Python version: 3.14
SQL Server version: SQL Server 2022
Operating system: Docker container
Activity
github-actions commentedon Mar 6, 2026
Hi Stephen Schaub (@sschaub), thank you for opening this issue!
Our team will review it shortly. We aim to triage all new issues within 24-48 hours and get back to you.
If you have additional information to share, please feel free to update the issue.
Thank you for your patience!
subrata-ms commentedon Mar 16, 2026
Hi Stephen Schaub (@sschaub) , I am able to reproduce the issue locally. I will work on the fix and update the bug with release details.
The passing tests confirm that when CP1252 decoding is explicitly configured via setdecoding(SQL_CHAR, encoding="cp1252"), the data decodes correctly. This proves the data is stored and retrievable — it's the default UTF-8 decode path on Windows that's currently not working for any CP1252 byte that isn't also valid UTF-8.
FIX: Inconsistent retrieval of CP1252 encoded data in VARCHAR columns…
subrata-ms commentedon May 13, 2026
Change Summary::
This pull request updates the default handling of SQL CHAR/VARCHAR columns to use UTF-16 (wide character) encoding instead of UTF-8, primarily to address encoding mismatches on Windows and ensure consistent Unicode decoding. The changes span the connection, cursor, and C++ binding layers, and update related tests to reflect the new default behavior.
Default Encoding and Decoding Changes:
The default decoding for SQL CHAR columns is now set to use "utf-16le" encoding and the SQL_WCHAR ctype, replacing the previous "utf-8"/SQL_CHAR defaults. This avoids issues where Windows ODBC drivers return raw bytes in the server's native code page, which may not decode as UTF-8. (mssql_python/connection.py, mssql_python/connection.pyR264-R271)
All cursor fetch methods (fetchone, fetchmany, fetchall) are updated to request UTF-16 decoding and pass the correct ctype when fetching CHAR data, ensuring consistent behavior across platforms. (mssql_python/cursor.py, [1] [2] [3]
C++ Binding and Processing Updates:
The ColumnInfoExt struct now tracks whether wide character (UTF-16) fetching is used for a column, and the ProcessChar function is updated to handle both wide and narrow character paths, decoding appropriately based on the new setting. (mssql_python/pybind/ddbc_bindings.h, [1] [2] [3]
Test Adjustments:
Tests are updated to expect "utf-16le" and SQL_WCHAR as the default decoding settings for SQL_CHAR columns, and to validate the new default behavior. (tests/test_013_encoding_decoding.py, [1] [2] [3]
subrata-ms commentedon May 13, 2026
Hi Stephen Schaub (@sschaub) , the fix should be available under mssql-python release 1.7.0.