Skip to content

Pgx native arrays - #770

Open
xrn wants to merge 7 commits into
stephenafamo:mainfrom
xrn:pgx-native-arrays
Open

xrn wants to merge 7 commits into
stephenafamo:mainfrom
xrn:pgx-native-arrays

Conversation

@xrn

@xrn xrn commented Sep 8, 2026

Copy link
Copy Markdown

Summary

Makes PostgreSQL array columns work with native pgx, without changing anything for existing lib/pq and pgx/v5/stdlib users.

Fixes #90
Fixes #739

The problem

Both issues are the same bug. Every Go type Bob generates for an array column is a sql.Scanner that only understands the Postgres text array literal ({a,b}): pq.StringArray and friends for primitive elements, and
pgtypes.Array[T] (a thin pq.GenericArray shim) for everything else.

Native pgx tries pgtype.ArraySetter first and, for anything else, falls back to the sql.Scanner path, where it hands the Scanner the raw binary wire format. That is the pq: unable to parse array; expected '{' at offset 0 in
#90. A null.Val[...] wrapper is itself a Scanner, so it receives the same binary bytes and forwards them to the inner type, which is the unsupported Scan, storing driver.Value type []uint8 in #739.

pgx stdlib only works because it forces the text format for array OIDs.

On the Go 1.26/1.27 driver.RowsColumnScanner question raised in #90: it does not help here. Native pgx never goes through database/sql, and pgx's stdlib implementation checks sql.Scanner before ArraySetter, so any Scanner type
still receives the text value. This PR needs no Go or pgx version bump.

What changed

Runtime (types/pgtypes)

  • pgtypes.Array[T] implements pgtype.ArrayGetter and pgtype.ArraySetter (modelled on pgx's pgtype.FlatArray), so native pgx scans and encodes it directly. It stays a sql.Scanner/driver.Valuer for database/sql drivers.
  • Its Scan also decodes pgx's binary array format, using pgx's own ArrayCodec with the element OID from the header. This is what makes the wrapped cases work: null.Val, sql.Null and orm.NullTypeConverter. Nullable arrays keep
    null.Val semantics, identical across drivers.
  • A pgtype.Map is not safe for concurrent use, so each scan takes one from a sync.Pool. A race-detector test covers this.
  • pgtypes.EnumArray[T] is now an alias of pgtypes.Array[T]; the two had identical method bodies.

Code generation (bobgen-psql)

  • github.com/jackc/pgx/v5 (native pgx, used with bob/drivers/pgx) is accepted as driver again, marked experimental.
  • With that driver, every array column is generated as pgtypes.Array[T]. The lib/pq and pgx/v5/stdlib drivers generate exactly what they generated before; the golden files are unchanged.
  • pq.BoolArray's built-in type definition was missing a compare expression; added.

Not breaking

No generated type changes unless the native driver is selected. The only API-visible changes are that EnumArray[T] is an alias of Array[T], and that types/pgtypes now imports github.com/jackc/pgx/v5/pgtype.

Tests

  • Unit tests in types/pgtypes use pgx itself to build binary fixtures and cover text and binary decoding for every element kind the generator emits, NULL vs {}, error cases, concurrency, and the exact null.Val/sql.Null shapes
    from Nullable array columns (text[], jsonb[]) fail to scan with the native pgx driver #739.
  • Table-driven unit tests for the array type translation per driver.
  • A generated round-trip test sends every distinct array type as a query parameter and scans it back: bare, wrapped in the nullable type as a result, wrapped in the nullable type as a parameter (what generated Setters send), empty, and
    NULL. It runs for all three drivers.
  • The native driver is exercised against a dedicated schema of array columns (text, integer, boolean, double precision, numeric, bytea, jsonb, enum), because the shared test schema contains other types that native pgx cannot yet scan
    (see below).

Known gaps, left for follow-ups

  • Native pgx remains experimental: interval and tsvector are mapped to string, which pgx cannot fill from its binary format. Documented, with a pointer to type replacements.
  • Array query parameters are always typed as string arrays because parser/args_cols.go passes the underscore-prefixed type name (_int4) to the array translation. Pre-existing and unrelated to scanning.
  • When replacing an array column's type for native pgx, use pgtypes.Array[T] rather than a plain []T; a plain slice inside null.Val cannot be scanned for the reasons above. Documented.

Commits

The history reflects the review iteration; the last commit b5b10d5 is the final state. Happy to squash or restructure on request.

🤖 Generated with Claude Code

RafRabenda and others added 6 commits September 8, 2026 10:19
pgtypes.Array and pgtypes.EnumArray now implement pgtype.ArrayGetter and
pgtype.ArraySetter, and their Scan method decodes pgx's binary array
format as well as the text literal. bobgen-psql generates pgtypes.Array[T]
for every array column instead of the lib/pq array types, which only
understand the text format.

Fixes stephenafamo#90, stephenafamo#739

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A pgtype.Map is not safe for concurrent use. Sharing one across all array
scans only worked because current pgx versions do not memoize scan plans,
which is an undocumented internal. Take a map from a sync.Pool per scan
instead; creating one is cheap since it defers to pgx's default registry.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Scanning the text format and producing a driver.Value for the seven
primitive element types is about twice as fast through lib/pq's typed
arrays than through pq.GenericArray and the pgx text parser, and it
produces the same bytes those columns produced before. Anything lib/pq
cannot parse, such as explicit bounds, still falls back to pgx.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Both types had identical method bodies. A generic type alias keeps the
exported name and method set while removing the duplication.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bobgen-psql no longer emits the lib/pq array types, so their built-in
definitions were dead code. Users who want them can still register
them under the types configuration.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Remove a mislabeled duplicate case, share the unknown-OID fixture helper
between the array and enum tests, and simplify the generated JSON
comparison helper.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@xrn

xrn commented Sep 8, 2026

Copy link
Copy Markdown
Author

@stephenafamo hope that you would not be angry that this code is AI generated. We are trying to move in our codebase to pgx - but we are blocked by #90 - so I am trying to solve it. If you have any feedback please share I would try to address.

Only the native pgx driver now generates pgtypes.Array for every array
column; lib/pq and pgx/v5/stdlib keep generating the same types as before
and the golden files are unchanged. The native driver is enabled as
experimental and tested against a dedicated schema of array columns.

The runtime drops the lib/pq fast paths that database/sql users no longer
reach, adds translator unit tests, a nullable-parameter round-trip case
and documentation for replacing array types under native pgx.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@xrn

xrn commented Sep 9, 2026

Copy link
Copy Markdown
Author

@jalaziz looking on your codebase I decided to have a change to which you inspred me.

#771 tackles the same problem by generating bare []T for the native driver and dropping null.Val for nullable arrays via a nil sentinel. This PR takes now thank to you the same non-breaking, driver-scoped stance and the same dedicated-schema testing approach, but keeps null.Val and fixes the wrapped-scanner case at the type level, so no core generator hook or driver-specific nullability model is needed.

@atzedus

atzedus commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Hi! I'd like to chime in and support this effort. In our team, this issue is also blocking the full transition to the native pgx driver. Thank you for working on this fix, it's highly appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nullable array columns (text[], jsonb[]) fail to scan with the native pgx driver Bob is not working with pgx array types

3 participants