Skip to content

Provide alternative choices for JSON libraries - #135

Open
max-moser wants to merge 2 commits into
front-matter:mainfrom
max-moser:mm.ujson
Open

Provide alternative choices for JSON libraries#135
max-moser wants to merge 2 commits into
front-matter:mainfrom
max-moser:mm.ujson

Conversation

@max-moser

Copy link
Copy Markdown

Once again, I've been doing some experimentation with free-threading Python in the hopes of simplifying our setup by improving the situation regarding the limited number of Python workers on the web server.

As part of this work, I stumbled upon the blockers that prevented the use of free-threading Python:

  • The binary distributions of psycopg don't support free-threading Python, but the source build does
  • orjson does not support free-threading Python either, and there doesn't seem to be a clear timeline; the issue tracker and PR tabs on the GitHub project have also been closed now (with the README stating that was due to the signal-to-noise ratio)...

In this (still very rough and unpolished) PR, I've implemented support for a few alternative libraries to handle JSON, via extras (e.g. commonmeta-py[msgspec]:

  • orjson: kept support for that via commonmeta-py[orjson]
  • ujson: similar to orjson; has free-threading support, but is a bit slower and in maintenance mode (even the maintainers suggest moving to orjson if possible)
  • msgspec: a performant alternative [1], with slightly different API
  • [json]: if no other implementation was chosen, fall back to the standard library

[1] https://msgspec.dev/benchmarks#json-serialization


Note that msgspec does not like Python 3.9:

  x No solution found when resolving dependencies for split (markers: python_full_version == '3.9.*'):
  `-> Because the requested Python version (>=3.9) does not satisfy Python>=3.10 and msgspec>=0.21.0 depends on Python>=3.10, we can conclude that msgspec>=0.21.0 cannot be used.
      And because only the following versions of msgspec are available:
          msgspec<=0.21.0
          msgspec==0.21.1
      we can conclude that msgspec>=0.21.0 cannot be used.
      And because commonmeta-py[msgspec] depends on msgspec>=0.21.0 and your project requires commonmeta-py[msgspec], we can conclude that your project's requirements are unsatisfiable.

hint: While the active Python version is 3.14, the resolution failed for other Python versions supported by your project. Consider limiting your project's supported Python versions using `requires-python`.

The library `orjson` does not yet support free-threading Python, and
there's no clear timeline yet when that will happen.
To better support several Python environments, we provide support for
different JSON implementations that the user can choose from:

* `orjson`:  was used previously in `commonmeta`
* `ujson`:   similar to `orjson`, trades compatibility for performance
* `msgspec`: similar performance to `orjson` and better compatibility,
             at the cost of some minor API differences
* `json`:    fallback to the standard library if no other JSON library
             was chosen
The binary distribution of psycopg is only recommended for development
and testing environments, and does not support free-threading Python.

https://pypi.org/project/psycopg2/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant