feat: add JSON Schema by ya7010 · Pull Request #3314 · nextest-rs/nextest

ya7010 · 2026-04-29T10:28:45Z

Support: #3315

I have created a JSON Schema using schemars so that nextest.toml can be validated with tombi or taplo.

codecov · 2026-04-29T10:34:37Z

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.57%. Comparing base (e760edc) to head (ab04bfc).

Files with missing lines	Patch %	Lines
cargo-nextest/src/dispatch/utility/self_cmd.rs	0.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3314      +/-   ##
==========================================
- Coverage   85.58%   85.57%   -0.01%     
==========================================
  Files         160      160              
  Lines       47927    47933       +6     
==========================================
+ Hits        41018    41019       +1     
- Misses       6909     6914       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sunshowers · 2026-04-29T16:02:06Z

Thanks for doing this!

Have you tested this end-to-end? I tried doing this a while ago and ran into tamasfe/taplo#779 which never got accepted upstream.

ya7010 · 2026-04-30T00:15:48Z

I was testing with Tombi, but when I tried it with Taplo, an error occurred.

Taplo is currently unmaintained, so it is unlikely that it will be fixed.

nextest.toml in Tombi:

Test Code:

#:schema https://raw.githubusercontent.com/ya7010/nextest/3118e9a40e73428151c7a6f97a0b90fee464aba3/jsonschemas/nextest.json

# Define serial test group for running tests sequentially.
[test-groups]
serial = { max-threads = 1 }

# Run ty file watching tests sequentially to avoid race conditions.
[[profile.default.overrides]]
filter = "binary(file_watching)"
test-group = "serial"

[profile.ci]
# Print out output for failing tests as soon as they fail, and also at the end
# of the run (for easy scrollability).
failure-output = "immediate-final"
# Do not cancel the test run on the first failure.
fail-fast = false

status-level = "skip"

# Mark tests that take longer than 1s as slow.
# Terminate after 60s as a stop-gap measure to terminate on deadlock.
slow-timeout = { period = "1s", terminate-after = 60 }

# Show slow jobs in the final summary
final-status-level = "slow"

sunshowers · 2026-04-30T07:33:11Z

Ahh, thanks, didn't realize you were the maintainer of Tombi! I'll have a look at this in more detail soon.

ya7010 · 2026-04-30T12:44:18Z

If there are any improvements needed on the Tombi side, I’ll take care of them.

sunshowers · 2026-04-30T22:39:07Z

+  "required": [
+    "store",
+    "profile"
+  ],


I started reviewing this and trying it out with tombi, but immediately noticed that we have these properties as required. In almost no cases are properties required -- an entirely empty config file is valid, for example, and is equivalent to not having a config file at all.

I think there's a difference here between the schema for the default config (where most things are required) and for custom repo-specific configs (where most things are optional). Could you look into this deeper?

I'll let you address this before continuing the review.

I was made aware that there are exceptions that cannot be automatically generated from Rust structs.

I have modified NextestConfigDeserialize and StoreConfigImpl to use exceptions and implemented a special generation method for them.

858ea7d

@sunshowers

I have finished fixing the lint errors in nextest.toml for this repository. Could you please review it?

Thanks. It's going to take me a little while because I have two very high priority things to work on this week.

It is not urgent. Thank you!

…generation

ya7010 · 2026-05-07T00:26:21Z

@@ -0,0 +1,6 @@
+[lsp]
+formatting.enabled = false


The formatting has been disabled as it is not the focus of this PR.

With this configuration, you can verify how the nextet.json schema functions in tombi.

Thanks! Tested it out a bit -- this is wonderful.

sunshowers

This is really, really great work -- very thorough, thank you. Just a few questions.

sunshowers · 2026-05-11T05:32:47Z

+#[cfg_attr(feature = "config-schema", schemars(deny_unknown_fields))]
 #[serde(rename_all = "kebab-case", deny_unknown_fields)]


You can rely on schemars's serde attribute inheritance here, I think.

Done: 0bc28ae

sunshowers · 2026-05-11T05:33:15Z

+        schemars::json_schema!({
+            "oneOf": [
+                { "type": "integer", "minimum": 0 },
+                { "type": "string", "enum": ["infinite"] }
+            ]
+        })


This is great.

sunshowers · 2026-05-11T05:34:32Z

 serde_ignored = "0.1.14"
 serde_json = "1.0.149"
 serde_path_to_error = "0.1.20"
+schemars = { version = "1.2.1", features = ["indexmap2"] }


I think this is fine, but it's funny -- at my workplace (Oxide Computer) we're still stuck on schemars 0.8 due to a few reasons. I was hoping at some point to use some tooling we have at work to also analyze schema evolution over time for nextest (both this config schema and things like the schema for JSON output). I guess if I want to do that, then I'll have to either downgrade this to schemars 0.8 or find a way to move the work tooling to schemars 1.

Support for schemars v0.8 has been added.

Since the new syntax cannot be used, it is necessary to use the more verbose syntax.

f0c2222

If you're okay with the latest schemars, I can revert it. How would you like to proceed?

Yeah I think I'd prefer the latest schemars here.

Reverted: 6769106

sunshowers · 2026-05-11T05:41:05Z

+# Generate the JSON schema for .config/nextest.toml.
+generate-config-schema:
+    cargo run --package nextest-runner --features config-schema --bin generate-config-schema


We should add this generate check as a CI step -- I'd probably add this as part of the lint job.

sunshowers · 2026-05-11T05:42:17Z

Could you move the parenthetical to a regular comment?

Done!

95771d4

sunshowers · 2026-05-11T05:45:01Z

+    #[cfg_attr(
+        feature = "config-schema",
+        schemars(with = "Option<HashMap<String, CustomProfileImpl>>")
+    )]


This is pretty non-obvious so I think it needs a comment.

I would either inline a comment here, or add a short comment here and refer to the longer writeup in the helper binary.

Done: b30accb

sunshowers · 2026-05-11T05:46:07Z

@@ -0,0 +1,6 @@
+[lsp]
+formatting.enabled = false


Thanks! Tested it out a bit -- this is wonderful.

sunshowers · 2026-05-11T05:47:39Z

+use nextest_runner::config::core::nextest_config_schema;
+use std::fs;
+
+fn main() -> Result<()> {


I feel like there needs to be a writeup somewhere of some of the internal design decisions taken while generating the schema:

how deny_unknown_fields is meant to play the role of serde_ignored (which is why you can't rely on schemars's serde inheritance

related, why some fields of type T say that their schema is Option<T>

how types with custom deserializers are handled

All of these make sense, but I'm likely going to forget about them when I look a few months from now :)

Thoughts about putting this writeup in this file? This would be a natural place to look. Maybe if it gets complex enough we can even have a small design doc on the nextest site.

I moved the comments to NextestConfigDeserialize because having them in generate-config-schema.rs felt too far away from the definitions.

b30accb

sunshowers · 2026-05-11T05:51:53Z

+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "NextestConfigDeserialize",
+  "type": "object",
+  "properties": {


As a followup PR, would you be willing to write something that (1) embeds this schema within nextest, and (2) adds a cargo nextest self config-schema command that dumps out a file? I would prefer we not have to include schemars as part of the release (so your optional feature is fantastic, thanks), but it would still be nice to have access to the schemal.

Done: 3b43238

sunshowers · 2026-05-11T05:54:42Z

+#[cfg_attr(feature = "config-schema", derive(schemars::JsonSchema))]
+#[cfg_attr(feature = "config-schema", schemars(deny_unknown_fields))]
 #[serde(rename_all = "kebab-case")]
-struct NextestConfigDeserialize {
+pub(crate) struct NextestConfigDeserialize {


Curious, how hard would it be to add a test which ensures the default config passes schema validation?

Added a test for .config/nextest.toml.

77d356a

…struct

…ences - Updated `schemars` dependency in `Cargo.toml` and `Cargo.lock` to version 0.8.22. - Modified JSON schema references in `nextest.json` to use the correct format. - Refactored JSON schema generation in various configuration elements to improve clarity and maintainability.

…uration structs - Enhanced `NextestConfigDeserialize` and `StoreConfigImpl` to indicate optional fields in the JSON schema. - Updated `ArchiveConfig` to reflect that the `include` field should also be optional, improving schema clarity.

- Introduced a new `ConfigSchema` subcommand in `SelfCommand` to output the embedded JSON Schema for `nextest.toml`. - Added a static string `CONFIG_SCHEMA` to hold the JSON Schema content.

ya7010 · 2026-05-11T08:06:43Z

@sunshowers

I’ve gone ahead and fixed all the review comments I could address.

…ma references" This reverts commit f0c2222.

sunshowers · 2026-05-13T23:42:34Z

Looks great. I'll address the remaining issues I identified myself before landing this. Thanks again for doing all this work and for maintaining Tombi (I'm excited to finally have JSON Schema support for the nextest config -- Taplo was a bit of a brutal experience)

sunshowers reviewed Apr 30, 2026

View reviewed changes

ya7010 force-pushed the ya7010/nextest-schema-gen branch 3 times, most recently from 00424fe to 687cc2e Compare May 6, 2026 14:56

[meta] add config schema generation

f9ab21c

ya7010 force-pushed the ya7010/nextest-schema-gen branch from 687cc2e to f9ab21c Compare May 6, 2026 15:00

[config] add x-tombi-toml-version to schema and update config schema …

4589516

…generation

ya7010 commented May 7, 2026

View reviewed changes

sunshowers reviewed May 11, 2026

View reviewed changes

test: add repository nextest.toml validation.

77d356a

ya7010 force-pushed the ya7010/nextest-schema-gen branch from f71ef61 to 77d356a Compare May 11, 2026 06:42

ya7010 added 5 commits May 11, 2026 15:48

refactor: simplify description for overrides in JSON schema and Rust …

95771d4

…struct

refactor: remove redundant JSON schema attribute from ArchiveInclude …

0bc28ae

…struct

feat: add command to print JSON Schema for nextest configuration

3b43238

- Introduced a new `ConfigSchema` subcommand in `SelfCommand` to output the embedded JSON Schema for `nextest.toml`. - Added a static string `CONFIG_SCHEMA` to hold the JSON Schema content.

Revert "chore: update schemars version to 0.8.22 and adjust JSON sche…

6769106

…ma references" This reverts commit f0c2222.

sunshowers added 6 commits May 13, 2026 16:43

merge main into branch

6b85c7e

add CI job, use NEXTEST_WORKSPACE_ROOT

b4278e9

add a doc comment + hakari

44118f8

add some doc comments

e448315

more fixups during review

4820630

move to repo-config.json

ab04bfc

sunshowers merged commit bb3909c into nextest-rs:main May 14, 2026
24 checks passed

		#[cfg_attr(feature = "config-schema", schemars(deny_unknown_fields))]
		#[serde(rename_all = "kebab-case", deny_unknown_fields)]

Uh oh!

Uh oh!

Conversation

ya7010 commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sunshowers commented Apr 29, 2026

Uh oh!

ya7010 commented Apr 30, 2026

Uh oh!

sunshowers commented Apr 30, 2026

Uh oh!

ya7010 commented Apr 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunshowers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ya7010 commented May 11, 2026

Uh oh!

sunshowers commented May 13, 2026

Uh oh!

Uh oh!

ya7010 commented Apr 29, 2026 •

edited

Loading

codecov Bot commented Apr 29, 2026 •

edited

Loading