fix: support setting logging.yml overrides via prefect config set by Abdulrehman-PIAIC80387 · Pull Request #22411 · PrefectHQ/prefect

Abdulrehman-PIAIC80387 · 2026-06-30T10:34:19Z

Draft for early feedback. This follows @desertaxle's suggestion on #18670 to make custom logging settings load through LoggingSettings / a custom source, and is built to avoid the three issues that closed #18670.

Root cause

prefect config set validates keys against registered Settings fields. PREFECT_LOGGING_LEVEL maps to a real field, but PREFECT_LOGGING_LOGGERS_PREFECT_FLOW_RUNS_LEVEL is a dotted path into logging.yml, not a Pydantic field — so it's rejected with Unknown setting name. The runtime honors these keys only when read directly from os.environ (in load_logging_config), which is why the env-var workaround works but config set (which writes to a profile) does not.

Fix

Make custom PREFECT_LOGGING_* keys first-class settings:

LoggingSettings.overrides (settings/models/logging.py) — a dict[str, str] holding override keys → values. Because it's a real field, the values integrate with get_current_settings() and render in prefect config view.
LoggingOverridesSource (settings/sources.py) — collects PREFECT_LOGGING_* keys that don't match a declared field, from the environment and the active profile, into overrides. Cheap key filtering only — it never reads or parses logging.yml.
load_logging_config (logging/configuration.py) — applies overrides from get_current_settings().logging.overrides (existing os.environ behavior preserved as the higher-priority source). This is lazy (runs at setup_logging, not import).
CLI (cli/config.py) — config set / unset accept logging-override keys (validated against logging.yml, parsed lazily and cached, CLI-only) and persist them as Setting objects; config view renders them.
_cast_settings (settings/profiles.py) — wraps unknown PREFECT_LOGGING_* profile keys as Setting objects (cheap prefix check) so they round-trip through profiles and config view without warnings.

How this avoids #18670's failure modes

#18670 failure	This PR
12% import perf regression (parsed logging.yml on import)	Override collection does cheap key filtering only; `logging.yml` is parsed lazily and only in the CLI. Guard test: `test_collecting_overrides_does_not_parse_logging_config`.
`config view` crashed (`'str' has no attribute 'name'`)	Logging keys are stored/loaded as `Setting` objects, not raw strings. Test: `test_view_shows_logging_override`.
Runtime ignored the value	`load_logging_config` reads from `get_current_settings().logging.overrides`. Test: `test_load_logging_config_applies_settings_overrides`.

Verification

$ prefect config set PREFECT_LOGGING_LOGGERS_PREFECT_FLOW_RUNS_LEVEL=ERROR
Set 'PREFECT_LOGGING_LOGGERS_PREFECT_FLOW_RUNS_LEVEL' to 'ERROR'.
$ prefect config view
PREFECT_LOGGING_LOGGERS_PREFECT_FLOW_RUNS_LEVEL='ERROR' (from profile)
# load_logging_config -> prefect.flow_runs level = ERROR, prefect.task_runs unchanged

Regression tests added in tests/cli/test_config.py, tests/test_logging.py. tests/test_settings.py updated (SUPPORTED_SETTINGS, regenerated _types.py). Full tests/test_settings.py, tests/cli/test_config.py, tests/test_logging.py pass; ruff clean; no new mypy errors.

Open questions for maintainers

Naming of the field (logging.overrides) — happy to rename.
Storing override keys as Setting wrappers in profiles vs. a dedicated mechanism — feedback welcome.

Checklist

This pull request references any related issue by including "closes <link to issue>"
If this pull request adds new functionality, it includes unit tests that cover the changes
If this pull request removes docs files, it includes redirect settings in mint.json.
If this pull request adds functions or classes, it includes helpful docstrings.

codspeed-hq · 2026-06-30T10:47:42Z

Merging this PR will not alter performance

✅ 8 untouched benchmarks
⏩ 1 skipped benchmark¹

_{Comparing Abdulrehman-PIAIC80387:fix-config-set-logging-overrides (63a2aab) with main (b75c060)}

1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: faa2573c02

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-06-30T11:06:29Z

+        elif setting_name in valid_logging_overrides:
+            parsed.add(_logging_override_setting(setting_name))
+        else:
            exit_with_error(f"Unknown setting name {setting_name!r}.")


Allow stored logging overrides to be unset

When a profile contains an override from a custom logging.yml and that file is later removed, changed, or the logging config path is unset, this validation rejects the key because it is no longer returned by _valid_logging_override_names() before checking the active profile. The stored PREFECT_LOGGING_* entry then cannot be removed with prefect config unset ..., leaving users to edit profiles.toml by hand; unset should also accept logging override names already present in profile.settings.

Useful? React with 👍 / 👎.

Fixed in b59ca3988 — config unset now also accepts any PREFECT_LOGGING_* key already stored in the active profile, so a stored override can always be removed even when it is no longer a valid override key (e.g. the logging config file changed). Added a regression test (test_unset_stored_logging_override_that_is_no_longer_valid).

desertaxle

Thanks for the PR @Abdulrehman-PIAIC80387! Looks like this mostly works based on some testing, but I think this needs one more fix before merge. The issue is that prefect config view doesn't show the effective value/source for custom logging overrides when the same PREFECT_LOGGING_* key is set in the environment. Runtime precedence is correct, but the CLI display is stale/misleading.

Would you be able to fix that in this PR?

Abdulrehman-PIAIC80387 · 2026-07-10T06:04:17Z

Thanks @desertaxle! Fixed in 02362a144.

prefect config view now surfaces custom PREFECT_LOGGING_* overrides set via environment variables with the env source, and gives them precedence over the profile value — matching runtime resolution (env > profile). So a key set in both the environment and a profile now displays the env value/source instead of the stale profile one, and a key set only in the environment is shown too (previously it was omitted).

# profile has WARNING, env sets ERROR
PREFECT_LOGGING_LOGGERS_PREFECT_FLOW_RUNS_LEVEL='ERROR' (from env)

Added regression tests test_view_shows_env_precedence_for_logging_override and test_view_shows_logging_override_set_only_in_env.

closes PrefectHQ#18666 Custom PREFECT_LOGGING_* keys that map into logging.yml (e.g. PREFECT_LOGGING_LOGGERS_PREFECT_FLOW_RUNS_LEVEL) are now first-class settings. A LoggingSettings.overrides field is populated by a new LoggingOverridesSource that collects unmatched PREFECT_LOGGING_* keys from the environment and profiles (cheap key filtering only, no logging.yml parsing on import). load_logging_config applies them via get_current_settings(), and the CLI accepts them in config set/unset/view.

…d keys Addresses review feedback: config unset now removes any stored PREFECT_LOGGING_* override from the profile even if it is no longer returned by the logging config key validation (e.g. the logging config file changed or was removed).

Memoize _declared_env_names per settings class so the LoggingOverridesSource does not re-walk the model fields on every settings build.

Addresses review feedback: config view now surfaces custom PREFECT_LOGGING_* overrides set via environment variables with the 'env' source, taking precedence over the profile value to match runtime resolution. Previously an override set in both the environment and a profile displayed the (stale) profile value, and an override set only in the environment was not shown at all.

desertaxle

Thanks for the updates @Abdulrehman-PIAIC80387! I found a couple more edge cases while testing that should be resolved before this PR is merged.

desertaxle · 2026-07-10T14:05:18Z

+class LoggingOverridesSource(PydanticBaseSettingsSource):
+    """Collect `PREFECT_LOGGING_*` keys that do not map to a declared field into the
+    `overrides` field.
+
+    This lets logging configuration file (logging.yml) paths such as
+    `PREFECT_LOGGING_LOGGERS_PREFECT_FLOW_RUNS_LEVEL` be configured through the
+    settings system (environment variables and profiles) and show up in
+    `get_current_settings()` / `prefect config view`.
+
+    Only cheap key filtering is performed here; the logging configuration file is
+    never read or parsed at this layer.
+    """
+
+    def __init__(self, settings_cls: Type[BaseSettings]):
+        super().__init__(settings_cls)
+        self.settings_cls = settings_cls
+        self._declared = _declared_env_names(settings_cls)
+
+    def get_field_value(
+        self, field: FieldInfo, field_name: str
+    ) -> Tuple[Any, str, bool]:
+        return None, field_name, False
+
+    def _is_override_key(self, key: str) -> bool:
+        return key.startswith("PREFECT_LOGGING_") and key not in self._declared
+
+    def _collect(self) -> Dict[str, str]:
+        overrides: Dict[str, str] = {}
+        # Lower priority: profile settings (config set writes here)
+        try:
+            profile_settings = ProfileSettingsTomlLoader(
+                self.settings_cls
+            ).profile_settings
+        except Exception:
+            profile_settings = {}
+        for key, value in profile_settings.items():
+            upper = key.upper()
+            if self._is_override_key(upper):
+                overrides[upper] = str(value)
+        # Higher priority: environment variables override profile values
+        for key, value in os.environ.items():
+            upper = key.upper()
+            if self._is_override_key(upper):
+                overrides[upper] = value
+        return overrides
+
+    def __call__(self) -> Dict[str, Any]:
+        overrides = self._collect()
+        return {"overrides": overrides} if overrides else {}


I think this should be split into two different classes because environment variables and profiles have two different priorities when resolving settings.

desertaxle · 2026-07-10T14:06:09Z

+            dotenv_settings,
+            file_secret_settings,
+        )
+        return (*sources, LoggingOverridesSource(settings_cls))


Because the LoggingOverridesSource is appended to the end of the sources, configuration in a TOML file will override environment variables, which doesn't align with the existing convention.

I mentioned this in src/prefect/settings/sources.py, but this seems like a good reason to split LoggingOverridesSource into two different classes.

desertaxle · 2026-07-10T14:07:37Z

+    # Custom logging overrides set via environment variables. The environment takes
+    # precedence over the profile at runtime, so surface it here first (later sources
+    # skip names already processed) to keep the displayed source accurate.
+    for env_name, env_value in os.environ.items():
+        upper = env_name.upper()
+        if not upper.startswith("PREFECT_LOGGING_"):
+            continue
+        if upper in valid_setting_names or upper in processed_settings:
+            continue
+        if upper in valid_logging_overrides or upper in profile_override_names:
+            _process_setting(_logging_override_setting(upper), env_value, "env")


Looks like this handles logging config via env vars, but we'll also want to handle display from a prefect.toml source.

desertaxle · 2026-07-10T14:14:20Z

+    names: set[str] = set(get_valid_setting_overrides(DEFAULT_LOGGING_SETTINGS_PATH))
+    configured = get_current_settings().logging.config_path
+    if configured and Path(configured).exists():
+        names |= get_valid_setting_overrides(Path(configured))


This looks like it merges the default logging config with a custom logging config file if one exists, but at runtime we completely ignore the default file if a custom logging file is provided. Can you align this helper with the runtime behavior?

github-actions Bot added bug Something isn't working status:stale labels Jun 30, 2026

github-actions Bot removed the status:stale label Jun 30, 2026

Abdulrehman-PIAIC80387 marked this pull request as ready for review June 30, 2026 11:01

Abdulrehman-PIAIC80387 requested review from chrisguidry, cicdw, desertaxle and zzstoatzz as code owners June 30, 2026 11:01

github-actions Bot added the status:stale label Jun 30, 2026

chatgpt-codex-connector Bot reviewed Jun 30, 2026

View reviewed changes

Abdulrehman-PIAIC80387 force-pushed the fix-config-set-logging-overrides branch from 05daecf to 3789f6f Compare July 1, 2026 07:34

github-actions Bot removed the status:stale label Jul 1, 2026

desertaxle requested changes Jul 9, 2026

View reviewed changes

Abdulrehman-PIAIC80387 added 4 commits July 10, 2026 11:26

perf: cache declared logging env-var names in override source

8ffc209

Memoize _declared_env_names per settings class so the LoggingOverridesSource does not re-walk the model fields on every settings build.

Abdulrehman-PIAIC80387 force-pushed the fix-config-set-logging-overrides branch from 02362a1 to 63a2aab Compare July 10, 2026 06:30

desertaxle requested changes Jul 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: support setting logging.yml overrides via prefect config set#22411

fix: support setting logging.yml overrides via prefect config set#22411
Abdulrehman-PIAIC80387 wants to merge 4 commits into
PrefectHQ:mainfrom
Abdulrehman-PIAIC80387:fix-config-set-logging-overrides

Abdulrehman-PIAIC80387 commented Jun 30, 2026

Uh oh!

codspeed-hq Bot commented Jun 30, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 30, 2026

Uh oh!

Abdulrehman-PIAIC80387 Jun 30, 2026

Uh oh!

desertaxle left a comment

Uh oh!

Abdulrehman-PIAIC80387 commented Jul 10, 2026

Uh oh!

desertaxle left a comment

Uh oh!

desertaxle Jul 10, 2026

Uh oh!

desertaxle Jul 10, 2026

Uh oh!

desertaxle Jul 10, 2026

Uh oh!

desertaxle Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Abdulrehman-PIAIC80387 commented Jun 30, 2026

Root cause

Fix

How this avoids #18670's failure modes

Verification

Open questions for maintainers

Checklist

Uh oh!

codspeed-hq Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Abdulrehman-PIAIC80387 Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

desertaxle left a comment

Choose a reason for hiding this comment

Uh oh!

Abdulrehman-PIAIC80387 commented Jul 10, 2026

Uh oh!

desertaxle left a comment

Choose a reason for hiding this comment

Uh oh!

desertaxle Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

desertaxle Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

desertaxle Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

desertaxle Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codspeed-hq Bot commented Jun 30, 2026 •

edited

Loading