Skip to content

Add configs for backup universe files and separate underlying/derivative history providers - #42

Merged
jhonabreul merged 3 commits into
QuantConnect:masterfrom
jhonabreul:feature-backup-universe-files
Aug 13, 2026
Merged

Add configs for backup universe files and separate underlying/derivative history providers#42
jhonabreul merged 3 commits into
QuantConnect:masterfrom
jhonabreul:feature-backup-universe-files

Conversation

@jhonabreul

@jhonabreul jhonabreul commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Description

Two additions to the universe generation configuration (one commit each), plus a test infrastructure fix:

Backup universe files

  • New bool config universe-generation-backup-files (default false), read in the DerivativeUniverseGenerator constructor.
  • When enabled, GetUniverseFileName suffixes every generated universe file with .backup (e.g. 20260810.csv.backup). This is the single choke point for all generated universe files (options, index options, futures, future options).
  • Lean can use these backup files in live trading as a last-resort fallback when the expected universe files are not available yet (Fall back to backup universe files in live trading when the expected ones are unavailable Lean#9690).

Separate history providers per leg

  • New configs universe-generation-underlying-history-provider and universe-generation-derivative-history-provider, read in the base Program. Each accepts the same format as history-provider (a single type name or a JSON list) and builds its own HistoryProviderManager.
  • When a key is not set (the default), that leg falls back to the shared manager built from history-provider — behavior is unchanged.
  • DerivativeUniverseGenerator now routes underlying history requests and derivative history requests through their own providers (_underlyingHistoryProvider / _derivativeHistoryProvider); the original single-provider constructor delegates with the same instance for both legs.

Test setup fix (pre-existing CI failures)

  • ImpliedVolatilityInterpolatorTests (and locally IndexHistoryProviderTests) were failing before this PR: Lean resolves the map file and factor file providers with Composer.GetPart<T>, which does not compose new instances, so in the bare test host any symbol mapping or corporate events lookup hit a null provider (Symbol.CreateSecurityIdentifier.GetFirstTickerAndDate NRE; DividendYieldProvider.LoadCorporateEvents NRE).
  • A new [SetUpFixture] (TestSetup) registers and initializes both providers once before any test runs, the same way Program does at startup.

Related Issue

N/A

Motivation and Context

  • Backup files give live trading a safety net for days when the expected universe files are late or missing.
  • Underlying and derivative history have different requirements (e.g. index underlyings served by IndexHistoryProvider), so each leg can now be pointed at its own provider chain instead of sharing one provider list for everything.

Requires Documentation Change

No

How Has This Been Tested?

  • New DerivativeUniverseGeneratorTests.GeneratesUniverseFilesAsBackupFilesWhenConfigured: generated universe file name with the config off (<date>.csv) and on (<date>.csv.backup).
  • New DerivativeUniverseGeneratorTests.UnderlyingAndDerivativeHistoryRequestsShareASingleProviderByDefault: with the single-provider constructor, all underlying and derivative requests land on the one provider.
  • New DerivativeUniverseGeneratorTests.UnderlyingAndDerivativeHistoryRequestsAreRoutedToTheirOwnProviders: each leg's requests land only on its own provider.
  • Solution builds and the fixture passes at each commit.
  • With the test setup fix, the full test project passes locally: 19/19 (previously 10 pre-existing failures in ImpliedVolatilityInterpolatorTests, which also failed in CI).
  • The Lean-side consumption of the backup files was verified in Fall back to backup universe files in live trading when the expected ones are unavailable Lean#9690 (unit tests plus live-paper deployments).

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor (non-breaking change which improves implementation)
  • Performance (non-breaking change which improves performance. Please add associated performance test and results)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Non-functional change (xml comments/documentation/etc)

Checklist:

  • My code follows the code style of this project.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

@jhonabreul
jhonabreul marked this pull request as ready for review August 13, 2026 22:00
@jhonabreul
jhonabreul merged commit e732bd6 into QuantConnect:master Aug 13, 2026
1 check passed
@jhonabreul
jhonabreul deleted the feature-backup-universe-files branch August 13, 2026 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants