56 decide on and implement dag solving strategy - #65
Merged
Merged
Conversation
pabloitu
commented
Nov 1, 2025
Collaborator
- Implementation to run an experiment concurrently, with tasks have associated dependencies, and are placed into a directed acyclic graph. This mode is only available for time dependent models, whose source code is encapsulated in a docker image. In practice, catalog and evaluations tasks are sequential, whereas forecasts creation is run in parallel. User can set run_mode in the configuration file as sequential (default) or parallel.
- Decoupled engine runner from profiling. Profiler prints (in debug) a timing summary grouped by task class. Added catch for keyboard interrupts, so all running processes can be killed before exiting (critical for parallel running and container deployment).
- Minor rework of Task.str, such that it prints its instance name by default.
- Docker environments now properly catch errors and logs. Added also a static method, called from TaskGraph, that effectively kills all containers through the python SDK.
- To handle parallelization, models require to have a unique input folder for each time window. Therefore, a volume is set between a folder from the host filesystem (other than model/input), to /app/model/input within the container. Contents are located dynamically before running the container, by the methods CatalogRepository.set_input_cats, Model.prepare_args and Model.prepare_extra_input.
- Important fix on time_config parsing, which sometimes caused that time-dependent experiments were not recognized as such. Also, catalog repository now parses existing dropped catalog, rather than filtering it for each task.
- Updated all of the existing unit tests, extended engine and environments tests. created engine-model and engine-experiment integration tests. Added remaining tutorial cases (h,i,j) for end-to-end tests, which can only be run locally for processing time and space limitations in the CI.
- Added run_mode and force_rerun parameters in experiment configurations. Updated all refactored classes from last commits
…han filtering it for each task.
…tasks have associated dependencies, and are placed into a directed acyclic graph. This mode is only available for time dependent models, whose source code is encapsulated in a docker image. In practice, catalog and evaluations tasks are sequential, whereas forecasts creation is run in parallel. User can set run_mode in the configuration file as sequential (default) or parallel. ft: decoupled engine runner from profiling. Profiler prints (in debug) a timing summary grouped by task class. Added catch for keyboard interrupts, so all running processes can be killed before exiting (critical for parallel running and container deployment). ft: minor rework of Task.__str__, such that it prints its instance name by default. ft: docker environments now properly catch errors and logs. Added also a static method, called from TaskGraph, that effectively kills all containers through the python SDK. ft: to handle parallelization, models require to have a unique input folder for each time window. Therefore, a volume is set between a folder from the host filesystem (other than model/input), to /app/model/input within the container. Contents are located dynamically before running the container, by the methods CatalogRepository.set_input_cats, Model.prepare_args and Model.prepare_extra_input. fix: important fix on time_config parsing, which sometimes caused that time-dependent experiments were not recognized as such. tests: updated all of the existing unit tests, extended engine and environments tests. created engine-model and engine-experiment integration tests. Added remaining tutorial cases (h,i,j) for end-to-end tests, which can only be run locally for processing time and space limitations in the CI. docs: Added run_mode and force_rerun parameters in experiment configurations. Updated all refactored classes from last commits
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.