One (PVNet) app to rule them all (⛰️💍)

[peterdudfield]

We currently have

• uk-pvnet-app
• india-forecast-app
• site-forecast-app

Note that uk-pvnet-app currently save to national-gsp database, where as india-forecast-app and site-forecast-app saves to site database. We hope to mvoe these all to the data-platform soon

It would be nice to just have one app, that is configured in different ways.
This means data checks on NWP and Satellite, can be used throughout.

Other upgrades, india-forecast-app currently runs 1 site at a time, but really we could batch these up and run them together.

Slightly negative of having everything together, is perhaps development speed is reduced.

[peterdudfield]

Im going to close this. I think we are happy with uk-pvnet-app and site-forecast-app

[peterdudfield]

I thought it would be nice to write down what the current forecasters have. This helps plan the features we want in the one forecast app to rule them all

Updated 2026-05-20

Feature	uk-pvnet-app	site-forecast-app
pydantic config for models	yes	yes
runs pvnet	yes	yes
runs summatiomal model	yes	yes
loop over	models	sites and models
load generation values	no	yes
load capacities	yes	yes
data validation	X%	Y%
batch validation	yes	yes
save to data-platform	yes	yes
read to data-platform	yes	yes
removed datamodel	yes	no
can run other models than PVNet	no	yes
code quality (not sure how much should trust ChatGPT on this)	7/10	6/10
Dynamic for groups of sites	no	yes
NL Curtailment	no	yes
Other

[Sukh-P]

Would like to create an implementation plan for this in Q2 26, alongside the work to add data platform to both of these tables which will make it much easier to do this

[peterdudfield]

We did some feature planning here, next steps are to

https://miro.com/app/board/uXjVHGr-b9k=/
Next steps

• Move to github (Peter)
• Plan config (Peter v0)
• Plan the main app structure
• Plan module structure
• Build

[peterdudfield]

General notes for this app

• We want to be able to select groups of models to run, run N models with N locations
• R&D models (or older data-sampler) should live on branches. This allows the flexibility of high quality code, but also allows us to use it for R&D
• Lets use data contracts (some in ocf-data-sampler) as needed
• Only data-platform, lets not include the old datamodels
• Just PVNet for the moment
• Load from icechunk for satellite data
• Clear logging if some models failed
• Deploy with new trained models, lets try to minimize the edge cases around old models, by just retraining new models
• Run on one energy source at once i.e only wind or pv
• Aim for one end-to-end tests
• Run models even if other models can’t be run
• Module code
• We want to be able to run the app locally

General / Infrastructure

• Logging, clear logging and discuss about what is the right level
• Tests, shold aim for X% coverage
• Linting shold be part of it
• CI, see standard OCF CI templates)
• Docker, we need to build a docker file
• use uv when possible, put we might need conda due to regridding library
• Regions: NL, GB, India + …

Data Layer

• Get generation data
• Get NWP data
• Get NL DA prices (might also be some other countries data we need from enstoe)
• Location metadata, like capacity and lat, long
• Satellite (and backup satellite)
• Validation for example no nans
• Validate enough timestamps that the data sources have enough timestamps, and remove models if needed
• Only download data once from S3
• Processing inputs
  • Regridding for MO and ...
  • options for Unit scale
  • Support icechunk (trim out stuff) for satellite

Forecasting

• Run on all GSPs (for UK)
• Run on a site group
• Run for NL
• Download models from HF
• Run PVNet (and only PVNet)
• Run summation model (only if neded)
• We want to use ML batches, not one sample at a time
• Lets do some batch validation

Post-processing

• Run curtailment (NL)
• Night time zeros
• run adjuster, note we need options for different observes in the adjuster
• validation, like no zig zag forecasts, or no forecast above X% of capacity

Saving

• Data platform, and lets save Input data metadata (with satellite)
• Option to save to csv
• Save uncurtailed models too

Configuration

• HF hashes for model
• HF hashes for summation model
• Grouping (e.g., UK, NL, India) or site groups (perhaps need to think about all the different use cases here)
• PV / wind (run one at a time)
• Different models, different processing
• Saving
• Post-processing (adjuster, curtailment, etc.)

Misc

• Lets include NL Blend, and later decide if we want to bring UK blend into this

[peterdudfield]

WIP: Here's a draft of what the configuration could look like

I'm trying to make one big configuration that has everything in it, like a list of all models.
Then have a second configuration (much smaller) that is more used for running the app

App configuration

data:
- [InputData]
models:
- [Model]
models_locations:
- [ModelsAndLocations].  
processing_steps:
- [ProcessingStep]

Model:
- name
- hf 
    - repo
    - hash
- summation_model: Model
- processing_steps: list[ProcessingStep.name]
- data
    - [InputData.name] 

ModelsAndLocations (this is like uk, nl or india or ....)
- name (e.g. uk_gsp_and_national, or nl_regions_and_national)
- model_names: [Model.name]
- location_type 
- location_type_summational
- organization_filter (optional, this is a way to filter on different ones, for example use NL National not UK nationa. We might want a more defined solution in the data-platform)
- energy_source
- processing_steps: [ProcessingStep.name] 

InputData:
- name
- type (nwp or satellite or ...)
- location (env_var)
- backup_location (env_var)
- processing_config: [ProcessingStep.name]
- observer_name: Optional

ProcessingStep
- name (e.g. nl_curtailed, adjuster)
- class: 
- type: e.g after model run, before save, inputdata
- add other inputs? (not quite sure how to do this)

The models are basically what we have here

App run configuration

This will be more like env_vars set

models_and_locations:
save_method: 
input_data_names: e.g satellite, satellite_backup, nwp_ecwmwf, nwp_mo, e.t.c

[peterdudfield]

Here's a possible example of a yaml file

models:
  - name: pvnet_v2 
    hf:
        repo: openclimatefix-models/pvnet_uk_region
        commit: cfcb233576c3c8110daed0179452e3d9913c744c
    summation: pvnet_v2_summation
    data: [uk_sat, uk_nwp_ecwmwf, uk_nwp_mo]
  - name: pvnet_v2_summation
    hf:
        repo: openclimatefix-models/pvnet_uk_region
        commit: cfcb233576c3c8110daed0179452e3d9913c744c
  - name: pvnet_ecmwf 
    hf:
        repo: openclimatefix-models/pvnet_uk_region
        commit: 6b7cc5fec03b83837672a759d7e00f249685de6b
    summation: pvnet_ecmwf_summation
    data: [uk_nwp_ecmwf]
  - name: pvnet_ecmwf_summation
    hf:
        repo: openclimatefix-models/pvnet_uk_region
        commit: 0c26b19ab90001ee51f9886227be8286e84a393c

models_locations:
- name: uk_gsp_and_national
  models_names:
  - pvnet_v2
  location_type: gsp
  location_type_summation: nation
  energy_source: solar
  processing_steps:
  - validate_zig_zag
- name: nl
  models_names:
  - TODO
  location_type: state
  location_type_summation: nation
  energy_source: solar
  processing_steps:
  - validate_zig_zag

data:
- name: uk_sat
  type: satellite
  location: ENV_VAR
  back_location: ENV_VAR
  processing_config: [Check_satellite_nans, check_timestamps]
- name: uk_nwp_ecwmwf
  type: nwp
  location: ENV_VAR
  back_location: ENV_VAR
  processing_config: [Check_nwp_nans]
- name: uk_nwp_mo
  type: nwp
  location: ENV_VAR
  back_location: ENV_VAR
  processing_config: [Check_nwp_nans]


processing_steps:
- name: Check_satellite_nans
  class: CheckSatelliteNans
  type: post_data_check
- name: Check_nwp_nans
  class: CheckNwpNans
  type: post_data_check
- name: check_timestamps
  class: CheckTimestamps
  type: post_data_check
- name: validate_zig_zag
  class: ValidateZigZag
  type: post_forecast_check

[peterdudfield]

Feedback

• make country while registry
• Dont want to load env vars in code, so make sure they are all in config. There are some ways when loading yaml to get env vars in the settings
• Processing step could just be a Class and not config
• Need thing about how we pass in HuggingFace token, aws region, zigzag thresholds, save batch dir, scratch_dir, dataplatform address + port,
• tags on models would be useful, like code, extra ....

[peterdudfield]

v0.2

Idea is to have a registry file for each country (or group) and then a main settings file

Registry configuration

- name (e.g. uk_gsp_and_national, or nl_regions_and_national)
- location_type 
- location_type_summational
- organization_filter (optional, this is a way to filter on different ones, for example use NL National not UK nationa. We might want a more defined solution in the data-platform)
- energy_source
- processing_steps: [ProcessingStep.name] 
- model_names: [Model.name]
- tags: [str]

Model:
- name
- hf 
    - repo
    - hash
- summation_model: Optional[Model]
- processing_steps: list[ProcessingStep.name]
 - tags: Optional[list[str]]

The models are basically what we have here

App settings

This can be set with env_vars and then when it loads the YAML we can load the envvars in

- registry_config_name:
- save_method: 
- hugging_face_token:
- aws_region:
- scratch_dir: 
- data_platform_address: (host and address)
- (other one off variables) ...
- input_data: [InputData]
- processing_step: [ProcessingStep]

InputData:
- name
- type (nwp or satellite or ...)
- location (env_var)
- processing_config: [ProcessingStep.name]
- observer_name: Optional
- kwargs # can put in a list of things like `observer_name` and `backup_location`

ProcessingStep
- name (e.g. nl_curtailed, adjuster)
- class: 
- type: e.g after model run, before save, inputdata
- kwargs # can put in things like `adjuster_limit_mw` or `zigzag_threshold`

Example - uk_gsp_and_national.yaml

name: uk_gsp_and_national
location_type: gsp
location_type_summation: nation
energy_source: solar
processing_steps:
- validate_zig_zag
- adjuster
models:
- name: pvnet_v2 
   hf:
     repo: openclimatefix-models/pvnet_uk_region
     commit: cfcb233576c3c8110daed0179452e3d9913c744c
   summation: pvnet_v2_summation
  tags: core
- name: pvnet_v2_summation
  hf:
      repo: openclimatefix-models/pvnet_uk_region_summational
      commit: cfcb233576c3c8110daed0179452e3d9913c744c
   tags: summation
- name: pvnet_mo
   hf:
      repo: openclimatefix-models/pvnet_uk_region
      commit: 6b7cc5fec03b83837672a759d7e00f249685de6b
  summation: pvnet_mo_summation
  tags: extra
- name: pvnet_mo_summation
  hf:
      repo: openclimatefix-models/pvnet_uk_region_summational
      commit: 0c26b19ab90001ee51f9886227be8286e84a393c
  tags: summation

Example 2 - nl

name: nl_state_and_national
location_type: state
location_type_summation: nation
energy_source: solar
processing_steps:
- adjuster
models:
- name: nl_all
   hf:
     repo: openclimatefix-models/pvnet_uk_region
     commit: cfcb233576c3c8110daed0179452e3d9913c744c
  summation: nl_all_summation
  processing_steps: 
  - nl_curtail_and_uncurtailed # save both the curtailed and uncurtailed, by DA prices
  tags: core
- name: nl_all_summation
  hf:
      repo: openclimatefix-models/pvnet_uk_region
      commit: cfcb233576c3c8110daed0179452e3d9913c744c
   tags: summation
- name: nl_ecmwf 
   hf:
      repo: openclimatefix-models/pvnet_uk_region
      commit: 6b7cc5fec03b83837672a759d7e00f249685de6b
  summation: nl_ecmwf_summation
  processing_steps: 
  - nl_curtail # curtail the forecast by DA prices
  tags: extra
- name: nl_ecmwf_summation
  hf:
      repo: openclimatefix-models/pvnet_uk_region
      commit: 0c26b19ab90001ee51f9886227be8286e84a393c
  tags: summation

Settings config

registry_config_name: REGISTRY_CONFIG_NAME
save_method: SAVE_METHOD
hugging_face_token: HUGGING_FACE_TOKEN
aws_region: AWS_REGION
scratch_dir: SCRATCH_DIR
data_platform_address: DP_ADDRESS:DP_HOST

input_data: 
- name: uk_sat
  type: satellite
  location: SATELLITE_ICECHUNK
  processing_config: [Check_satellite_nans, check_timestamps]
  kwargs:
  - satellite_icechunk_backup:SATELLITE_ICECHUNK_BACKUP
- name: uk_nwp_ecmwf
  type: nwp
  location: NWP_ECMWF_ZARR
  processing_config: [Check_nwp_nans]
- name: uk_nwp_mo
  type: nwp
  location: NWP_MO_ZARR
  processing_config: [Check_nwp_nans]

processing_steps:
- name: Check_satellite_nans
  class: CheckSatelliteNans
  type: post_data_check
- name: Check_nwp_nans
  class: CheckNwpNans
  type: post_data_check
- name: check_timestamps
  class: CheckTimestamps
  type: post_data_check
- name: validate_zig_zag
  class: ValidateZigZag
  kwargs:
  - forecast_validate_zig_zag_error: FORECAST_VALIDATE_ZIG_ZAG_ERROR
  - forecast_validate_zig_zag_warning: FORECAST_VALIDATE_ZIG_ZAG_WARNING
  type: post_forecast_check
- name: nl_curtail
  class: NLCurtailed
- name: nl_curtail_and_uncurtailed
  class: NLCurtailedAndUncurtailed
- adjuster:
  class: Adjuster
  kwargs:
  - observer_name: OBSERVER_NAME
  - uncurtailed_observer_name: UNCURTAILED_OBSERVER_NAME
  - adjuster_limit_mw:ADJUSTER_LIMIT_MW
  - adjuster_limit_per:ADJUSTER_LIMIT_PER