📢 Policy Update: Mandatory Test Data Licensing & Attribution Requirements

[Yaswant Pradhan (yaswant)]

We have introduced a new policy regarding the inclusion of ancillary test data across all Met Office managed systems and related external platforms (such as JULES data on JASMIN, or Met Office managed files in BIG_DATA_DIR or ANCILDIR).
Please review these prerequisites before submitting your next Pull Request or testing framework change.

🚨 Key Requirements

• Mandatory Approval: All test data must be approved by an Information Asset Owner (IAO) prior to deployment. We'll update the pull request template so that the contributors and reviewers are aware of the process.
• Licence & Provenance: Authors must provide a record of data origin, generation methods, and licensing terms. Derived works and multi-source datasets must feature compatible licensing.
• Early Engagement: Contact the IAO (or email the MIAO team for UM datasets) early in your development process to avoid PR delays.

🛠️ Metadata Formatting Standards

All file metadata must be recorded based on the file format:

• NetCDF Files: Include information within the file using global attributes:

  • license (ESIP/ACDD standard) - required
  • title (a clear description of what is in the dataset) - required
  • references (for papers, CF conventions) - required
  • institution (for organisations, CF conventions) - required
  • source (see CF conventions)
  • history (see CF conventions)
  • acknowledgement (for individuals, ESIP/ACDD standard)
  • restrictions (for usage constraints like "research only")

• Other File Formats: Follow existing UM ANCILDIR conventions or include information in separate, accompanying plain-text files located in the same directory. Use the exact same base name as the data file with these suffixes:

  • .license
  • .attribution
  • .restrictions

---

💬 Questions or feedback? Please drop a comment in this discussion thread so we can address it transparently.

[iboutle]

For netcdf files produced by lfric (checkpoint files, um2lfric dumps etc), do these already have the required attributes appended or are there instructions somewhere on how to do this?

[mo-marqh]

thanks Yaswant Pradhan (@yaswant)

following iboutle 's observations, i think that there is not currently functionality within LFRic to meet these requirements when writing files.

I'm quite concerned about expecting files to be updated using external command line / programming tools to adapt the files created by LFRic. whilst this can work, it brings significant risk in terms of file reproducability and unintended change.

I'm also a bit wary about including code in LFRic configurations to enable the simulation system to meet this criteria set when needed, and behave usefully in cases where such provenance content metadata is not required.

of the two facets of wariness, I think that LFRic Adaptions are potentially safer, but will need developing and testing.

just throw the NCO Kitchen Sink at it is quite a scary option to me for data management and performance management.

[iboutle]

Yes, this is my worry - I can see why this information might be required for data coming from an external source (like ancillary files), but for data written by the model itself, surely the answer to all the questions is exactly the same for everyone producing that data with the model, and as such it should just be written by default or provided by a standard template. We as users are not the right people to know what the answers to many of these questions will be!

[Yaswant Pradhan (yaswant)]

Thank you iboutle and mo-marqh for engaging. Since these requirements are effective immediately, I think we should aim to avoid risky post-processing tools like NCO. What we need is an automated, reproducible solution built directly into the simulation systems.

I agree that individual developers should not be guessing legal licensing terms or institutional references, so I am in favour of a standard template for data written by the model itself. Writing global attributes at the moment of file creation via the netcdf API has zero noticeable performance impact, whereas post-processing huge netcdfs using third-party utils can be expensive and unmaintainable.

I'd also argue that external data (e.g. a new soil map from partners) must include proper attribution and licensing terms. And ideally they should never creep into model dumps; however, in practice I have noticed that they do which introduces additional complication.

I think we have a lot of work to do...

[iboutle]

Just for info Yaswant Pradhan (@yaswant), the plan we have as part of MetOffice/lfric_apps#494 requires a lot of new input files for the test suite to be included as part of the current release cycle. Paul tells me the ancillaries themselves are compliant, but I will need some instructions for what to do about the start dumps within the next few weeks please.

[Yaswant Pradhan (yaswant)]

Hi iboutle - that's pretty tight, and you have a valid question. Are these start-dumps required for partner testing? If they do, we cant distribute the data without proper attribution. I'd seek some advice from Ben Fitzpatrick (@benfitzpatrick) and Andrew Clark (@arjclark) regarding instructions.

The key message is that the transition to formal data licensing is causing necessary delays, meaning we must invest some effort into tracking down the origins of data intended for external distribution. If this tracking effort fails, we will pivot to discussing risk mitigations instead.

[Andrew Clark (arjclark)]

iboutle - you'll want to have a conversation with Ben Fitzpatrick (@benfitzpatrick) as IAO and final say on this. I'd strongly suggest you book something in with him as a priority to discuss this.

However, I understand that your model dumps contain ancillary files? If this is so, then at minimum I believe you need to be accompanying the dumps themselves with the licensing, attribution, etc. associated with those ancillary files (as stored in fields in your dumps). Orography, for instance, is a good example of having that in your file means you should have the associated license etc. because you are in effect redistributing someone else's IP and need to follow the terms of that.

I have not seen the MetOffice/lfric_apps#494 ancillary files, so cannot comment on those. If you already have them, then feel free to message me to take a look at what you're proposing distributing and I can give you an initial look at whether they are policy compliant or not (IAO needs to sign off at the end though). I can prioritise looking at that for you first thing next week. Please don't send me links to files on our disks on this public forum though and please be cognoscente that I don't know where "big data" is so full paths will be needed (and I don't have time to go digging through tickets or config files to try extract paths for myself).

Based on what we've done elsewhere for historic files, if it isn't viable/sensible/safe to edit existing files in place or resolve LFRic codes to support recording of data in the dumps themselves on these timescales then I believe an IAO would be supportive of you providing sidecar files alongside them. In practice, it is not always viable to encode metadata of this nature in the files themselves - see e.g. UM files - so clearly associated and clearly marked information that is distributed alongside can typically be considered an acceptable method of meeting IPR requirements - as long as we make sure we have processes in place and followed to make sure that's done.

Happy to advise where I can, but it's Ben Fitzpatrick (@benfitzpatrick) that will have final say - I can just help make that a shorter conversation 😄