Add metadata to the flowrep dictionary?

[samwaseda]

For quite some time I'm wondering how to pass metadata to the knowledge graph. In particular I would need the following information:

• Representation format (flowrep, semantikon, PWD etc.)
• User information (name, email address)
• Server data
• Date (start - end)

Should we create a new field? metadata?

[liamhuber]

This was somewhat on my radar already because we need somewhere to dump what version of flowrep is used for the recipe. However, as with that, all of these seem like once-per-file things and not once-per-node things, so I'm not sure whether such a field belongs in the recipe, or something we rather store beside the recipe when we dump it to a file or send it off for KG digestion. If you think we can do it beside, that's my current preference.

[samwaseda]

all of these seem like once-per-file things and not once-per-node things

Well, except for the representation format, there's a possibility that they differ for different nodes (as a matter of fact, date definitely does). So I guess there should be a field on the node level

[liamhuber]

Date is only for the retrospective data though, yeah? So we can at least kick that can.

I hadn't thought of mixing representations inside a single recipe object. I guess we could. It is another thing that I'm not sure we should.

[samwaseda]

No representations shouldn't be mixed (I guessed you missed this part in my comment?) but it's true that the rest is mostly about retrospective, but shouldn't the retrospective representation be the prospective representation + data?

[liamhuber]

but shouldn't the retrospective representation be the prospective representation + data

Agreed, that would be my first line of a attack. It's not necessarily the case though. For instance, the retrospective isn't going to need the same non-DAG elements. And indeed, as Niklas mentioned in this weeks meeting, figuring out how to express "this retrospective DAG was produced by this prospective dynamic node" might be an ontological headache, so it's always possible the pro- and retrospective formats don't wind up being trivially related by a data extension.

[liamhuber]

@samwaseda, this question is important to me, but I think we can now migrate the discussion out of flowrep.

The question of how to indicate ontologically that some result comes from some recipe is probably best followed-up in semantikon. Here, the "instance view" data simply directly links the result DAG to the underlying recipe, so we've got the data-side link we need already:

flowrep/src/flowrep/retrospective.py

Lines 82 to 89 in c5a1b68

	RecipeType = TypeVar("RecipeType", bound=base_models.NodeRecipe)
	@dataclasses.dataclass(frozen=False)
	class NodeData(Generic[RecipeType], abc.ABC):
	recipe: RecipeType
	input_ports: InputDataPorts
	output_ports: OutputDataPorts

The other metadata you listed in the opening post is all stuff that looks runtime-related to me, not something for putting in the underlying recipe. In my work to rebase pyiron_workflow onto flowrep, I've introduce a new Run object to return from runs. The critical data is the flowrep.NodeData instance-view results, but it also snags some of the stuff you wanted:

https://github.com/pyiron/pyiron_workflow/blob/f733e40c32aab0557135fffce0175894e4a135db/pyiron_workflow/_wfms/execution.py#L38-L61

I don't know if semantikon is the best place or not to discuss what we want, but IMO it will be the responsibility of the WfMS and not flowrep to provide it.

Think we could close this and continue in those other places?