Parsing python operators

[liamhuber]

As a corollary to #269 and with recipes necessary to resolve #258, it should be pretty straightforward to inject operator recipes into the graph when parsing a @workflow and ast finds an operation. I.e., we should be able to parse something like

import flowrep as fr

@fr.workflow
def wf(a, b):
    if a < b:
        x = "less"
    else:
        x = "more"
    return x

When building the condition node for the if flow controller, it the ast parser should figure out to add a standard less-than recipe with edges for a, b-->less-than inputs.

The reverse, where we transform these operator nodes back into in-lined operators when recompiling the python might not be as easy. Still, I currently believe these difficulties to be of a book-keeping nature, and not fundamental obstacles.

[XzzX]

Without verification, I am pretty sure every operator has a function equivalent in operator. I have not checked how the operators are represented in the AST, but I think there is a straight forward mapping. At least < -> lt. Getting lt back to < is mostly syntactic sugar but might be worthwhile.

[liamhuber]

Without verification, I am pretty sure every operator has a function equivalent in operator.

Yes, almost. It's contents of operator that I wrap here. The only unfortunate thing is that most of them are not nicely inspectable.

I have not checked how the operators are represented in the AST, but I think there is a straight forward mapping. At least < -> lt. Getting lt back to < is mostly syntactic sugar but might be worthwhile.

Yeah, we're on the same page here 👍

[XzzX]

The only unfortunate thing is that most of them are not nicely inspectable.

Yes, but that is not a big issue here. flowrep is already a dependency and the number of operators is fixed and limited. So we can provide wrappers that everyone automatically has access to.

The problem is more severe for dataclass packing/unpacking nodes.

[liamhuber]

The only unfortunate thing is that most of them are not nicely inspectable.

Yes, but that is not a big issue here. flowrep is already a dependency and the number of operators is fixed and limited. So we can provide wrappers that everyone automatically has access to.

Ah, I meant in the sense that it's a shame we can't simply do add_recipe = fr.parse_atomic(operator.add); add_data = fr.tools.recipe2data(add_recipe) because somewhere along the line we do an important inspect or typing.get_type_hints that borks on most of the operator methods.

The problem is more severe for dataclass packing/unpacking nodes.

In #274 I get extend @fr.atomic to work on class definitions, and #275 powers it up nicely so @fr.dataclass gives you exactly a DecoratedDataclass.flowrep_recipe for the packing, type hints and everything. I'm sure we can do unpacking too, so this will be fine.

I'm still confident we can get operator injection working, i.e. that

@fr.workflow
def wf(x):
   y = x + 1
   return y

parses and winds up with a constant node and an add node, where the constant and input feed add, and add feeds output.

However, after going through #281, I'm a little discouraged about how much work it will wind up being. The basic problem with injection is that it makes it much harder to maintain clear scope of available symbols; there's just a lot more book-keeping headache.

[XzzX]

The only unfortunate thing is that most of them are not nicely inspectable.

Yes, but that is not a big issue here. flowrep is already a dependency and the number of operators is fixed and limited. So we can provide wrappers that everyone automatically has access to.

Ah, I meant in the sense that it's a shame we can't simply do add_recipe = fr.parse_atomic(operator.add); add_data = fr.tools.recipe2data(add_recipe) because somewhere along the line we do an important inspect or typing.get_type_hints that borks on most of the operator methods.

Why do we need the source code? Is the signature enough? It should be possible to get the signature. At least I had some parser for the operator module at some point for sim-atlas.

[liamhuber]

Why do we need the source code? Is the signature enough? It should be possible to get the signature. At least I had some parser for the operator module at some point for sim-atlas.

We have a few layers:

1. Recipe: inspect.signature for inputs
2. Recipe: inspect.getsource for automatically parsing outputs
3. Data object: typing.get_type_hints for turning annotations into real objects

Most of the builtins fail for (2), some fail for (1).

We could add a "trust me bro" flag to skip step (2) as long as output_labels have been explicitly provided. I'm open to that, but I also don't see a immediate need for it as I'm actually now quite happy with the stdlib @atomic-decorating functions that call operator (and other bodies). If we do add it later, it's something that would default-false in the recipe, so it will be backwards compatible.

[samwaseda]

Sorry for missing the discussion so far here. I just wanted to jump in because I really strongly oppose the idea of injection (as you might remember me saying this a few times...). I think we really need to talk about what's workflow and what constitutes a node. Here I would like to throw in a list of requirements, out of which a node must fulfill at least one:

• Ontologically meaningful step
• Computationally intense
• Data must be stored
• Intuitively clearer for GUI

The reason behind this is simply because a workflow environment has the following components: KG (+node store), executor, storage and GUI. And if a node does not benefit from any of these components, then people come back to the common question "what's the point of writing a workflow in the first place?", i.e. it actually makes the whole thing only more complex. And on top of this, we'll get the obvious problem that it becomes more difficult to maintain the code.

So, in the end the impression that it makes it easier is illusional; in reality I think it makes the workflow only more complex and it makes the code basis more difficult to maintain.

[liamhuber]

Sorry for missing the discussion so far here. I just wanted to jump in because I really strongly oppose the idea of injection (as you might remember me saying this a few times...).

So, in the end the impression that it makes it easier is illusional; in reality I think it makes the workflow only more complex and it makes the code basis more difficult to maintain.

I first want to clarify and reiterate, that we're talking exclusively about how the @workflow-python --> recipe parser works. The final recipe is whatever static recipe it is, whether it was written as JSON in a text editor or parsed from decorated python.

I find that the way we've made clean divisions between the recipe stage and data stage, and how a WfMS is not the "living" embodiment of both has alleviated many of my own concerns about "injection". At this point, we're not really "injecting" new nodes into some living and otherwise already-running workflow, we're just defining the map between @workflow-decorated python code and nodes. We've already agreed that calling a callable x = foo(...) "injects" a foo-node, and I find the step from there to z = x + y "injects' an add-node to be much smaller and less dramatic than vintage pyiron_workflow "injection".

Having written a bunch of workflow-decorated code in a couple different systems, I can only say that my lived experience is that I find it much easier and more pleasant to be able to use basic python operators. This is not a hill I will absolutely die on, but the QoL improvement on the author side is not an illusion at all, and I would eagerly search for a maintenance-light way to provide it.

---

I think we really need to talk about what's workflow and what constitutes a node. Here I would like to throw in a list of requirements, out of which a node must fulfill at least one:

- Ontologically meaningful step
- Computationally intense
- Data must be stored
- Intuitively clearer for GUI

The reason behind this is simply because a workflow environment has the following components: KG (+node store), executor, storage and GUI. And if a node does not benefit from any of these components, then people come back to the common question "what's the point of writing a workflow in the first place?", i.e. it actually makes the whole thing only more complex. And on top of this, we'll get the obvious problem that it becomes more difficult to maintain the code.

This list is great, I really like it as guiding principles to think about when designing a workflow or building workflow/atomic nodes.

I don't think we can call these "requirements". We intentionally require a minimal amount, all encoded in the pydantic validators for the recipes. I think its much stronger to frame things in terms of what we provide rather than what we require. In the context of this issue, the main thing we provide is to use your node in an "atomic" fashion where it is a black box around a python function, or a "workflow" fashion, where the internal structure of the operations remains retrospectively available down to the atomic level. From this "provide" perspective, I would then interpret

- Ontologically meaningful step

If you ontologically annotate your underlying functions, we will provide the ability to ontologically validate/inspect your workflow graph. This holds for both workflow- and atomic-level nodes.

- Computationally intense

If you use our WfMS, we will let you apply computational resources to any node within it. In the context of atomic/workflow, you should think about the granularity with which you want to be able to apply these resources and apply workflow nodes until the resulting atomic nodes exist at your desired granularity.

- Data must be stored

Similar to computational resources, you should make explicit workflow nodes to whatever granularity you want because steps inside atomic nodes will not be retrospectively inspectable.

- Intuitively clearer for GUI

Similar to data storage, if you want a GUI representation of your steps, you'll need to apply a workflow node. If GUI users don't need to see what's happening inside (or can just read the code for what's happening inside), and atomic node is fine.

---

What we don't do, at any point, is explicitly police what people put in an atomic node vs what they put in a workflow node. In particular, we place no lower limit on how simple the python can be inside an atomic node. We ought not, and indeed we would exceptionally hard pressed, to impose constraints so that users can't write

@flowrep.atomic
def increment(x):
    return x + 1

under the argument that it is "too simple" and must be combined together with other too-small atomic nodes to make one bigger, more "satisfying" atomic node.

I find it useful to view this through two lenses. The first is factoring, which I find has rough analogies ("") to atomic/workflow nodes. When should a monolithic function ( a big atomic node) be decomposed into sub-function calls (~ multiple atomic nodes strung together in a workflow)? What is too small to factor out into a function? Some people advocate pulling out one-liners if it means getting a nice, readable name for things! When to introduce a class to some function monolith, for state management (~ workflow node for retrospective sub-step data inspection)?

Neither python, nor C++, nor Rust, nor any imperative programming language I'm aware of in widespread use today puts any constraint on users about how short or long their functions can be. In the same way, I find it not at all our place to try to enforce to users "oh, that is too small to be an atomic node, you must group several of those together" nor "oh, that is too large to be an atomic node, you must decompose this into a workflow made of multiple atomic steps". Because of this, I really push back strongly on the language of "requirements" above.

The second lense I find useful is to consider use-case. Sometimes, users will be writing production-quality code for long-term posterity. Other times, they're just going to be putzing around. In the former case, we want to give them all the tools they need to write something highly specific like

@flowrep.atomic("foobar")
def annotated_addition(
    foo: semantikon.u(Foo, EX.Some), 
    bar: semantikon.u(Bar, EX.Thing)
) -> u(FooBar, EX.SomeVeryParticularMeaning):
    return foo + bar

@flowrep.workflow
def wf(...):
    ...
    fb = annotated_addition(f, b)
    ...

But in the latter case, it's going to be perfectly sufficient to write

@flowrep.workflow
def wf(...):
    ...
    fb = f + b
    ...

And use some generic adding node.

Now, I'm of two minds on this -- where I'm at personally is that I typical start with the typing, and a significant part of me wishes untyped languages didn't even exist -- but I lean towards pragmatically saying that there is a time a place for each of these approaches. I think that many users much of the time will not be operating in a fully-ontologically-annotated world, but will still derive benefit from the way a graph-and-node based WfMS can support resource allocation, data storage, and GUI interaction. So, while being sensitive to how much maintenance overhead it adds to the parser, I don't see a fundamental imperative to disallow f + b as a shortcut in the parser for

@flowrep.workflow
def wf(...):
    ...
    fb = flowrep.std.add(f, b)
    ...

(or more extreme, to force them to write they're own def add(...) function every project they want to add things together as some little step in a workflow)

---

Arguments I find convincing:

• We may wish to break apart parsing and the actual recipe structure into an ideologically pure and slowly-changing data structure package, and an ideologically pragmatic and potentially quickly-changing parser package
• We should be sensitive to how much maintenance headache is added when we add parsing sugar
• We should think about what practices we encourage or discourage when introducing syntactic conveniences

Arguments I do not find convincing:

• Anything to do with imposing requirements on what users put inside an atomic node
• Claims that the injection syntax is not convenient

Not that I'm claiming you're explicitly making each of these. This is just to summarize what I got out of your comment, and extract the aspects I find to be convincing arguments to stop or go carefully when thinking about adding parser sugar for operators.

[samwaseda]

ok let's see if we can find a compromise. Let's take a look at this one:

def my_workflow(a, b, c, d, e, f):
    result = a * (b + c) / (d - e) ** f
    return result

Is this internally translated to:

def black_box_injection(a, b, c, d, e, f):
    return a * (b + c) / (d - e) ** f

def my_workflow(a, b, c, d, e, f):
    result = black_box_injection(a, b, c, d, e, f)
    return result

or

def mul(x, y):
    return x * y

def add(x, y):
    return x + y

def div(x, y):
    return x / y

def sub(x, y):
    return x - y

def exp(x, y):
    return x ** y

def my_workflow(a, b, c, d, e, f):
    added = add(b, c)
    multiplied = mul(a, added)
    subtracted = sub(d, e)
    divided = div(multiplied, subtracted)
    exponential = exp(divided, f)
    return exponential

I guess I can live with black_box_injection, but I really don't want individual injections, because it would be a nightmare to explain what is a node in this workflow to someone learning how to use it for the first time.

[liamhuber]

The implementation I had in mind is closer to the latter, where each operation gets its own step. You can do this (and re-render it) already in #277 with item and attribute access:

import flowrep as fr

@fr.workflow
def access(complex, key1, key2):
    subdata = complex[key1].attr1[key2].attr2
    return subdata

print(
    fr.tools.flowrep2python(
        access.flowrep_recipe.model_copy(
            update={"reference": None}
        )
    ).source
)

gives

from __future__ import annotations

import flowrep
import typing

@flowrep.workflow("subdata")
def compiled_from_workflow_recipe(complex, key1, key2):
    getitem = complex[key1]
    get_attr = getitem.attr1
    getitem_0 = get_attr[key2]
    get_attr_0 = getitem_0.attr2
    return get_attr_0

Taking the same direct approach naively, if we were to parse and then re-compile my_workflow, it would give something like your final example (note that your BEDMAS is off):

from __future__ import annotations

import flowrep
import typing

@flowrep.workflow("subdata")
def compiled_from_workflow_recipe(a, b, c, d, e, f):
    add = flowrep.std.add(a, b)
    subtract = flowrep.std.sub(d, e)
    pow = flowrep.std.pow(subtract, f)
    truediv = flowrep.std.truediv(add, pow)
    mul = flowrep.std.mul(a, truediv)
    return mul

There is none of the def add... because the parsed recipe uses extant functions to represent the operations. This lies at the crux of the problem, since we want

@fr.workflow
def my_workflow(a, b, c, d, e, f):
    result = a * (b + c) / (d - e) ** f
    return result

to directly make available a plain JSONable my_workflow.flowrep_recipe: WorkflowRecipe. The atomic nodes of that recipe need to be able to reference extant python functions -- we aren't going to compile out some secret def black_box_injection or the recipe becomes completely unportable. The work to get result = ... needs to be immediately representable in the recipe structure. This is one of the motivations for the existence of a standard library of nodes to interface with python's builtin operator module. Now, with a little more massaging discussed in #298, I think I could coerce the injection to make recipes directly on the operator functions (although the library is still nice for other reasons and I like having it as an interface). But, critically, this still winds up a series of atomic recipes, because there exists no such real, importable black_box_injection function definition for a clumped-together recipe to reference!

So unfortunately, I think automatically grouping arbitrary operation combinations like this into a single atomic node is simply not possible. The closest we could do here is to create a new WorkflowRecipe and nest the individual operations as children of that whenever multiple operations are detected in a single line. This would keep the top level workflow cleaner, but I suspect it doesn't satisfy you because under the hood there's still all the small atomic nodes each representing only a single operation. My understanding is that it is somehow this smallness of these atomic nodes that is at the heart of your objections.

---

because it would be a nightmare to explain what is a node in this workflow to someone learning how to use it for the first time.

I don't understand this argument at all. IMO there is nothing philosophical going on here. To me this reads concern reads identically to "I don't know how I could explain to someone what a function is if they ever saw bunch of short functions in a row". Functions can be short or functions can be long -- an atomic node is just a wrapper around a python function, so of course they can be either short or long too.

---

@fr.workflow
def my_workflow(a, b, c, d, e, f):
    result = a * (b + c) / (d - e) ** f
    return result

Is easy to write and easy to read. With straightforward parsing of operators each into an atomic node, we get an unambiguous but verbose recipe back out.

@fr.atomic
def black_box_injection(a, b, c, d, e, f):
    return a * (b + c) / (d - e) ** f

@fr.workflow
def my_workflow(a, b, c, d, e, f):
    result = black_box_injection(a, b, c, d, e, f)
    return result

Is more work to write, but equally clear and gives us a much more compact recipe. This example is at one extreme end of the spectrum. I don't have any objection to offering operator parsing to people here, but it seems like a great case for writing a custom atomic node to handle it all at once.

Consider the other end of the spectrum,

@fr.workflow
def my_workflow(a, b, c):
    complex1 = some_node(a, b)
    complex2 = some_other_node(b, c)
    together = complex1 + complex2
    return together

vs

@fr.atomic
def my_add(a, b):
    return a + b

@fr.workflow
def my_workflow(a, b, c):
    complex1 = some_node(a, b)
    complex2 = some_other_node(b, c)
    together = my_add(complex1, complex2)
    return together

In this case, the resulting recipes have identical verbosity, but the shortcut is just much more pleasant to write. My argument is that in many cases, the operator parsing is just much nicer to both read and write at zero cost, so I think it would be nice to have it available.

[liamhuber]

Ok, I'm lying there a little bit. We could, at parse time, dynamically generate a new function object(s) to represent the black box(es), and assign it to a new (dynamically named) attribute on the decorated function. We do something similar but simpler with dataclasses, copying a static function to update its type hints, and assigning it at a known attribute name on the dataclass. I'm wary of descending so deep into dark magic to pull a similar trick with an arbitrary number of arbitrarily structured black box functions.