Add "with ... do" statement for context management

[d-torrance]

After writing #4350 (comment), I got excited about this idea and figured I'd throw it together.

We introduce a new type of statement in Macaulay2, with x do y for context management, similar to Python. We can install entrance and exit methods for the type of x, and the exit methods are guaranteed to run even if y raises an error.

By default, if x is a file, then we close it on exit. If it's a mutex, then we lock on entrance and unlock on exit.

Silly example:

i1 : Foo = new Type of HashTable;

i2 : Foo.EnterMethod = x -> print "hi!";

i3 : Foo.ExitMethod = y -> print "bye!";

i4 : with new Foo do 1/0
hi!
stdio:4:17:(3):[1]: error: division by zero
bye!

Draft for now for comments. I also haven't written any documentation or tests yet.

AI Disclosure

Some brainstorming with Claude Chat, but the code is all mine.

[mikestillman]

Can we do

R = QQ[x,y,z]
x = a
f = with R do (
    x^3 + y^3 + z^3
    );

[d-torrance]

Yeah we should be able to by defining the right enter and exit methods for PolynomialRing. Maybe stuff the old values of the symbols somewhere (the ring object? no, that's not thread-safe) and call use in the enter method. Then assign the old values back in the exit method

[mikestillman]

I don't think stuffing it in the ring object is good, for parallel reasons. Unless we can give it a unique key, that the enter method and exit method can use.

[d-torrance]

I remembered that we have pushvar and popvar for exactly this sort of thing. Not thread-safe yet, but the following works great:

PolynomialRing.EnterMethod = R -> (
    scan(R.generatorSymbols, sym -> pushvar(sym, value sym));
    use R)

PolynomialRing.ExitMethod = R -> scan(R.generatorSymbols, popvar)

Then this gives:

i1 : R = QQ[x,y,z];

i2 : x = a

o2 = a

o2 : Symbol

i3 : with R do x^3 + y^3 + z^3

      3    3    3
o3 = x  + y  + z

o3 : R

i4 : x

o4 = a

o4 : Symbol

[d-torrance]

Just pushed a commit to make pushvar and popvar thread-safe.

Edit: Well, they're thread-safe, but I hope I didn't break the behavior. We have a different copy of the stack in each thread. What if we push in the parent thread and want to pop in another thread? Is that something we might want to do?

Edit 2: Never mind -- I guess they're still not thread-safe. Getting some weird behavior. I'll remove that commit

[mahrud]

I prefer this syntax for mutex management. e.g. with mutex do ... or alternatively repurpose lock and add lock mutex do ....

[d-torrance]

I went ahead and added documentation and tests, and also added the short-term fix for #3239 using the new proposed syntax.

In addition to the cases mentioned above, I also added with ... do ... support for inexact fields for adjusting defaultPrecision:

i1 : with RR_100 do numeric pi

o1 = 3.141592653589793238462643383279

o1 : RR (of precision 100)

I also add with support in the Python package.

[d-torrance]

Hmm, adding this syntax slows down the interpreter (by ~35% according to Claude). Going back to draft while I investigate

[d-torrance]

After some analysis of the code by Claude, I think I have a simple fix. The moral of the story is apparently not to add more local variables to evalraw...

[mahrud]

Can you say more? Would we get a speed up by removing some local variables from evalraw? What about other functions that run frequently? Would be good to pick a representative example as baseline for interpreter benchmarks.

[d-torrance]

That seemed to fix it -- the builds are taking the usual amount of time again!

Currently, the way that scc1 generates the c files, it lists every local variable at the top of a function, regardless of whether they're in some specific is case. So the register allocation gets more complicated.

I had Claude move all the various cases in evalraw into helper functions and am running an experiment on my fork right now to see if it makes a noticable difference.

[d-torrance]

... so moving all the evalraw cases into helper functions actually slowed things down by a little bit. the autotools-macos build on my fork took 77 min, whereas here it was only 73 min. 🤷

[mahrud]

I don't think you can rely on GitHub builds for accurate benchmarks, there's quite a bit of variation based on time of day.

[d-torrance]

After discussing this with Mike, I've updated the implementation a bit. The return value of the entrance method is passed as the second argument of to the exit method. This gives us a nice thread-safe way of restoring values, rather than trying to save them somewhere.

Edit: Simplified this a bit further, so the exit method only takes one argument, the return value of the enter method.

[mahrud]

I have to say I find the combination of the same syntax for mutexes, rings, fields, and files potentially very confusing.

[d-torrance]

I have to say I find the combination of the same syntax for mutexes, rings, fields, and files potentially very confusing.

That's a fair point, but is it substantially different from overloading methods on different types?

---

Something else I'm considering adding to this PR:

lock Function := f -> (
    mutex := new Mutex;
    x -> with mutex do f x)

Then users could write thread-safe functions without having to deal with mutexes directly, and the makeMonomialOrdering bug could be fixed by just surrounding it with lock(...).

Edit: Done

[mahrud]

This is what I was suggesting for an interpreter keyword with the mutex stored in the closure.

[d-torrance]

This is what I was suggesting for an interpreter keyword with the mutex stored in the closure.

Right -- I just wasn't sure how we could implement something like this specifically for function closures. Keywords just act on Code objects and we wouldn't know if we got a function closure or not until after we evaluated it.

Looking back, I suppose we could have a compiled function that takes a function closure (or any function object, for that matter), and constructs a compiled function closure with a mutex in it. That might work, I think, but it's a lot less general than this proposal.

[d-torrance]

Converting to a draft -- after I made the above comment, I decided to go ahead and code it up. See #4378. Both PR's provide a lock(Function), but the new one isn't also proposing sweeping additions to the Macaulay2 language. :)