Support for build/install as dynamic native modules

[jonnor]

Hi. This is a fascinating project, great job!

The announcement video mentioned that right now one needs to rebuild entire MicroPython to use this module, which can be a bit tedious/tricky.

As you may be aware, MicroPython has support for building modules that can be packed into .mpy files and installed at runtime using mip install.
https://docs.micropython.org/en/latest/develop/natmod.html

This would make it considerably easier for people to get started. The support has improved a lot over the last year, though it still has some rough edges. More projects using this functionality would help further improve this unique and very nice feature of MicroPython.

To be applicable in practice it would probably be necessary to split into multiple independent modules, so that people can install and load only what they need.
It is possible to support both styles of building with the same code. I do this in emlearn-micropython, a Machine Learning package for MicroPython. Example: https://github.com/emlearn/emlearn-micropython/blob/master/src/emlearn_iir/iir_filter.c

[sfe-SparkFro]

Thank you! A lot of work has gone into this, and we're excited to see the excitement for it!

Yes, we did consider creating this as a native module that compiles to a .mpy file early on in this project. However there were some technical challenges and unknowns that made us decide against it. We admittedly didn't look into it too hard, and it seems like you know a lot more about it than we do, so forgive me if I have incorrect information or assumptions!

One of the big challenges is memory management with MicroPython's garbage collector:

1. OpenCV creates global variables on the heap (for some crazy reason unknown to me...), which must be initialized at boot before the garbage collector has even been initialized. For these, we have to force MicroPython to make space for the C heap (it's normally zero), and wrap malloc() to use __real_malloc() if the GC hasn't been initialized, or the GC allocation functions if it has so all the Mat objects are allocated in the GC memory pool (which has access to the PSRAM).

   • As I write this, I realize that this may not be an issue with compiling to a .mpy file, since it would get loaded after the GC has been initialized. Still, it's a thing that gave us a lot of headache with this project.

2. OpenCV also creates some static variables with "lazy" initializers, so they get initialized one time the first time they're needed, and then get reused on successive function calls. If those are allocated in the GC memory pool, all works fine until you do a soft reboot, at which point those static variables get overwritten with garbage data and causes bizarre behavior (see my debugging adventures here and here). Our solution is to create our own global dummy variable that secretly calls those lazy initializers to force them to allocate on the C heap using our malloc wrapper.

There's also some other practical issues with compiling this to a .mpy file.

1. For example, you need separate .mpy files for each architecture, and users don't always know which architecture is theirs, or what the term "architecture" even means. We find it's a lot easier if they can download something for their specific development board, so we're pre-compiling firmware for SparkFun products under the Red Vision branding (which includes this OpenCV port).

2. Another issue is that I'm not sure how MicroPython would handle a 3+MB .mpy file. Since it's not frozen into the firmware, it would have to be loaded and executed from RAM. How long would it take to load? Is execution time degraded? Would it even work? Truthfully, we don't know.

I'm also not clear on how we would make this work with ulab NumPy. Does it need to be included in the OpenCV .mpy file? Or does the user need to install it separately? Can we even have a dependency like this?

None of that's to say it's impossible. We just saw all the challenges above and decided it wasn't worth our development time to figure it all out. We wanted to get this project out the door, and we were reasonably confident that it would be either not work or be a worse solution to compile to a .mpy file. But we'd love to be proven wrong! Like mentioned at the end of the video, we'd be happy to work with the community to create a more official MicroPython port of OpenCV, and if it goes with the solution of compiled .mpy files, that's great! However it's probably not something that we will work on, at least for the time being.

I'll leave this issue open until it's determined for certain whether compiling to .mpy files is an objectively good or bad idea for this project.

[jonnor]

Hi and thank you for the detailed response. It is very useful to understand what painpoint exists with dynamic native module support at the moment, to see if they can be addressed (now or in the future). It seems that there are several aspects that indeed make it quite difficult to have this as a dynamic native module. As the code is a port of a desktop-class C++ library, this is probably not that surprising.

Some comments on the particular cases follows.

1. OpenCV creates global variables on the heap (for some crazy reason unknown to me...), which must be initialized at boot before the garbage collector has even been initialized. For these, we have to force MicroPython to [make space for the C heap](https://github.com/sparkfun/micropython-opencv/blob/e6c40d6b590061a0548fe2021cfb61f4fb897391/micropython_opencv.cmake#L34-L37) (it's normally zero), and [wrap `malloc()`](https://github.com/sparkfun/micropython-opencv/blob/e6c40d6b590061a0548fe2021cfb61f4fb897391/src/alloc.c#L37-L47) to use `__real_malloc()` if the GC hasn't been initialized, or the GC allocation functions if it has so all the `Mat` objects are allocated in the GC memory pool (which has access to the PSRAM).
   
   * As I write this, I realize that this may not be an issue with compiling to a .mpy file, since it would get loaded after the GC has been initialized. Still, it's a thing that gave us a lot of headache with this project.

There is no way to have a custom build parameters with a .mpy file. So if the C heap must be extended, then this might be a blocker to having it work out-of-the-box on standard MicroPython builds. It could perhaps be possible to defer this to use the MicroPython managed heap instead, though at the moment that will give slight degradation in GC performance because more memory must be scanned. There has been a proposal for a mechanism to mitigate this.

2. OpenCV also creates some static variables with "lazy" initializers, so they get initialized one time the first time they're needed, and then get reused on successive function calls. If those are allocated in the GC memory pool, all works fine until you do a soft reboot, at which point those static variables get overwritten with garbage data and causes bizarre behavior (see my debugging adventures [here](https://github.com/sparkfun/micropython-opencv/issues/13) and [here](https://github.com/sparkfun/micropython-opencv/issues/17)). Our solution is to create our own [global dummy variable that secretly calls those lazy initializers](https://github.com/sparkfun/micropython-opencv/blob/e6c40d6b590061a0548fe2021cfb61f4fb897391/src/core.cpp#L27-L74) to force them to allocate on the C heap using our malloc wrapper.

Having a global / singleton class to manage this seems like a good approach. That should be the same with a dynamic native module, I think.

There's also some other practical issues with compiling this to a .mpy file.

1. For example, you need [separate .mpy files for each architecture](https://docs.micropython.org/en/latest/develop/natmod.html#supported-features-and-limitations), and users don't always know which architecture is theirs, or what the term "architecture" even means. We find it's a lot easier if they can download something for their specific development board, so we're pre-compiling firmware for SparkFun products under the Red Vision branding (which includes this OpenCV port).

Yep. This is something we should fix within the mip package manager. I think we should make it such that users can just copy/paste a single installation command that will work regardless of platform. I will try to create an RFC in MicroPython for such a feature in the near future.
With the recently added "romfs" feature, it is supposed to get easier to bundle a set of files (including .mpy).

2. Another issue is that I'm not sure how MicroPython would handle a 3+MB .mpy file. Since it's not frozen into the firmware, it would have to be loaded and executed from RAM. How long would it take to load? Is execution time degraded? Would it even work? Truthfully, we don't know.

There is no execute-in-place support (XIP) for natmods right now (it is theoretically possible to build that on top of romfs). Without XIP, a 3 MB .mpy file would be very problematic. It might not work at all on some boards - on ESP32 for example, I think that only internal ram (and not PSRAM) can be executed from - and that would be too small.
But I think that for a good natmod experience, one would want to split the library into a set of modules that can be installed independently. Targeting something like a 10-100 kB each. This would also enable users to just have the things they actually use in the application, instead of the entire "kitchen sink".

I'm also not clear on how we would make this work with ulab NumPy. Does it need to be included in the OpenCV .mpy file? Or does the user need to install it separately? Can we even have a dependency like this?

Do you use ulab from C code? If one uses it only from Python, then the users can install it separately, and it would work fine together. Having ulab be installable as a dynamic native module is something I am looking into - it would be a serious exercise, but probably more realistic first target than OpenCV.

[sfe-SparkFro]

There is no way to have a custom build parameters with a .mpy file. So if the C heap must be extended, then this might be a blocker to having it work out-of-the-box on standard MicroPython builds. It could perhaps be possible to defer this to use the MicroPython managed heap instead, though at the moment that will give slight degradation in GC performance because more memory must be scanned. There has been a proposal for a mechanism to mitigate this.

Having a global / singleton class to manage this seems like a good approach. That should be the same with a dynamic native module, I think.

Thinking about this some more, as long as these global and static variables can be safely stored somewhere that won't get wiped by garbage collection, then I think it would be feasible to not need the C heap. I think the C heap is really only required if OpenCV is built into the firmware.

Yep. This is something we should fix within the mip package manager. I think we should make it such that users can just copy/paste a single installation command that will work regardless of platform. I will try to create an RFC in MicroPython for such a feature in the near future.

🎉

With the recently added "romfs" feature, it is supposed to get easier to bundle a set of files (including .mpy).

There is no execute-in-place support (XIP) for natmods right now (it is theoretically possible to build that on top of romfs). Without XIP, a 3 MB .mpy file would be very problematic. It might not work at all on some boards - on ESP32 for example, I think that only internal ram (and not PSRAM) can be executed from - and that would be too small.

Oh, I hadn't considered ROMFS! That feature got added after we started this project, so it didn't get a proper consideration from us.

From the release notes, sounds like it does support XIP, no? Or is it only bytecode and not natmods?

which supports executing bytecode in-place, that is, without the need to copy it to RAM

Regardless, this would have the added challenge that ROMFS is not enabled by default for many boards, so those users would need to build new firmware anyways. Even for boards where it is enabled by default, some users really struggle with command-line tools like mpremote (think about students on school-provided Chromebooks), meaning it would be a real challenge for them to load the .mpy file to ROMFS themselves. So IMO neither is a perfect solution from a user-experience perspective, which is why we're making pre-compiled firmware available for our boards that has everything pre-loaded.

But I think that for a good natmod experience, one would want to split the library into a set of modules that can be installed independently. Targeting something like a 10-100 kB each. This would also enable users to just have the things they actually use in the application, instead of the entire "kitchen sink".

We're really trying to replicate the standard Python OpenCV experience, which is very kitchen sink. We anticipate that this project may grow, in which case splitting it up may become required. But until we find a true need to split it up, keeping it all together works well for our needs while maintaining the most similar experience to standard Python OpenCV as we can feasibly achieve.

Do you use ulab from C code? If one uses it only from Python, then the users can install it separately, and it would work fine together. Having ulab be installable as a dynamic native module is something I am looking into - it would be a serious exercise, but probably more realistic first target than OpenCV.

Yes, we use ulab from C code. It suppose might be possible to instead call the mp_call_method functions to remove that dependency, but that's probably more development effort than we're interested in putting in at this time.