WebAssembly as a new target architecture

[vadimkantorov]

I've experimented with it successfully in my https://github.com/busytex/busytex. I've made several tex programs compile statically for Emscripten/WASM and also linked them into a single binary (busybox-like)

I've seen some people actually deploy this mode:

• https://texlyre.github.io
• https://afastudio.ch/latex-publisher/

So would be nice if my experiment somehow got upstream.

If there is anyone with enough resources / time to champion this from TexLive side, I can explain all that's going on in busytex Makefile

[kberry]

Thanks for writing, but I'm sorry, I can't even begin to conceive how this could be integrated into the standard TL build, or even the distribution. Maybe one of the other TL developers has an idea. -k

[vadimkantorov]

Actually, not so many fixes to Makefile for just building core TexLive programs for wasm (and a bit more fixes needed for busybox-style static build - and very few fixes/new ifdefs can improve the buildability as busybox-style).

I guess, the simplest could be if TexLive supported some testing on free GitHub Actions CI runners. Then at least wasm could be supported in the sense that the build is known to work / with some basic testing. So maybe actual bits release could be postponed.

In the simplest case - maybe my Makefile could be upstreamed directly as a separate Makefile script, but ideally, it could be integrated in TexLive makefiles as config option

[vadimkantorov]

Also, TexLyre is funded by public money:

Maybe they could sponsor or do some of this packaging / upstreaming work...

[gucci-on-fleek]

In principal, I don't see how this is any different from the other platform ports, so I think that the instructions at https://tug.org/texlive/build.html#newsys still apply. TeX Live is frequently built with Musl and Clang, so without having looked at any of your links yet, there hopefully shouldn't be too many changes needed.

As long as the patches are not onerous to maintain and are coherent with the rest of the build system, we should have no problem accepting it upstream (aka here).

I guess, the simplest could be if TexLive supported some testing on free GitHub Actions CI runners.

We already do this for the other platform ports, see the .github folder for details.

In the simplest case - maybe my Makefile could be upstreamed directly as a separate Makefile script

That's probably not an option, since it's a maintenance nightmare for us to have different scripts for different platforms.

but ideally, it could be integrated in TexLive makefiles as config option

Yes, I agree. WASM is pretty popular these days, and gazillions of projects use autotools, so there should presumably be some "standard" way to do this.

[vadimkantorov]

We already do this for the other platform ports, see the .github folder for details.

This is great. At least there won't be any extra dollar cost for adding some basic building smoke tests for WASM

so there should presumably be some "standard" way to do this.

There is the Emscripten clang/musl/libc standard toolchain. It almost works out of the box. It's the one I used for my Makefile in busytex. To make it work without my hacks, it would require some fixes to existing Makefile toolchain - mainly to pass-through some extra (per-dependency) compilation flags around. That's why currently in busytex I manually invoke make for dependencies first

To support busybox-like static linking (which incidentally is also very practical for wasm / running on the web), also some extra ifdefs/defines are required in concerned Tex programs - e.g. to rename main function (this already is somehow supported for tex programs in so-called "dll" build mode - for Windows), and also to hide similar-named symbols (or renaming them) to avoid linking clashes. Unfortunately, currently llvm-objcopy does not support yet renaming symbols for WASM (llvm/llvm-project#50623), so my hack is to pass around a bunch of renaming defines. A better solution would be to unify more of same-named symbols (like libjpeg / zlib - IIRC currently there are some copies of these in the codebase in different tex programs) in some common linkable library.

The problem is I don't have enough time (and knowledge of Makefile/autotools) to actually propose a proper patch series to upstream TexLive. So if there is anyone interested in doing this from TexLive side, I can explain in details the hacks/reasons-for-them I did in https://github.com/busytex/busytex/blob/main/Makefile (my script basically invokes existing Makefiles in specific ways)

[kberry]

If you can send a patch against the current sources, we can consider it.

[vadimkantorov]

The problem is that I don't understand autotools / TexLive build system enough to make a reasonable patch. And also I worked actively on busytex in 2020-2022, so many years have passed

At this point I can only try to explain the hacks to someone if there is someone from TexLive who'd like to champion this feature. If there is no such person, I guess busytex/wasm build script will just lie around until someone will take this on. At least it's in a workable state, as evidenced by TexLyre people who are build upon busytex approach.

[vadimkantorov]

Apart from other things, some of nuisance is related to the fact that I couldn't figure out if TexLive build scripts support cross-compilation scenario: e.g. some programs should still be built with host compiler as they are to be used in build time:

Just found my old email from 2021 to tlbuild about quirks I found back then (and back then it was fresh, now I don't remember all the quirks as well):

https://tug.org/pipermail/tlbuild/2021q1/004806.html

---

I guess the first actionable thing is to figure out how to specify emcc as main compiler and regular gcc as host compiler to build icupkg, pkgdata, apinames, ctangle, otangle, tangle, tangleboot, ctangleboot, tie, fixwrites, makecpool, splitup, web2c (as they are executed at build-time only). Currently, when just using emconfigure and emmake in my https://github.com/busytex/busytex/blob/main/Makefile, emcc got pushed down as the compiler everywhere. Maybe it can be done properly, so that host compiler gets propagated to freetype / icu / webc dependencies

And then I guess the path should be starting from https://github.com/busytex/busytex/blob/main/Makefile and then simplifying it step-by-step by reducing amount of dependency-specific compilation flags / introducing mechanisms for passing these down from the main configure script. Maybe the first task here should be figuring out how to compile libexpat / icu / freetype / fontconfig dependencies with vanilla main TexLive configure/Makefile script without per-dependency reconfiguration / extra make args. Some frequent extra flags are disabling building and running tests, disabling shared libraries. Expat probably should be the simplest to tackle (needing extra flags), then freetype (needing to properly have apinames built by host compiler). icu / fontconfig are a bit more intricate

[kberry]

Sorry, my backlog of bug reports and enhancements regarding the TL build is already much longer than I'll be able to finish in this lifetime. So I can't take up working on this, or the reports from your tlbuild message. If another volunteer comes forward, great, but I have no expectations. Meanwhile, I'm going to close this here since the TL github is mainly just for our CI runs, not the preferred place for issues. OTOH, I will open a new report at https://puszcza.gnu.org.ua, which is where we keep (and link to) the list of long-term open issues. https://puszcza.gnu.org.ua/bugs/index.php?687 The answer to cross-compiling is yes, the TL build system supports cross-compiling. Some people use it routinely to build Windows binaries. I'm not surprised by your encountering problems, but fundamentally it should be possible. https://tug.org/texinfohtml/tlbuild.html#Cross-compilation Thanks, Karl

[vadimkantorov]

That's fair. Thank you for recording this suggestion in the proper tracker. I can maybe add comments there with some concrete small fixes suggestions towards fixing the wasm (and cosmocc) build.

[vadimkantorov]

@kberry Does there exist a chat / IRC for this kind of building / modifying Makefile questions for TexLive? Because maybe I can over time (could take years, haha) propose small modifications towards solving the big task, but it would be much faster if I could discuss the pathway with someone before actually working on a patch.

Thank you!

[kberry]

I am not on any kind of chat platform. I have far too much incoming already :). I'm not aware of any chat for TL in general, but there might be something somewhere. Small/comprehensible changes would certainly be better than giant multi-purpose patches. Thanks.