ML systems & performance engineering · CS @ University of Toronto, 2024-2027

Interested in ML systems, inference runtimes, heterogeneous compute, performance engineering, and Apple Silicon.

MLX multi-node distributed. Eight merged fixes across both backends. In ring, a hang when a peer disconnects: a closed socket returns 0 from recv() without touching errno, so the failure was tested against a stale value and skipped. And a single transfer of 2 GiB or more failing outright, because send(2) rejects a length above INT_MAX while ring handed the syscall the whole remaining task size; the measured boundary is exactly INT_MAX.

In JACCL, an out-of-bounds read that reached ibv_reg_mr as a protection domain, an uninitialized GID handed to the queue pair whose failure surfaced much later and somewhere else, and an errno dropped at the one point it was known. Also a configured JACCL never reached because Open MPI returns a size-1 world outside mpirun and backend="any" accepts it, a hostfile whose rank order leaves every rank alone in a group of size one while passing every check that existed, and a version string compared numerically in the JACCL build gate, where an SDK reporting 26.10 reads as 26.1 and silently disables the backend.

A common thread: most of these fail without saying anything. A hang, a fallback, a group of size one that reports success.

Not all of it landed. sum_scatter and split were both declined on ring, and both gaps are still in the tree: RingGroup::sum_scatter throws, so nn.fully_shard cannot run on ring at all, and mlx.launch uses ring locally. CONTRIBUTIONS.md has everything, including what was declined and why.

mx.searchsorted — a new public MLX operation, added as a native primitive with hand-written CPU, Metal and CUDA kernels. The binary search runs through the same comparator sort uses, so ordering, including where NaNs land, stays consistent across all three backends.

mlx-lm server — a non-streaming chat completion came back with no content key at all when the model stopped while still inside a reasoning block, so a client reading message["content"] got a KeyError rather than an empty answer.

DuckDB hash join — a regression where dictionary emission changed a chain field from a pointer to an index, breaking the existing dead-end sentinel.

coreml-compute-placement — measuring where Core ML actually places operations on Apple Silicon, and what each choice costs. The Neural Engine is faster than the GPU on M4 Pro and 4.7x slower on M5 Max; the default ComputeUnit.ALL is sometimes the slowest option available. A second axis measures how much of the memory bus each chip's CPU can reach: 91% of peak on M4 Pro against 49% on M5 Max, so the same offload strategy is worth very different amounts on the two chips. Includes a cost-weighted ANE residency verifier with a CI gate.

Full list, including work in flight and issue investigations: CONTRIBUTIONS.md

Available for reproduction and benchmarking:

• M5 Max — 128 GB unified memory
• 2x M4 Pro Mac mini — 64 GB
• 2x M4 Pro Mac mini — 48 GB
• Multi-node Apple Silicon over Thunderbolt 5 with RDMA

Built up as chain, then ring, then full mesh, measuring at each stage: 4.23 -> 16.70 -> 21.48 GB/s all-reduce bus bandwidth, 5.1x end to end. Most of the distributed work above came out of running that rig rather than out of reading the source, which is why the bugs are mostly hangs, silent fallbacks and failures that surface somewhere other than where they happen.

I am particularly interested in issues that need recent Apple Silicon, large unified-memory configurations, or multiple Macs to reproduce. Several of the JACCL investigations in CONTRIBUTIONS.md had been open for months without a reproduction, for exactly that reason.

Apple Silicon ML runtimes · Metal / GPU compute · Core ML and ANE execution · quantized inference · multi-node inference · compiler and runtime behavior · performance and correctness bugs that only appear at scale

Earlier academic / course projects: @Erwinnnnnnn