Repository navigation
Parallelize lg native test compilation and reuse saved core state - #4
Merged
Merged
Conversation
- Replace the single 19.5MB generated test module with a base chunk (literal + lui + app + lg-test framework, chained off the core library's saved state) plus 8 independent test-namespace chunks, so ocamlopt compiles them in parallel instead of one giant unit. - Emit a saved state from the core library rule (--emit-state) so the test base chunk can extend it without recompiling the core directory. - Compile the chunk modules in an unwrapped native library and force them into the link with -linkall so test-registration initializers survive dead-stripping. - Replace the broken (:include lg_native_cmi_deps.sexp) dep lists with an lg-native-cmi-deps alias that actually builds the vendored duniverse cmis, so lg does not fall back to same-named opam cmis. - Bump the lg/lg-test pin to the revision carrying saved-state provenance keys, interface dedup, and chunk --emit-state support. Cold 'rm -rf _build' dune build @shared/native/runtest: 5m33s -> ~40s on this machine, and ~13s with a warm dune shared cache. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Author
|
I'll fix CI failures and address comments. I'll skip comments containing "(aside)".
|
Runs 'dune build @shared/native/runtest' on Ubuntu with OCaml 5.5. Vendored deps are restored from the committed opam-monorepo lockfile (public dirs) plus clones of the private pin-depends repos (lg, lui, signal-lg, datascript-ocaml, mldoc, persistent-sorted-set-ocaml) at their pinned commits. Cached layers: - setup-ocaml caches the opam switch (compiled compiler + deps); - duniverse is cached on the lockfile/opam hash; - ~/.cache/dune (dune shared cache, enabled via ~/.config/dune/config) and .lg-cache are cached per-ref with a main fallback, keeping cold 'rm -rf _build' runs in the ~13s range after the first warm run. Requires the repo secret LOGSEQ_GITHUB_PAT (read access to the private logseq OCaml repos). Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dune build @shared/native/runtestcold (rm -rf _build) went from 5m33s to ~40s on this box (~13s with the dune shared cache enabled), all 707 tests passing. Two structural changes on top of the lg fixes in tiensonqin/lg#1:.mlthatocamloptcompiled serially (~44s). Now a base chunk compilesliteral + lui + app + lg-test frameworkoff the core library's saved state (emittinginclude Logseq_chat_lg_core_native), and the 41 test namespaces compile as 8 independent chunks via--compile-files-chunk-from+--emit-state, soocamloptruns them in parallel (~3s each).--emit-state logseq_chat_lg_core_native.state), so the test base extends the compiled core instead of recompiling it.logseq_chat_lui_test_partslibrary — dune only emits compile rules for generated modules listed on a library — and the runner executable uses-linkallso test-registration initializers survive dead-stripping.(:include lg_native_cmi_deps.sexp)dep lists (whose targets were never built) with anlg-native-cmi-depsalias that actually builds the vendored duniverse cmis before lg runs — otherwise lg's typecheck falls back to same-named opam cmis and fails withinconsistent assumptions.lg/lg-testpin-depends to the revision with saved-state provenance keys + chunk--emit-state(Make the compile prefix cache survive cold builds; parallelize the clojure-suite smoke tiensonqin/lg#1). If that PR is rebased on merge, this pin must be updated to the merged SHA.Verification
dune build @shared/native/runtest: 707/707 tests pass; coldrm -rf _buildwall time ~39.5s, warm no-op ~0.9s, ~12.7s with(cache enabled)in~/.config/dune/config(11.6s of which is the test run itself — a user rule, not cached).Note for environments
The ~13s path requires the dune shared cache. It is off by default; enable per machine via
~/.config/dune/config:Link to Devin session: https://app.devin.ai/sessions/ef32b286005d4d049cc1affebba0aab3
Open in Devin Desktop: https://app.devin.ai/desktop/session/ef32b286005d4d049cc1affebba0aab3?variant=devin
Requested by: @tiensonqin