Rewrite Sia schema compiler in Rust - #18
Open
Dirdmaster wants to merge 51 commits into
Open
Conversation
Workspace members: - sia-schema-core: library (logos, serde, heck) - sia-schema-cli: binary (clap) - sia-schema-lsp: binary (tower-lsp, tokio) - sia-schema-wasm: cdylib (wasm-bindgen)
Expected outputs for all 4 generators (TS, Go, Python, C++) and sample.sia input file for parser tests.
All errors carry byte-offset spans from the start for LSP integration. LineIndex converts offsets to line/column.
Definition, SchemaDefinition, PluginDefinition, FieldDefinition, MethodDefinition, Value. Type classification helpers for string, byte, number, and boolean types. Options are flattened in JSON output for compatibility with the TypeScript compiler.
20 token types, skips whitespace and comments. Post-lex pass
demotes keywords to identifiers inside schema {} and method {}
blocks so fields can be named 'schema', 'method', etc.
Includes 9 unit tests.
11 grammar rules matching the Sia schema language spec. Parses schemas, plugins, methods, fields, type options, default values, arrays, and optional markers. Recovers from errors by skipping to synchronization points. Includes 10 unit tests including sample.sia round-trip.
Target enum (TypeScript, Go, Python, Cpp) with extension parsing and project file auto-detection (walks up to 3 parent directories looking for tsconfig.json, go.sum, etc).
Generates interfaces, encode/decode functions, and full plugin RPC client classes. Pre-formatted output matches prettier defaults (80-col wrapping). Handles ascii encoding, optional defaults with ??, array callbacks, byteN fixed lengths, and custom schema type references.
Generates structs with json tags, Sia()/FromSia() receiver methods, and FromSiaBytes() helpers. Non-array fields are processed before array fields (matching TS compiler behavior). Default values use Go IIFE pattern. Custom schemas use EmbedSia/FromSia pattern.
Generates classes with __init__ (required params first, optional with defaults), encode(sia), and @classmethod decode(cls, sia). Lambda callbacks for arrays. Forward-ref strings for custom type annotations.
Generates separate .hpp (structs, forward decls, function signatures) and .cpp (encode/decode implementations). Uses shared_ptr for custom schema refs, _t suffix for integer types, AddArray8/ReadArray8 with lambdas for arrays.
lib.rs exports parse(), generate(), generate_cpp(). Generator tests compare output byte-for-byte against the existing TypeScript compiler's expected outputs for all 4 target languages.
Backwards-compatible with the TypeScript compiler's CLI: sia compile <file> [-s] [-o <file>] [-e ts|go|py|cpp] sia ir <file> [-s] [-o <file>] Auto-detects target from output extension or project files. C++ generates both .hpp and .cpp files.
tower-lsp based server with: 1. Diagnostics (parse errors, undefined types, duplicates) 2. Completion (built-in types, schema names, keywords) 3. Hover (type descriptions, schema field summaries) 4. Go to definition (custom type -> schema) 5. Document symbols (schemas, plugins, fields, methods) 6. Rename (schema name + all references) 7. Formatting (consistent 2-space indentation) Includes symbol table built from IR for cross-referencing.
Exports: compile(source) -> IR JSON, generate(ir_json, target) -> code string, compile_and_generate(source, target) -> code. Errors returned as JsValue with diagnostic messages.
Launches sia-lsp binary over stdio. Includes TextMate grammar for syntax highlighting, language configuration for brackets and comments, and configurable LSP binary path.
npm: @timeleap/sia-schema with optionalDependencies for platform-specific binaries. Finds and executes native binary. PyPI: timeleap-sia-schema with platform detection and subprocess delegation to native binary.
Cross-compiles for linux-x64, linux-arm64, darwin-x64, darwin-arm64, win32-x64. Builds WASM via wasm-pack. Publishes to GitHub Releases, crates.io, npm, and PyPI. Triggered on version tags (v*).
Removed: - src/ (TS compiler source: lexer, parser, visitor, generators, CLI) - tests/ (Jest test suite and expected outputs, now in crates/sia-schema-core/tests/fixtures/) - vscode-sia/ (old VS Code extension, replaced by packages/vscode/) - .vscode/ (Yarn SDK settings) - .editorconfig, .gitattributes (Yarn/JS-specific) - package.json, tsconfig.json, eslint.config.js, jest.config.cjs - yarn.lock, .yarnrc.yml - sample.json (superseded by sia ir command)
Opinionated formatter for .sia files with 7 rules: consistent 2-space indentation, blank line normalization, trailing whitespace removal, field alignment, option alignment, brace style enforcement, and comment preservation. - sia-schema-core: new formatter module with format() entry point - sia-schema-cli: new `sia fmt` command (--write, --check flags) - sia-schema-lsp: format_document now delegates to core formatter - 16 tests: 12 unit + 4 integration (snapshot, idempotency, round-trip)
Core linter module with 9 semantic checks: - duplicate-schema: duplicate schema names - duplicate-plugin: duplicate plugin aliases - duplicate-field: duplicate field names within a block - undefined-type: references to undefined schema types - unused-schema: schemas defined but never referenced - invalid-option: options not valid for the field type - invalid-default: default value type doesn't match field type - naming-convention: PascalCase schemas, camelCase fields/methods - empty-definition: schemas/plugins with no content LSP semantic_diagnostics now delegates to core linter — all 9 rules show as real-time diagnostics with rule codes in the editor. New `sia check` CLI command runs parse errors + linter on files. 16 unit tests for the linter, 55 total tests passing.
The `string` type without an explicit encoding option caused a panic in the TypeScript generator. Now defaults to utf8 (addString8/readString8), matching the Go, Python, and C++ generators.
Flatten the Sia schema language to only support schema definitions. Removes the Definition enum, plugin/method parsing, code generation, lint rules, formatter constructs, LSP completions, Tree-sitter grammar rules, and VS Code TextMate patterns for plugins and methods. Updates all test fixtures and expected outputs. ~980 lines removed across 32 files.
- Bump sia-schema-core, sia-schema-cli, sia-schema-lsp, sia-schema-wasm to 2.0.0 - Bump VS Code extension to 2.0.0 - Add pyproject.toml with maturin bindings=bin (like Ruff) - Add python/sia_schema/ wrapper for 'python -m sia_schema' support - Remove old packages/pip/ setuptools wrapper
- Replace tag-triggered release with workflow_dispatch (patch/minor/major) - bump-version job: updates all Cargo.toml, pyproject.toml, package.json, commits, tags, pushes - Add maturin-based PyPI build matrix (5 targets + sdist) - Add npm platform package generation in build matrix - Add 30s crates.io index wait between core and cli publish - Rewrite README: installation (cargo/npm/pip/binaries), schema syntax, CLI commands, editor support, supported languages, project structure
- Convert sia-schema-lsp from binary crate to library crate - Add `sia lsp` subcommand to sia-schema-cli that runs the LSP server - Remove the separate sia-lsp binary target - Now every installation method (cargo, npm, pip) gives users the LSP for free — editors run `sia lsp` instead of `sia-lsp`
- Resolve sia binary from npm platform packages or PATH (like Biome) - Launch LSP via `sia lsp` with stdio transport - Remove bundled binary resolution (bin/<platform>-<arch>/sia-lsp) - Update setting description
Only build and package the sia binary — the LSP is now a subcommand.
Dirdmaster
marked this pull request as ready for review
March 9, 2026 21:04
pouya-eghbali
approved these changes
Mar 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete rewrite of the Sia schema compiler from TypeScript to Rust, with full tooling.
Core compiler (
sia-schema-core)sia fmt)sia check)CLI (
sia-schema)sia compile/sia ir/sia fmt/sia check/sia lspLSP server (built into
sia lsp)WASM bindings (
sia-schema-wasm)VS Code extension (
packages/vscode/)siabinary from npm packages or PATHnpm distribution (
packages/npm/)@timeleap/sia-schemaPyPI distribution (
packages/python/)bindings = "bin"(like Ruff)timeleap-sia-schemaCI
push-checks.yml: check, test, clippy, fmtrelease.yml: manual workflow_dispatch, builds 5 targets, publishes to crates.io/npm/PyPI/GitHub ReleasesOther
Breaking changes
sia-schema-clitosia-schema(cargo install sia-schema)sia lspinstead of separatesia-lsp)