Skip to content

README.md

Latest commit

 

History

History
381 lines (301 loc) · 21.1 KB

README.md

File metadata and controls

381 lines (301 loc) · 21.1 KB
Raw

Nixstasis logo

Nixstasis

State synchronization

Nixstasis is an IoT monitoring and remote access platform. The current supported server deployment path for this feature is Docker Compose via deploy/compose.

The client and server have been rewritten from their original Bash and Python/Reflex implementations into Go and Elixir/Phoenix, respectively.

Key Components

  • Client (Go): Lightweight agent that registers devices, polls telemetry plugins, and manages FRP tunnels.
  • Server (Elixir/Phoenix + LiveView): API and web UI for device monitoring, approvals, alerts, and reporting.
  • Packaging and Release: Releases publish OCI images for the server, Caddy, and FRPS to GHCR, and GoReleaser client packages (.deb, .rpm, .run, and archives) to GitHub Releases.
  • Infrastructure: FRP, Caddy, and AuthCrunch integrate to provide secure, on-demand remote access.

Repository Structure

  • packages/client: Go-based client agent.
  • packages/server: Phoenix server application.
  • deploy/compose: Supported server deployment assets for this migration.
  • packages/caddy: Caddy image and legacy packaging assets.
  • packages/frp: FRPS image build assets and shared FRP binary acquisition scripts used by server/client packaging flows.
  • website/docs/features: Documentation-first feature designs and delivery records.

Development workflow

Work is tracked in Beads, and feature designs and delivery records live under website/docs/features. Install the locked developer tools and inspect ready work with:

mise install --locked
bd ready --json

Build the documentation with mise run docs:build and the complete quality contract with mise run check. See AGENTS.md for contribution rules and website/docs/development/releasing.md for releases.

Package READMEs

Deployment status

  • Supported server deployment path: deploy/compose
  • Supported client release path: GoReleaser from packages/client
  • Abandoned server package deployment assets are not part of the supported release path for this migration.

Client release status

Build client snapshot artifacts from packages/client with the tracked production version pins:

grep -E '^(FRP_VERSION|CADDY_VERSION|POSTGRES_VERSION)=' prod.env
goreleaser release --snapshot --clean
./build/bin/verify_artifacts.sh

The supported client artifacts install nixstasis to /usr/bin/nixstasis and the bundled frpc runtime to /usr/libexec/nixstasis/frpc.

E2E Tests (Client ↔ Server)

This repo includes a client-driven E2E harness:

  • The client executes journeys and emits structured JSON logs.
  • The server validates contracts, seeds baseline data, stores runs/results/log refs, and renders reports in LiveDashboard.

Quick Start

Prerequisites:

  • Server is running from packages/server (for example: mix phx.server).
  • Server migrations are up to date.
  • You run the harness from packages/client.

Set the dedicated local runner credential (the tracked development server uses the same local-only value):

export NIXSTASIS_E2E_RUNNER_ID=local-runner
export NIXSTASIS_E2E_RUNNER_TOKEN=dev-e2e-runner-token-0123456789abcdef0123456789abcdef

Run full suite:

cd packages/client
scripts/e2e/run --suite full --env local --trigger manual --protocol-version 1

Run one journey:

cd packages/client
scripts/e2e/run --journey auth --env local --trigger manual --protocol-version 1

Run with idempotent create semantics (1-hour window scoped by runner and environment):

cd packages/client
scripts/e2e/run --suite full --env local --trigger manual --protocol-version 1 --idempotency-key local-full-001

Inspect results:

  • API: /e2e/runs, /e2e/runs/:id, /e2e/runs/:id/results, /e2e/runs/:id/results/:journey_id/log
  • Suite catalog API: /e2e/suites (server source of truth for available suites)
  • LiveDashboard: /dev/dashboard/e2e

Contract Rules (Server-Enforced)

  • Every /e2e request requires X-E2E-Runner-ID plus Authorization: Bearer <token>; browser/AuthCrunch roles do not grant E2E access.
  • POST /e2e/runs requires X-E2E-Protocol-Version header.
  • client_version/server_version request fields are rejected (immediate break).
  • trigger_source accepts manual|ci and is reporting metadata only.
  • Runs, results, cancellation, and logs are owner-scoped to the authenticated runner.
  • Idempotency applies to run creation only: (runner_id, environment_label, idempotency_key) with 1-hour TTL.
  • Only one active run per environment is allowed at a time (409 environment_locked).
  • Run creation fails if baseline seed fails (422 seed_failed).
  • Action/expect pairs are validated by server-side registry before accepting run execution (400 invalid_action_expectation).

Execution Model

  1. Client loads config and journey specs.
  2. Client authenticates as an E2E runner and sends POST /e2e/runs with protocol header and optional idempotency key.
  3. Server validates protocol/env/suite/journeys/action-expect pairs, acquires environment lock, runs seed script, and queues run+journey rows.
  4. Client executes journey steps and writes JSONL logs using schema e2e_log.v1.
  5. Client submits per-journey outcomes via POST /e2e/runs/:id/results.
  6. Server marks final run status from journey outcomes and releases lock.

Journey semantics:

  • Fail-fast inside each journey.
  • Continue other journeys in the same run.
  • Aggregate run status from all journey results.

JSONL v1 semantics:

  • journey started record includes run context (run_id, suite_id, environment_label, trigger_source, protocol_version).
  • Exactly one terminal record per step (status=passed|failed) with duration_ms.
  • journey completed record includes duration_ms, steps_run, steps_passed, steps_failed, and failure summary fields.
  • Response metrics (http_status, response_type, bytes, truncated) are canonical top-level fields on step records.
  • action_data stores action-specific payloads; sensitive keys are redacted before writing logs.
  • bytes is the original response-body size before truncation.
  • truncated=true means action_data.body_preview was clipped to the first 500 bytes.

Add a New E2E Journey

  1. Generate scaffold: packages/client/scripts/e2e/scaffold --id <journey_id> --suite <suite_id> --description "<text>" --steps action_a:expect_a,action_b:expect_b (preview-only: add --dry-run; overwrite existing files: add --force) Optional for advanced cases: step_id=action:expect (when the step label must differ from the action token).
  2. Fill in client action handlers from the generated stub at packages/client/internal/e2e/scaffold/<journey_id>.go.stub.
  3. Register action/expect tokens using generated server stub at packages/server/lib/nixstasis/e2e/scaffold/<journey_id>.ex.stub.
  4. Add journey to suite mappings in packages/client/scripts/e2e/config.example.yaml, packages/server/config/dev.exs, and packages/server/config/test.exs.
  5. Run E2E and verify UI/API rendering.
  6. Add tests in packages/client/internal/e2e and packages/server/test/nixstasis/e2e + controller tests.

An E2E Test by Example

Reference journey: runtime_step_labels

Run command:

cd packages/client
scripts/e2e/run --suite runtime_step_labels --journey step_labels --env local --trigger manual --protocol-version 1

Files that make up this example journey:

  • packages/client/scripts/e2e/journeys/runtime_step_labels.yaml: Journey definition with optional step_id labels (step_id != action) to demonstrate cleaner dashboard labels.
  • packages/server/config/dev.exs: Adds suite mapping "runtime_step_labels" => ["runtime_step_labels"] for local runs.
  • packages/server/config/test.exs: Adds suite mapping "runtime_step_labels" => ["runtime_step_labels"] for test env parity.
  • packages/client/internal/e2e/journey.go: Supports optional step_id in YAML and fallback behavior (step_id defaults to action).
  • packages/client/internal/e2e/journey_executor.go: Emits step_id only when different from action, reducing redundant log fields.
  • packages/server/lib/nixstasis_web/live_dashboard/e2e_log_presenter.ex: Prefers step_id over action in Journey Log display when custom labels are provided.

Reliability + Lifecycle

  • Environment lock table prevents overlapping runs in same environment.
  • Retention is enforced in server app via periodic worker:
    • retention_days (default 14)
    • max_run_count (default 2000)
    • max_log_bytes (default 1_000_000_000)
    • pruning order is oldest-first.
  • Deleting a run also deletes associated log files.
  • Missing/pruned logs return typed log_unavailable errors (HTTP 410) so UI can show non-blocking unavailable states.

CI + GitHub Pages

  • Workflow: .github/workflows/e2e-pages.yml
  • Runs all server-configured suites in CI by calling:
    • packages/client/scripts/e2e/run_all_suites
    • suite list is fetched from GET /e2e/suites
  • Publishes static pages through:
    • packages/server/lib/mix/tasks/e2e.export_static.ex
  • Pages output is manifest-led:
    • runs.json is source of truth
    • run directories live under runs/<ref-name>/<short-sha>/
    • root index.html loads runs.json client-side
    • manifest/report values are rendered with text-only DOM APIs; untrusted run metadata is never inserted as HTML
  • Retention:
    • governed by MAX_E2E_RUNS (fallback 200)
    • non-release runs are pruned oldest-first
    • semver tags (X.Y.Z and vX.Y.Z) are preserved forever

File Types and Ownership

File Type Path Pattern Owned By Purpose
Journey spec YAML packages/client/scripts/e2e/journeys/*.yaml Client Declarative journey specs (id, description, steps).
Runner entry script packages/client/scripts/e2e/run Client Shell entrypoint for E2E CLI.
Runner CLI main packages/client/scripts/e2e/main.go Client Flag parsing and run launch.
Runner config YAML packages/client/scripts/e2e/config.example.yaml Client Default suite/env/protocol/journeys.
Runner orchestration packages/client/internal/e2e/runner.go Client Creates run, executes journeys, submits results.
Journey execution + logs packages/client/internal/e2e/journey_executor.go Client Action execution + JSONL log emission.
E2E API client packages/client/internal/e2e/api.go Client Calls run creation/results endpoints and sets protocol header.
E2E context packages/server/lib/nixstasis/e2e.ex Server Run lifecycle, idempotency, locking, retention, log reads.
Contract guards packages/server/lib/nixstasis/e2e/protocol.ex, journey_selection.ex, data_policy.ex, expectation_registry.ex Server Protocol, suite/journey, env policy, and action/expect validation.
Locking packages/server/lib/nixstasis/e2e/environment_lock.ex, environment_locks.ex Server Single active run per environment.
Log storage packages/server/lib/nixstasis/e2e/log_store.ex Server (reads client logs too) Safe read/write/delete + availability semantics.
Retention worker packages/server/lib/nixstasis/e2e/retention_worker.ex Server Periodic pruning by age/count/size policy.
HTTP controllers packages/server/lib/nixstasis_web/controllers/e2e_run_controller.ex, e2e_run_result_controller.ex Server Typed API responses and error codes.
Dashboard page packages/server/lib/nixstasis_web/live_dashboard/e2e_page.ex Server Runs/results/log visualization.
Migrations packages/server/priv/repo/migrations/20260209235459_create_e2e_runs.exs, packages/server/priv/repo/migrations/20260212163000_harden_e2e_contracts.exs Server Schema for runs/results and reliability hardening fields.
Seed script packages/server/priv/e2e/seed.exs Server Baseline data reset before run acceptance.
E2E API docs website/docs/development/e2e/server.md, website/docs/client-server-interface.md Both Contract reference for endpoints/payloads/errors.

Architecture Overview

Status Snapshot

  • Client rewrite (Go): Core functionality is implemented, including registration, heartbeat polling, command polling, FRP config rendering, script execution guardrails, and native packaging.
  • Server rewrite (Elixir/Phoenix): Core monitoring, dashboard, device management, reporting, E2E control APIs, and Compose-based deployment are implemented. The server uses Phoenix, Ash, and Ecto migrations.

FRP (Fast Reverse Proxy)

This service has two components:

  • frps: The server, which must run on the same server as Nixstasis and must have a wildcard domain name assigned to the IP Address that the server is running. The IP Address must be publicly reachable on the Internet.
  • frpc: This is the client that must be deployed on the devices that Nixstasis will monitor and provide remote access.

These services are configured using a toml file. See the FRP docs for configuration details.

This repo includes a fully deployable configuration for the server designed for this service. It currently supports both ssh and https proxying. This allows Nixstasis to provide remote access to the device's Web UI (Cockpit) and to connect to the device via a terminal using ssh if for some reason the Web UI becomes unreachable due to a configuration issue.

The nixstasis client includes a configuration file as well, that uses environment variables to provide a template. When the Nixstasis client starts the frpc program, it provides the required values to ensure that each device gets a unique subdomain. The current pattern for subdomains is atom-${MAC_ADDRESS} where the MAC Address is lowercased with the : stripped. For example atom-8268a89d95e7.

Caddy Server

Caddy is the front-end server and provides these features:

  • Reverse proxying to FRPS
  • Automatic TLS (HTTPS) certificates per device
  • Static file serving for Nixstasis's UI
  • Authentication via the AuthCrunch plugin

FRPS provides the public tunnel to devices but is not used for TLS termination. FRPS also does not reliably handle HTTP upgrade requests (required for WebSockets) and lacks built-in ACME support. Caddy handles TLS termination and ACME, which is why it's used in front of FRPS.

For on-demand certificates, Caddy requests approval the first time a subdomain is accessed. Nixstasis exposes the /api/v1/check_domain endpoint to authorize these requests. The endpoint expects a domain query parameter and only allows a certificate when the domain should be exposed for remote access. If no device matches the requested domain, or if remote access hasn't been requested, the request is denied. Requests for the nixstasis, auth, and frp-admin domains are always approved because Nixstasis depends on them.

Caddy may be configured using its Caddyfile or JSON formats; the repo includes a deployable Caddyfile covering the current features.

AuthCrunch

AuthCrunch is a plugin for Caddy that provides authentication, authorization, and accounting (AAA). Nixstasis uses it to sign operators in through any OpenID Connect or OAuth 2.0 identity provider it supports, such as Microsoft Entra ID, Google, GitHub, GitLab, Okta, or Meta. The production Caddyfile uses Microsoft Entra ID as its example.

AuthCrunch transform user rules map provider groups to the nixstasis/viewer, nixstasis/operator, and nixstasis/admin roles, which the server uses to decide what each operator can do. See Gateway Caddy for switching providers.