M1: the free probe, and a language that is part of a page

.github/workflows/ci.yml

@@ -156,6 +156,13 @@ jobs:
         run: npm run typecheck
         working-directory: web
 
+      # node --test, running the TypeScript directly by stripping the types. Needs
+      # Node 22.18 or newer for that to be on without a flag; NODE_VERSION is "22",
+      # which setup-node resolves to the latest 22.x.
+      - name: Test
+        run: npm test
+        working-directory: web
+
       - name: Build
         run: npm run build
         working-directory: web

.gitignore

@@ -20,6 +20,16 @@ config.local.yaml
 /web/node_modules/
 /web/dist/
 /web/.vite/
+# Where `vite build --ssr reader-check.tsx` and `search-check.tsx` put their
+# bundles. Both render a screen to HTML on the command line, which is what a
+# headless browser is then pointed at to take a screenshot of an authenticated page.
+/web/.reader-check/
+/web/.search-check/
+# Where `vite build --config folio-browser-check.vite.ts` puts its bundle. Unlike
+# the two above this one is a whole page a browser opens directly, with a real
+# document's conversion JSON compiled into it — which is the other reason it is
+# never committed.
+/web/.folio-check*/
 *.tsbuildinfo
 
 # go
@@ -38,6 +48,11 @@ coverage.html
 /tmp/
 /scratch/
 
+# Agent worktrees. Root-anchored for the same reason as /manualbox above: these
+# are checkouts of this repository, and `git add -A` will otherwise stage one as
+# an embedded repository.
+/.claude/worktrees/
+
 # Vite builds the SPA into internal/frontend/dist/app so it can be go:embed-ed.
 # It writes into the app/ subdirectory rather than dist/ itself because Vite's
 # emptyOutDir wipes its output directory on every build — which silently deleted

CONTRIBUTING.md

@@ -86,6 +86,9 @@ measurements:
 | | |
 |---|---|
 | [ingest.md](docs/design/ingest.md) | How a 560-page, 34-language manual is reduced to the 16 pages you actually read, before any model is called |
+| [layouts.md](docs/design/layouts.md) | Manuals are arranged differently — sequential sections or parallel columns — and where the seam between them goes |
+| [regions.md](docs/design/regions.md) | Contract for storing a language that occupies part of a page, and what it deliberately leaves unsolved |
+| [language-detection.md](docs/design/language-detection.md) | The four signals that say what language a page is in, what each one costs, and why the detector choice is deliberately still open |
 | [providers.md](docs/design/providers.md) | Why a subscription CLI or a local model comes before a metered API key, and why a CLI adapter must batch a whole document |
 | [privacy.md](docs/design/privacy.md) | What manualbox holds, ranked by how it actually leaks |
 | [keys.md](docs/design/keys.md) | Encryption keys: choosing, storing, and recovering them |

Makefile

@@ -49,10 +49,17 @@ web-build: ## Build the frontend into web/dist
 web-typecheck: ## Typecheck the frontend
 	cd web && npm run typecheck
 
+# `node --test`, which runs the TypeScript directly by stripping the types. No test
+# framework and no new dependency: the alternative was adding a runner to assert
+# rules that are a few lines each.
+.PHONY: web-test
+web-test: ## Run frontend tests
+	cd web && npm test
+
 ## ---- quality ----
 
 .PHONY: check
-check: test lint web-typecheck ## Everything CI runs
+check: test lint web-typecheck web-test ## Everything CI runs
 
 .PHONY: test
 test: ## Run Go tests with race detector

README.md

@@ -7,9 +7,12 @@
 
 Self-hosted. Free. MIT. Single binary, SQLite, no external services required, and **no API key needed** to get real value out of it.
 
-> ⚠️ Early development. The server runs, you can create an account and sign in, and background
-> jobs report live progress — but **you cannot add devices or manuals yet**. That is M1, the next
-> milestone. See the [Roadmap](#roadmap).
+> ⚠️ Early development. You can create an account, add devices, and upload a manual — manualbox
+> reads it locally and tells you what it contains, in which languages, before anything is
+> converted or sent anywhere. Approve it and the pages you read are converted to blocks, and
+> `GET /api/v1/search?q=` finds the sentence you need across every manual in the house.
+> **It stops there for now:** the reader screen, the search screen and export are still to
+> come. See the [Roadmap](#roadmap).
 
 ## Try it
 
@@ -75,7 +78,7 @@ Then it gets out of the way: notifications where you already look, and a calenda
 | | |
 |---|---|
 | **M0** ✅ | Skeleton: config, SQLite + migrations, blob store, job queue, auth, API, frontend shell, Docker, CI |
-| **M1** | Registry, document pipeline (convert → language-segment → index), reader, full-text search, **export** — see [ingest design](docs/design/ingest.md) |
+| **M1** | Registry ✅, document probe and language map ✅, conversion ✅, full-text search ✅, then the reader and **export** — see [ingest](docs/design/ingest.md), [conversion](docs/design/conversion.md) and [search](docs/design/search.md) |
 | **M2** | Maintenance: schedules, battery charge cycles, service log, notifications, ICS calendar feed |
 | **M3** | Translation: per-block, glossary, translation memory, side-by-side, post-editing |
 | **M4** | Extraction: maintenance plans with citations, printable per-device cheat sheets, error-code lookup |
@@ -101,8 +104,10 @@ Optional external binaries, used when present and degraded gracefully when not (
 | `tesseract` | OCR for scans and photos | `brew install tesseract tesseract-lang` |
 
 Design decisions, with the measurements behind them, are in `docs/design/`:
-[ingest](docs/design/ingest.md) · [providers](docs/design/providers.md) ·
-[privacy](docs/design/privacy.md) · [keys](docs/design/keys.md).
+[ingest](docs/design/ingest.md) · [layouts](docs/design/layouts.md) ·
+[language detection](docs/design/language-detection.md) ·
+[providers](docs/design/providers.md) · [privacy](docs/design/privacy.md) ·
+[keys](docs/design/keys.md).
 
 Conventions and the things that have already caused bugs here:
 [CONTRIBUTING.md](CONTRIBUTING.md).

cmd/manualbox/main.go

@@ -24,8 +24,10 @@ import (
 	"github.com/gordon2/manualbox/internal/config"
 	"github.com/gordon2/manualbox/internal/db"
 	"github.com/gordon2/manualbox/internal/extern"
+	"github.com/gordon2/manualbox/internal/ingest"
 	"github.com/gordon2/manualbox/internal/jobs"
 	"github.com/gordon2/manualbox/internal/logging"
+	"github.com/gordon2/manualbox/internal/registry"
 	"github.com/gordon2/manualbox/internal/store"
 )
 
@@ -68,6 +70,8 @@ func run(ctx context.Context, args []string, stdout, stderr io.Writer) error {
 		return cmdServe(ctx, rest, stdout, stderr)
 	case "doctor":
 		return cmdDoctor(ctx, rest, stdout)
+	case "verify":
+		return cmdVerify(ctx, rest, stdout, stderr)
 	case "version", "--version", "-v":
 		fmt.Fprintf(stdout, "manualbox %s (%s)\n", version, commit)
 		return nil
@@ -89,6 +93,7 @@ Usage:
 Commands:
   serve      Run the web server and background workers
   doctor     Report configuration and which optional tools are available
+  verify     Convert a PDF and report what is wrong with the conversion
   version    Print the version
   help       Show this help
 
@@ -98,6 +103,10 @@ Flags (serve, doctor):
 Flags (doctor):
   -redact          Replace your home directory with ~, for pasting into a bug report
 
+Flags (verify):
+  -limit <n>       How many findings of each kind to print (default 20)
+  -all             Print every finding
+
 Configuration comes from defaults, then the config file, then MANUALBOX_*
 environment variables. Run "manualbox doctor" to see what was resolved.
 `)
@@ -197,19 +206,31 @@ func cmdServe(ctx context.Context, args []string, stdout, stderr io.Writer) erro
 	queue := jobs.NewQueue(database, log)
 	defer queue.Broker().Close()
 
+	registryService := registry.New(database, registry.Options{Logger: log})
+	ingestService := ingest.New(ingest.Deps{
+		Config:   cfg,
+		Registry: registryService,
+		Store:    blobs,
+		Jobs:     queue,
+		Logger:   log,
+	})
+
 	pool := jobs.NewPool(queue, cfg.Jobs, log)
-	// M1 registers the real conversion, OCR, translation, and extraction
-	// handlers here. Until then the pool runs with none, and a job of an unknown
-	// kind fails with a clear message rather than hanging.
+	// The document pipeline's handlers. Conversion, OCR, translation, and
+	// extraction register here as they land; a job of an unknown kind fails with a
+	// clear message rather than hanging.
+	ingestService.Register(pool)
 
 	server := api.New(api.Deps{
-		Config:  cfg,
-		DB:      database,
-		Store:   blobs,
-		Auth:    authService,
-		Jobs:    queue,
-		Logger:  log,
-		Version: version,
+		Config:   cfg,
+		DB:       database,
+		Store:    blobs,
+		Auth:     authService,
+		Jobs:     queue,
+		Registry: registryService,
+		Ingest:   ingestService,
+		Logger:   log,
+		Version:  version,
 	})
 
 	httpServer := &http.Server{

cmd/manualbox/verify.go

@@ -0,0 +1,121 @@
+package main
+
+import (
+	"context"
+	"errors"
+	"flag"
+	"fmt"
+	"io"
+	"sort"
+	"text/tabwriter"
+	"time"
+
+	"github.com/gordon2/manualbox/internal/doc"
+	"github.com/gordon2/manualbox/internal/verify"
+)
+
+// cmdVerify converts a PDF and reports what is wrong with the conversion.
+//
+// It exists to be run on a manual the fixtures do not contain, which is where
+// this check earns its keep: the two measured documents are the two whose defects
+// are already written down. Nothing is stored and nothing is uploaded — the same
+// stance doctor takes — so it is safe to point at a file and read the answer.
+//
+// The document is converted for EVERY language it holds rather than for the
+// household's, because coverage is measured against all the text on a page and a
+// page of a parallel-columns manual holds five languages. See [verify.Check].
+func cmdVerify(ctx context.Context, args []string, stdout, stderr io.Writer) error {
+	fs := flag.NewFlagSet("verify", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	limit := fs.Int("limit", 20, "how many findings of each kind to print")
+	all := fs.Bool("all", false, "print every finding rather than the first few of each kind")
+	if err := fs.Parse(args); err != nil {
+		return err
+	}
+	if fs.NArg() != 1 {
+		return errors.New("usage: manualbox verify [-limit n] [-all] <file.pdf>")
+	}
+	path := fs.Arg(0)
+
+	start := time.Now()
+	res, err := doc.Analyze(ctx, path)
+	if err != nil {
+		return err
+	}
+	langs := res.Languages()
+	names := make([]string, 0, len(langs))
+	for i := range langs {
+		names = append(names, langs[i].Lang)
+	}
+	fmt.Fprintf(stdout, "%d pages, %d language(s): %s\nprobed in %v\n",
+		res.Info.Pages, len(langs), join(names), time.Since(start).Round(time.Millisecond))
+
+	start = time.Now()
+	conv, err := verify.ConvertAll(ctx, path, res)
+	if err != nil {
+		return err
+	}
+	fmt.Fprintf(stdout, "%s\nconverted in %v\n", conv.Summary(),
+		time.Since(start).Round(time.Millisecond))
+	for _, n := range conv.Notes {
+		fmt.Fprintf(stdout, "  note: %s\n", n)
+	}
+
+	start = time.Now()
+	rep, err := verify.Check(ctx, path, conv)
+	if err != nil {
+		return err
+	}
+	fmt.Fprintf(stdout, "\n%s\nchecked in %v\n\n", rep.Summary(),
+		time.Since(start).Round(time.Millisecond))
+	for _, n := range rep.Notes {
+		fmt.Fprintf(stdout, "  note: %s\n", n)
+	}
+
+	tw := tabwriter.NewWriter(stdout, 0, 8, 2, ' ', 0)
+	fmt.Fprintf(tw, "  kind\tfindings\tpages\n")
+	kinds := rep.Kinds()
+	for _, k := range verify.AllKinds {
+		if kinds[k] == 0 {
+			continue
+		}
+		fmt.Fprintf(tw, "  %s\t%d\t%d\n", k, kinds[k], rep.PagesFlagged(k))
+	}
+	tw.Flush()
+
+	// The worst pages by coverage, whether or not they were reported: the
+	// measurement is what a person reads this for, and a document whose worst page
+	// scores 0.99 has been told something by that number.
+	cov := make([]verify.PageCoverage, len(rep.Coverage))
+	copy(cov, rep.Coverage)
+	sort.Slice(cov, func(a, b int) bool { return cov[a].Ratio < cov[b].Ratio })
+	fmt.Fprintf(stdout, "\nmedian coverage %.3f; least covered pages:\n", rep.MedianCoverage())
+	for i := 0; i < len(cov) && i < 5; i++ {
+		fmt.Fprintf(stdout, "  page %d: %.3f (%d block characters against %d from pdftotext)\n",
+			cov[i].Page, cov[i].Ratio, cov[i].Blocks, cov[i].Text)
+	}
+
+	shown := make(map[verify.Kind]int, len(kinds))
+	fmt.Fprintln(stdout)
+	for i := range rep.Findings {
+		f := &rep.Findings[i]
+		if !*all {
+			if shown[f.Kind] >= *limit {
+				continue
+			}
+			shown[f.Kind]++
+		}
+		fmt.Fprintf(stdout, "%s: %s\n", f.Kind, f.Detail)
+		if f.Sample != "" {
+			fmt.Fprintf(stdout, "    %s\n", f.Sample)
+		}
+	}
+	if !*all {
+		for k, n := range kinds {
+			if n > shown[k] {
+				fmt.Fprintf(stdout, "… %d more %s finding(s); -all prints them\n", n-shown[k], k)
+			}
+		}
+	}
+	return nil
+}