AutoNotes Project

AutoNotes is a prototype of a math handwritten notes OCR system based purely on AI using multimodal models.

This project explores a workflow for automatically transcribing complex mathematical notes while maintaining strict verification. When studying new material, blindly trusting AI transcriptions is risky as it's hard to distinguish between correct content and hallucinations. AutoNotes addresses this by leveraging the spatial reasoning of models like Gemini 3 Flash (which are pretty good at retrieving box_2d coordinates) to create a "ground truth" mapping between the digital transcription and the original handwriting.

The result is an HTML visualization where everything is first transcribed as is and then reworded by the LLM (keeping the thread of source boxes). Every theorem or definition is linked back to a specific region in the source image, ensuring you can always verify the output against your original notes.

As models are also able to recognize math notation and figures in the image, we also get for free a tool that automatically inserts them in the transcribed text.

Interactive Fuzzy Search (New!)

AutoNotes now includes a robust Fuzzy Search Dashboard to locate definitions, theorems, propositions, corollaries, and lemmas instantly.

• Instant Matching: Employs Fuse.js client-side to do fuzzy indexing across all notes, lesson titles, and courses.
• Math-Rich Previews: Search results display fully rendered styled mathematics via MathJax inline rendering.
• Native Masonry Layout: Styled using a high-fidelity CSS Grid Masonry layout (display: grid-lanes;) for optimized visual presentation.
• Direct Navigation: Clicking on any card opens the corresponding lesson note and navigates directly to the specific target element.

Study Repeat Mode

If you want to read through notes without getting distracted by a wall of text, you can turn on Study Repeat Mode (or Reword Focus).

• Focus Fading: Everything fades out to 5% opacity except for the block right in the middle of the screen.
• Auto Scroll Tracking: The focus moves automatically as you scroll, guiding your eyes naturally to the next section.
• Side Indicator: A small arrow pointer follows you on the left margin so you never lose your place.
• Step-by-step Reading: Great for reviewing proofs, definitions, or equations line-by-line, keeping your focus strictly on what you're currently reading.

Collapsible Spoilers

AutoNotes supports collapsible <spoiler> blocks to hide details and let you drill down into math arguments with more and more depth.

• Progressive Disclosure: Hide long calculations, checks, or proofs so you can grasp the big picture first, then click to reveal the details.
• Nested Details: Since spoilers can be nested inside other spoilers, you can drill down into proofs of proofs, or step-by-step expansions of equations.
• Seamless Inline Flow: Spoilers integrate right inside text paragraphs with proper punctuation handling and inline math.
• Click to Toggle: Just click the text to expand or collapse the section, complete with responsive Lucide icons.

Agent-Assisted Transcription

This project is explicitly designed to be operated by AI agents. Agents are responsible for viewing source images, extracting mathematical content, and mapping specific regions (using a 1000x1000 coordinate system) into .note files.

Recommended Environment

For best performance in OCR, math transcription and spatial mapping, I recommend using Google's Gemini 3 Flash (or higher) models. I used these for free using Antigravity.

For detailed instructions on the transcription protocol, agents must refer to AGENTS.md.

Getting Started

1. Install Go: Ensure you have Go (Golang) installed on your system.
2. Initialize: Run the following command to download dependencies and ensure everything is built:

   go run ./tool/cmd/autonotes --help

3. Create a Collection: Create a new directory under src/ for your notes and an images/ folder inside it:

   mkdir -p src/my-course/images

4. Add Images: Place your handwritten note images (JPEG/PNG) into the src/my-course/images/ directory.
5. Start Transcription: Once images are in place, you can use the process-new-images.md workflow in a coding agent like Antigravity to kick-start the conversion process.

How It Works

The core philosophy of AutoNotes is a human-in-the-loop (or rather, agent-in-the-loop) transcription workflow:

1. Source Tracking: The go run ./tool/cmd/autonotes status command scans your src/ directory. It parses all existing .note files to extract the names of images already mapped. By comparing this list with the files in your images/ folders, it identifies exactly which pages still need to be transcribed (to immediately tell the agent what he has to do).

2. Spatial Mapping: Using the 1000x1000 coordinate system, agents define <box> elements that link specific regions of an image to their literal transcription. This provides the "ground truth" and enables the interactive lens-crop view in the final HTML.

3. Formalization (Reword): Literal transcriptions are often messy. The <reword> tag is used to provide a professional, LaTeX-formatted version of the content. These blocks are linked back to the source <box> elements, allowing users to hover over a formal proof and see exactly where it came from in the original notes.

4. Fast Static Generation: The go build and serve commands compile these structured notes into a standalone static website that features MathJax math rendering and can be hosted wherever you want.

Flashcards

The project integrates with Hashcards, a plain-text spaced repetition system. Transcription agents can automatically generate flashcards in the cards/ directory based on the theorems and definitions extracted from the notes.

Folder Structure

• src/: Contains the source materials.
  • [collection-name]/: E.g., ist-geom/.
    • images/: The original handwritten photos (JPEG).
    • [filename].note: The structured transcription files.
• out/: The generated standalone website.
• tool/: Core logic and CLI implementation.
  • cmd/autonotes/: CLI entry point.
• AGENTS.md: Detailed workflow and instructions for transcription agents.

CLI Commands

The converter tool provides several subcommands for managing the transcription workflow.

Global Flags

• -d, --debug: Enable debug mode for more verbose output.

Subcommands

• go run ./tool/cmd/autonotes status: Lists images that have not been transcribed yet.
• go run ./tool/cmd/autonotes build: Generates the HTML website in the out/ directory.
• go run ./tool/cmd/autonotes sync: Runs status followed by build.
• go run ./tool/cmd/autonotes serve: Starts a local server with live-reload for previewing changes.
  • -p, --port <port>: Port to serve on (default 8080).
  • -H, --host <host>: Host to serve on (default localhost).
  • --reload-static: Also watch the tpl/ folder for changes.
• go run ./tool/cmd/autonotes check: Validates all .note files in src/ for syntax errors.
• go run ./tool/cmd/autonotes query [path]: Search and filter content across all notes or a specific path.
  • -s, --select <types>: Filter by block types (e.g., theorem,definition).
  • -g, --grep <pattern>: Search for text within blocks.
  • -e, --extract <types>: Extract specific child blocks (e.g., reword).
  • -v, --verbose: Enable verbose logging to stderr.
  • query summary <path>: Extract the lesson summary from a specific file.
  • query histogram [subfolder]: Print counts for all found tag types (optionally filtered by subfolder).

Usage

Typically, you will want to keep the server running in one terminal:

go run ./tool/cmd/autonotes serve

And then use other commands like status or query in another terminal to find work or search through existing notes.

Syntax Overview (.note files)

Note files use an XML-like syntax to map transcriptions to images using a 1000x1000 coordinate system.

Root & Document Structure

• <lesson date="YYYY-MM-DD" course="Course Name">: The root wrapper for every note file.
• <summary>: The first element inside <lesson>, containing a short and concise summary of the lesson (renders as the card description on the dashboard).
• <section title="Section Title">: Standard heading block used to divide note files into sections.

Base Elements

• <box image="name.jpg" top="Y" right="X" bottom="Y" left="X" uid="unique-id">: Maps transcribed text to a region on an image. Triggers a red highlight and lens view on hover. Must include a unique uid.
• <math> and <math display="true">: KaTeX-powered mathematical expressions (inline or display block).
• <reword ref="box-uid">: Formal textbook-style rewrite of the transcription. The ref attribute links it back to one or more <box> elements (supports bracket group syntax like ref="subgroup-[thm,dim]").
• <image src="name.jpg" top="..." right="..." bottom="..." left="..." />: Inline cropped diagram. Also supports hover highlighting and lens view.
• <spoiler>: Collapsible element to hide details. Requires exactly two children: <preview> (visible text) and <content> (hidden details).

Structural Blocks

• <definition>, <theorem>, <lemma>, <proposition>, <corollary>, <exercise>, <fact>: Semantic boxes for theorems, statements, and observations (fact is used for osservazioni). Renders with specific background colors.
• <dim>, <richiami>: Semantic blocks for proofs (dimostrazioni) and recalls (richiami).
• <itemize>, <enumerate>, <item>: Bulleted and numbered lists.

Inline Formatting

• <strong>...</strong>: Renders text in bold.
• <emph>...</emph>: Renders text in italics/emphasis.
• <a href="...">...</a>: Creates hyperlink cross-references between notes.

TODO

• Convert the syntax to real XML, not XML-like

• Refactor the images/ folder concept to a sources/ folder that can contain both images, zip of images, pdfs, etc. There is a command called go run ./tool/cmd/autonotes extract that unpacks archives and pdfs to a folder with a similar name that only contains image files. When generating the website, everything is flattened, given uuids, and pdfs are converted to images (with mutools).

• Add the concept of symbols, also embed in the math content, e.g. <math><ref id="symbol-name">\alpha</ref></math>. When converting to HTML, the refs get replaced with \href or more complicated \htmlData tags (katex feature).

  • Global Cross-Referencing: The <ref id="...">...</ref> tag is a universal linking mechanism that can wrap math symbols or plain text, creating an interconnected web of concepts.

  • Symbol Definitions:

    • Use <symbol id="..." name="..." description="..." latex="..." /> to define global identifiers.
    • Definitions can be centralized in a symbols.note or declared at the start of a chapter.

  • Enhanced Examples:

    • Constants: <math><ref id="const-e">e</ref>^{i\pi} + 1 = 0</math> links to the definition of the Euler's number.
    • Formal Terms: L'<ref id="def-omeomorfismo">omeomorfismo</ref> tra i due spazi... links to the topological definition.
    • Theorems: Per il <ref id="theorem-cauchy">Teorema di Cauchy</ref>... links to the statement and its proof.

  • Interactive HTML Delivery:

    • Contextual Tooltips: Hovering over a reference displays a floating card with the target's name, description, and LaTeX preview.
    • Back-references: Each definition automatically tracks and lists all the places where it is referenced across the collection.
    • Navigation: Deep links that scroll to the exact block (theorem, definition) or highlight the specific occurrence.

  • Agent Tooling:

    • Symbol Inspection: A subcommand go run ./tool/cmd/autonotes symbols will provide a CLI interface for agents to list all defined symbols, verify their metadata, and trace their cross-reference graph (where they are used and what they link to).

• Add a search bar using FuseJS

---

For more detailed instructions on transcription, see AGENTS.md.