Skip to content

Commit a26d02e

Browse files
nodeeeeeenodeeeeee
committed
added GUI
1 parent de80df5 commit a26d02e

5 files changed

Lines changed: 2736 additions & 70 deletions

File tree

‎README.md‎

Lines changed: 297 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,297 @@
1+
# auto_note
2+
3+
Automated lecture note generator for NUS Canvas courses.
4+
Downloads lecture videos and slides, transcribes audio, aligns transcripts to slides, and produces comprehensive Markdown study notes using GPT.
5+
6+
---
7+
8+
## Overview
9+
10+
```
11+
Canvas LMS
12+
│
13+
├─ downloader.py Unified downloader (videos + materials)
14+
│ ├─ [course_id]/videos/ .mp4 files
15+
│ └─ [course_id]/materials/ PDFs, PPTXs, etc.
16+
│
17+
▼
18+
extract_caption.py Transcribe video → timestamped JSON
19+
│
20+
▼
21+
semantic_alignment.py Align transcript segments → slide pages
22+
│
23+
▼
24+
note_generation.py Generate Markdown study notes (GPT)
25+
│
26+
▼
27+
[course_id]/notes/
28+
├─ CourseName_notes.md
29+
└─ sections/ Per-chunk cached section files
30+
```
31+
32+
---
33+
34+
## Setup
35+
36+
### Conda environment
37+
38+
```bash
39+
conda activate auto-note
40+
```
41+
42+
### API keys
43+
44+
| Key | Location |
45+
|-----|----------|
46+
| Canvas token | `canvas_token.txt` or `CANVAS_TOKEN` constant in `downloader.py` |
47+
| OpenAI key | `openai_api.txt` or `OPENAI_API_KEY` env var |
48+
49+
### Course IDs
50+
51+
Known NUS courses:
52+
53+
| ID | Course |
54+
|----|--------|
55+
| 85367 | CS2101 |
56+
| 85377 | CS2103 |
57+
| 85397 | CS2105 |
58+
| 85427 | CS3210 |
59+
60+
---
61+
62+
## Pipeline — Step by Step
63+
64+
### 1. Download videos and materials — `downloader.py`
65+
66+
Unified Canvas downloader combining video (Panopto) and material (Files API) downloads into a single CLI.
67+
68+
#### Course discovery
69+
70+
```bash
71+
python downloader.py --course-list # list all academic courses with IDs
72+
```
73+
74+
#### Video operations
75+
76+
```bash
77+
python downloader.py --video-list # list all available videos (global numbers)
78+
python downloader.py --video-list --course 85427 # list videos for one course (course-local numbers)
79+
80+
python downloader.py --download-video 1 3 5 # download by global number
81+
python downloader.py --download-video 2 4 --course 85427 # download by course-local number
82+
python downloader.py --download-video-all # download all pending videos
83+
python downloader.py --download-video-all --course 85427 # all pending for one course
84+
```
85+
86+
**Output:** `[course_id]/videos/[title].mp4`
87+
88+
A `manifest.json` in the project root tracks downloaded videos and prevents re-downloads.
89+
90+
#### Material operations
91+
92+
```bash
93+
python downloader.py --material-list # list all downloadable files
94+
python downloader.py --material-list --course 85427 # files for one course
95+
96+
python downloader.py --download-material "L02" # download by partial filename match
97+
python downloader.py --download-material-all # download all pending materials
98+
python downloader.py --download-material-all --course 85427
99+
```
100+
101+
**Smart size filter:** If a course's total files exceed 1 GB, GPT identifies only lecture notes and tutorials for download, skipping large multimedia assets.
102+
103+
**Output:** `[course_id]/materials/[canvas_subfolder]/[filename]`
104+
A `download_log.json` per course prevents re-downloading already-present files.
105+
106+
#### Common flags
107+
108+
| Flag | Description |
109+
|------|-------------|
110+
| `--course ID` | Restrict all operations to a single course |
111+
| `--secretly` | Random delays between downloads to avoid rate-limiting (5–15 min between videos, 2–5 min between folders) with tqdm countdown bars |
112+
| `--path PATH` | Override base directory (default: directory of `downloader.py`) |
113+
114+
#### Stealth mode example
115+
116+
```bash
117+
python downloader.py --download-video-all --course 85427 --secretly
118+
python downloader.py --download-material-all --secretly --path /data/courses
119+
```
120+
121+
---
122+
123+
### 2. Transcribe videos — `extract_caption.py`
124+
125+
Transcribes `.mp4` lecture videos to word-level timestamped JSON using **faster-whisper large-v3** on GPU. The model reads video files directly (no separate audio extraction step).
126+
127+
```bash
128+
python extract_caption.py # process all pending videos
129+
python extract_caption.py --video PATH # single video file
130+
```
131+
132+
**Requires:** CUDA GPU (enforced at startup — will not silently fall back to CPU).
133+
134+
**Output:** `[course_id]/captions/[title].json`
135+
Each JSON contains word-level timestamps and confidence scores.
136+
137+
---
138+
139+
### 3. Align transcript to slides — `semantic_alignment.py`
140+
141+
Maps Whisper transcript segments to specific slide pages using dense semantic search.
142+
143+
```bash
144+
# Align one caption to one slide file
145+
python semantic_alignment.py \
146+
--caption 85427/captions/lecture.json \
147+
--slides 85427/materials/LectureNotes/L02.pdf
148+
149+
# Align one caption to multiple slide files (multi-part lecture)
150+
python semantic_alignment.py \
151+
--caption 85427/captions/L04.json \
152+
--slides 85427/materials/L04-Part1.pdf 85427/materials/L04-Part2.pdf
153+
154+
# Auto-discover all unaligned pairs in a course
155+
python semantic_alignment.py --course 85427
156+
```
157+
158+
**How it works:**
159+
160+
1. Extracts text from each slide (PDF / PPTX / DOCX), including speaker notes
161+
2. For slides with fewer than 20 words, uses **GPT-4o-mini vision** to generate a text description of the visual content (result cached in `[slide].image_cache.json`)
162+
3. Embeds all slides with **all-mpnet-base-v2** (sentence-transformers, GPU) into a FAISS cosine-similarity index
163+
4. Queries the index for each transcript segment, using a ±30 s context window to pool nearby speech into richer queries
164+
5. Applies **Viterbi temporal smoothing** with a forward bias so slide assignments only move forward in time (with configurable backward-revisit cost)
165+
6. Flags segments that match no slide well as `off_slide`
166+
7. Collapses consecutive same-slide assignments into a compact timeline
167+
168+
**Caption↔slide file matching (auto-discovery):**
169+
When running `--course`, the aligner must decide which slide file(s) belong to each caption. It tries three strategies in order:
170+
171+
| Priority | Strategy | Example |
172+
|----------|-----------|---------|
173+
| 1 | **Lecture number** — regex extracts the number from both filenames and pairs equal numbers | `L02-foo.json` → `L02-Processes-Threads.pdf` |
174+
| 2 | **Token overlap** — Jaccard similarity on filename words (threshold > 0.05) | `Processes-Threads.json` → `L02-Processes-Threads.pdf` |
175+
| 3 | **Content embedding** — samples transcript text and slide text, embeds both with `all-mpnet-base-v2`, pairs the slide file with the highest cosine similarity (threshold ≥ 0.20) | `alpha_recording.json` → `L02-Processes-Threads.pdf` (sim=0.79) |
176+
177+
The content-based fallback means **files can be named anything** — if name matching fails, the aligner reads the actual content to find the right pair. A `[content-match]` line in the log indicates the fallback fired.
178+
179+
**Multi-file support:** When multiple slide files share the same lecture number (e.g. `L04-Part1.pdf` and `L04-Part2.pdf`), the aligner builds a single combined FAISS index, runs one Viterbi pass, then splits results back per file with local slide numbering. One JSON is saved per slide file (named `{slide_stem}.json`) so `note_generation.py` can locate them automatically.
180+
181+
**Output:** `[course_id]/alignment/[slide_stem].json`
182+
183+
---
184+
185+
### 4. Generate study notes — `note_generation.py`
186+
187+
Produces a single comprehensive Markdown note file covering all lectures in a course.
188+
189+
```bash
190+
python note_generation.py --course 85427 --course-name "CS3210 Parallel Computing"
191+
```
192+
193+
#### Key options
194+
195+
| Flag | Description |
196+
|------|-------------|
197+
| `--detail 0–10` | Note verbosity. `5` = medium (bullet points), `8` = detailed paragraphs (default `7`) |
198+
| `--lectures N-N` | Process a subset of lectures, e.g. `--lectures 1-3` or `--lectures 1,4,5` |
199+
| `--force` | Regenerate all sections even if cached |
200+
| `--merge-only` | Skip generation; re-run only the merge + image filter pass |
201+
| `--iterate` | Auto-increase detail level until self-score target is reached |
202+
203+
#### Architecture
204+
205+
**Section-by-section generation:**
206+
Each lecture is split into chunks of ~15 slides. Each chunk is sent to GPT in a separate API call and saved as an individual section file (`sections/L04_S02.md`). On re-runs, existing section files are reused unless `--force` is set.
207+
208+
**Multi-file lecture support:**
209+
Multiple slide files for the same lecture number (e.g. `L04-Part1.pdf` and `L04-Part2.pdf`) are grouped automatically. Their sections are written to separate files (`L04_S01.md`, `L04_F02_S01.md`) and merged under a single `## Lecture 4` heading with `### Part 1:` / `### Part 2:` sub-headings.
210+
211+
**Detail levels:**
212+
213+
| Range | Style |
214+
|-------|-------|
215+
| 0–2 | Bullet-point outline only |
216+
| 3–5 | Hierarchical bullets — main concept + indented sub-details |
217+
| 6–8 | Full paragraphs with examples and analogies |
218+
| 9–10 | Maximum detail including edge cases and cross-lecture links |
219+
220+
**Image injection:**
221+
Slide images are rendered to `notes/images/L{N}/` and injected into notes at relevant positions. For multi-file lectures, additional files render to `notes/images/L{N}_F{idx}/`.
222+
223+
**Image filter agent:**
224+
After all sections are merged, a vision-based agent (GPT-4o-mini) reviews every embedded image and removes slides that do not contain meaningful visual elements. The decision priority is:
225+
226+
1. Slide has a cached visual description from semantic alignment → **KEEP**
227+
2. Slide text matches a title/divider pattern → **REMOVE**
228+
3. All other slides → **vision API** with a structured prompt that keeps diagrams, flowcharts, architecture illustrations, graphs, and tables, and removes text-only or code-only slides
229+
230+
**Generator–verifier loop:**
231+
Each section draft is verified by `gpt-4.1-mini`, which checks technical term accuracy against the slide content. Suspicious verifier responses are discarded to prevent overwriting valid drafts.
232+
233+
**Exam notes:**
234+
A final `## Exam Notes` section is auto-generated summarising up to 30 key exam points across all lectures.
235+
236+
**Self-scoring:**
237+
After generation, the pipeline prints a heuristic self-score (word coverage, terminology hit-rate, callout density, code block count).
238+
239+
#### Output structure
240+
241+
```
242+
[course_id]/notes/
243+
├─ CourseName_notes.md Final merged note
244+
├─ CourseName_notes.score.json Self-score breakdown
245+
├─ sections/
246+
│ ├─ L01_S01.md Cached section files (single-file lectures)
247+
│ ├─ L04_F02_S01.md Cached section files (multi-file lecture, part 2)
248+
│ └─ exam_notes.md
249+
└─ images/
250+
├─ L01/ Rendered slide PNGs (single-file lectures)
251+
└─ L04_F02/ Rendered slide PNGs (multi-file lecture, part 2)
252+
```
253+
254+
---
255+
256+
## Incremental updates
257+
258+
| Scenario | Command |
259+
|----------|---------|
260+
| New lecture added | Normal run — new sections generated, existing ones cached |
261+
| New slides added to existing lecture | `--lectures N --force` to regenerate that lecture only |
262+
| New video/alignment added to existing lecture | `--lectures N --force` |
263+
| Only prompts or image filter changed | `--merge-only` to re-merge without regenerating sections |
264+
265+
---
266+
267+
## Utilities
268+
269+
### `alignment_parser.py`
270+
271+
Converts the full alignment JSON (~300 KB) into a compact per-slide representation (~30 KB) for token-efficient LLM prompting. Used internally by `note_generation.py`.
272+
273+
```bash
274+
python alignment_parser.py 85427/alignment/lecture.json
275+
python alignment_parser.py 85427/alignment/lecture.json --out compact.json
276+
```
277+
278+
---
279+
280+
## Hardware requirements
281+
282+
| Component | Requirement |
283+
|-----------|-------------|
284+
| GPU | CUDA-capable (RTX series recommended; tested on RTX 5070 Ti 16 GB) |
285+
| VRAM | ≥ 8 GB for Whisper large-v3 + sentence-transformers |
286+
| PyTorch | 2.10+ with CUDA 12.8 for Blackwell (sm_120) support |
287+
288+
---
289+
290+
## Notes on writing style
291+
292+
Notes are written in **Chinese** by default (as configured in `_SYSTEM` prompt), with English technical terms preserved inline (e.g. 进程 (Process)). The prompt enforces:
293+
294+
- No professor-centric narration ("老师说…", "教授指出…") — only first-person knowledge statements
295+
- LaTeX for formulas (`$...$` inline, `$$...$$` block)
296+
- Complete, compilable code examples with correct language tags
297+
- `> [!IMPORTANT]` callout blocks for exam-critical content

0 commit comments

Comments
 (0)