Repository navigation
Conversation
Reader / Shaper / Mapper / Classifier / Reconciler, each with a fixed data shape between them. Covers payment platforms, banks, securities, exchanges, on-chain sources, and PDF/EML/XML/JSON readers. Maps upstream issues to the layer that absorbs them and gives a migration order from the current dev-v3 templateRules path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
…ate reader block Readers produce one shape, a Table of string cells, and know nothing about amounts or transactions. csv/xlsx/xls move over unchanged (sheet selection and utf-16 added). json and xml share one XPath contract: records selects a node per row, columns map names to relative paths. text matches layout text with a named-group regexp plus a continuation regexp, and converts pdf through pdftotext -layout when asked. Templates may declare a reader: block; without one the legacy template.fileFormat/encoding/delimiter fields are translated, so all 50 template-repo pins produce byte-identical output before and after. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
A shape: list settles everything about rows that is not one bill line per transaction, before any rule runs: locateHeader finds the header by anchor column names instead of counting leading rows, dropMatching and dropIf remove preamble, totals and failed orders, merge folds split fills into one record, split fans a line carrying a fee into two. Without a locateHeader step the legacy skipLeadingRows/sourceHeaders/ headerLocate fields still decide the header, so a template can adopt a single dropIf without touching anything else. All 50 template-repo pins produce byte-identical output. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
Templates that declare slots fill only Beancount fields and metadata. Personal rules assign accounts, and missing sides fall back to FIXME. Co-authored-by: fatsheep <fatsheep2@users.noreply.github.com> (cherry picked from commit b69a3ab)
Slot mappings, personal rules, and engine FIXME sides each leave a FieldSource on the order: which rule, which columns, and which slot. Beancount output is unchanged, and templates without slots still use templateRules with empty Sources. Co-authored-by: fatsheep <fatsheep2@users.noreply.github.com> (cherry picked from commit 4411cdc)
…count bindings Builds on the slot contract: direction replaces amountSign with three explicit forms (one column's values, two amount columns, or the sign); template vars may carry conditions and may never be accounts; legs declare multi-leg shapes by role from a closed core set (from, to, cash, custody, position, fee, gas, pnl, plus x- custom roles) and a template leg naming an account is rejected. The rules file binds roles once under accounts: and overrides per rule with an accounts action; unbound core roles fall back to their FIXME account and are recorded as engine-filled. Also fixes two drifts on dev-v3 that securities templates depend on: format() now honours width and zero-pad flags (%06.0f), and arithmetic results take their scale from the operands instead of the expression text (0.85 + 0 prints 0.85, not 0.85000000). config init prefers the registry's starter rules over the generated skeleton. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
Readers: html shares the XPath records/columns contract over the DOM;
eml is a container that picks a MIME part (media type or
attachment:<glob>, nested multipart, base64 and quoted-printable) and
hands it to an inner reader; api fetches the bill from an endpoint where
{source} is the import argument and {env.NAME} an environment variable,
failing early when a variable is unset and never echoing query values
in errors.
Reconcile: template.balance.column asserts the running balance between
consecutive rows in either row order, so a misread layout-text row is an
error rather than a wrong entry. reconcile.dedupe drops transactions
repeated in the bill or already present in an existing ledger by exact
key, flags near matches within a window, and flagEngineFilled marks
transactions whose accounts the engine had to fill. import logs what the
stage did. All 50 template-repo pins are unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
… rules in slot mode self is the bill's own account and lands on from for an outflow and on to for an inflow, so a bank template's starter rules bind one role. direction gains invert (credit-card statements list charges positive) and default (a column value in neither list, such as 不计收支). With slots declared, templateRules still run but may only shape fields; accounts, postings and account-valued vars are rejected. Rule-added metadata keys keep the position the template declares. Two-column direction treats a zero beside a value as a placeholder, and the rule-warning scanner no longer reads comparison operators as columns. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
cmd/wasm-runtime exposes degRuntimeImport(template, rules, name, bytes) and returns the Beancount text plus transaction, FIXME and warning counts, so the template hub can run a statement locally. The rules-file merge moves into pkg/importer (ParseRulesFile/ApplyRulesFile) for the CLI and the browser to share, and the analyser interface and runtime analyser move to pkg/analyser/api so compilers no longer depend on every provider package. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
…ration and metadata filtering output.payee / output.narration rewrite every transaction after the template mapping and before personal rules; output.metadata.drop or keep filters metadata last, so rule-added keys are covered. The CLI's private rules-file struct is replaced by importer.RulesFile so the CLI, the browser build and the template hub read one schema. config init documents the block in the generated file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
…; stable ordering Four generic primitives from the provider audit, each in one layer: - shape.capture lifts statement-level values (account number, card alias, statement month) from rows outside the table into <file.x>, visible to shape conditions, slots, vars and rules. - shape.failIf stops the import on a matching record, with a message and the row, so a changed export fails loudly instead of writing entries. - template.timezone pins wall-clock dates (and .timestamp) to an IANA zone instead of the machine's. - role other is the counterparty side, to on an outflow and from on an inflow, so one rule covers a purchase and its refund. Rules may also set direction: outflow|inflow when only the user knows, such as which wallet address is theirs. Ordering is now deterministic: compilers sort with a stable sort and a newest-first statement is reversed first, so same-day entries keep the order they happened in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
…=~ operators, and the admission criteria for new syntax Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
…les file; keep empty captures Template rules that arrive through a rules file are now checked against the slot contract at import, so postings or from/to there fail loudly instead of being silently ignored. A capture group that matches an empty value (a blank card alias) counts as found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
…te writes an operator; shape conditions see cleaned cells A var copied from a bill column (a 12/29 date, a 2024-01 code) stays text; only expressions whose template text has an operator are evaluated. Shape conditions now see cells after the same cleanup rows get (BOM, ="…" wrappers), so failIf/dropIf match what rules match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
v3 importer: five-layer pipeline (reader, shape, slots, roles, reconcile)
The compilers now sort with a stable sort, so entries sharing a timestamp keep the order the provider emitted them. The two expected files differ only in that order (sorted contents are identical); make test is 37/37 again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
Drop importer helpers left unused after the slot contract, check Close/ Remove/WriteFile errors, lowercase error strings, and take the tagged switches and ReplaceAll calls staticcheck asks for. No behaviour change; make test stays 37/37. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
…n't deadlock degRuntimeImport now returns a Promise and does the work on a goroutine. Templates with timezone: call time.LoadLocation, which probes the zoneinfo directory through syscall/js fs; that callback can't fire while the synchronous export holds the event loop, so every pinned template hung the page. time/tzdata is embedded since the browser has no zoneinfo. Checked with a 1,481-row Alipay export: the wasm output is byte-identical to the CLI's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
fix: make dev-v3 green and stop the wasm runtime deadlocking on timezone templates
…L-3.0 xlsReader
github.com/shakinm/xlsReader is GPL-3.0, so any binary that links the
importer (the CLI, the wasm runtime, an app embedding the engine) is
bound by it. The .xls path now uses a copy of extrame/xls (Apache-2.0)
under pkg/reader/internal/xls with two fixes: numeric cells with a
built-in number format no longer panic, and non-date custom formats
("#,##0.00") no longer render amounts as dates. Rows keep the cells the
file defines, so the ccb and spdb_debit providers read through the same
function.
All 15 sample .xls bills (template repo + examples) produce the same
rows as before, blank rows aside; make test 37/37, template verify
78/78, golangci-lint clean.
Refs fatsheep/mirato#431
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
pkg/embed runs one import in memory and returns a versioned result (contract 1): compiled Beancount text plus structured transactions - date, payee, narration, ordered metadata, parsed postings (account, amount, currency, cost, price), a from/to summary for plain transfers, and per-slot sources (template / rule id / engine). cmd/libdeg exports it over C for apps that embed DEG (Mirato via Rust FFI): deg_import returns that JSON, deg_free releases it, deg_contract_version reports the contract. Calls are thread-safe: memory output is keyed per call and compilation is serialised because the beancount compiler keeps its templates in package variables. The browser runtime now goes through the same function. test/libdeg/smoke.sh builds the c-archive, links a C program against it and runs an import; CI runs it via make test-libdeg. Refs fatsheep/mirato#431 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
Embed DEG in apps: Apache-licensed .xls reader and a C library (libdeg)
Templates on deg-provider-template are read by this engine. Each pull request and push to dev-v3 builds DEG from the change and runs the template repo's verify script, so a DEG change cannot quietly break a template. The template repo already verifies against DEG dev-v3, so both sides check each other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
ci: verify every provider template on each DEG change
Brings in upstream's CIB debit and IBKR providers, the new Alipay format, the keep-refunds switch, Go 1.25 and the dependency bumps. - go.mod/go.sum: newer versions from both sides; blackfriday/v2 stays for cobra 1.10. - validator: upstream's multi-file check, lower-case error strings as lint wants. - cib_debit: reads .xls through pkg/reader like the other providers, so the GPL xlsReader dependency stays out. - ExactMoney gate: cib_debit and ibkr registered as not yet migrated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
Formatting only; no logic changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
excelize 2.11 (from upstream's bump) applies custom number formats, so a stored 0.85 in a "0.000" cell read as "0.850" and the htsec template's commissions gained a decimal place. When the shown text is a plain decimal of the same value as the stored one, its trailing fractional zeros are dropped; dates, percentages, thousands separators and text cells keep their shown text. Found by the provider-template check on the upstream merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
Merge upstream master into dev-v3; keep the GPL xls reader out
The built-in provider removes a matched full refund and its original order; the template provider works row by row, so both stay, as if keepRefundRecords were true. Balances are the same; to drop them, ignore both sides in the rules file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
docs(alipay): the template provider always keeps refund records
Upstream dev-v3 already lists them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JpABFTvN18pARUNZFqCHis
Merge upstream dev-v3 into dev-v3
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
把 fork 上 dev-v3 的工作合回上游 dev-v3。设计见
doc/rfc-v3-importer-architecture.md：五层流水线 reader → shape → slots → roles → reconcile。pkg/reader：csv / xls / xlsx / json / xml / html / eml / api / 纯文本统一成表格。xls 改用打过补丁的 Apache-2.0 版extrame/xls，去掉 GPL-3.0 的xlsReader（另有单独 PR 提交给 master）。xlsx 按显示格式读出的数字不再补零：excelize 2.11 起会把0.85读成0.850。pkg/importer：shape：locateHeader按表头名定位、dropIf等，在规则之前整理表格；slots：把账单映射到 Beancount 的日期、付款方、说明、金额、元数据；direction：按列值、两列或金额正负判断收支，规则也能改方向（例如支付宝「不计收支」的退款）；Motivation and Context
Dependencies
github.com/shakinm/xlsReader（GPL-3.0）github.com/antchfx/htmlquery、jsonquery、xmlquery、xpath（html / json / xml 读取，MIT）Type of change
现有 provider 和
translate用法不变。与 master 相比，现有 provider 的输出只有一处差异：mt、oklink 中同一时间的交易顺序不同（稳定排序导致，排序后内容一致）。How has this been tested?
go test ./...通过make test-providers41/41 通过make test-libdeg通过（C API 跑一次导入）GOOS=js GOARCH=wasm go build ./cmd/wasm-runtime通过Verify provider templates）复现：
make test && make test-libdeg；模板回归见.github/workflows/templates.yml。Is this change properly documented?
doc/rfc-v3-importer-architecture.md：v3 架构 RFCdocs/docs/providers/template.md：模板语法，包括 capture、failIf、timezone、other / direction、^=$==~运算符，以及新增语法的准入标准docs/docs/providers/payment/alipay.md（中英文）：支付宝退款在模板下的行为。Go provider 默认成对移除「原交易 + 全额退款」，v3 模板逐行处理、两笔都保留（合计为零）