diff --git a/.gitignore b/.gitignore index dd3363b..669b3e3 100644 --- a/.gitignore +++ b/.gitignore @@ -21,3 +21,6 @@ docs/*.zip *.obj *.exe __pycache__/ + +# macOS +.DS_Store diff --git a/LICENSING.md b/LICENSING.md index 152c36c..69abc44 100644 --- a/LICENSING.md +++ b/LICENSING.md @@ -29,11 +29,29 @@ the copyright. | `doubletalk/nvda/**` | NVDA add-on driver, manifest, build script (our code) | BSD-3-Clause | David Sexton | | `doubletalk/mame/i86.*`, `i186.*`, `i86inline.h` | Vendored MAME 8086/80186 + 80C188EB CPU core | BSD-3-Clause | Carl; **Christopher Toth** (I80C188EB / EB Peripheral Control Block additions) | | `doubletalk/mame/endianness.h` | Vendored MAME utility | BSD-3-Clause | Aaron Giles, Vas Crabb | +| `doubletalk/rcdict/rcdict*.{c,h}`, `example.dict` | Pronunciation-dictionary layer, shared verbatim with its upstream project | BSD-3-Clause | see the file headers | +| `doubletalk/rcdict/remimu.h` | Vendored single-header regex engine | **CC0 / public domain** | wareya | | `docs/**`, `notes/**`, `*.md` | Our documentation and research notes | BSD-3-Clause | David Sexton (except `notes/investigation-audio-path.md`: **Christopher Toth**) | | `doubletalkpc.bin` (the ROM) | DoubleTalk PC firmware | **Proprietary — not included** | RC Systems, Inc. | Every source file carries a matching `license:` / `copyright-holders:` header -(MAME-style `// license:` for C/C++, `# license:` for scripts/manifests). +(MAME-style `// license:` for C/C++, `# license:` for scripts/manifests); the +`doubletalk/rcdict/**` files use SPDX identifiers instead, because they are +shared byte-for-byte with another repository and must read the same in both. + +### About `doubletalk/rcdict/**` + +These files are mirrored from their upstream project and must not be edited +here — a sync check there fails if the two copies differ. They are BSD-3-Clause +and depend on nothing beyond libc, which is exactly what lets them serve both +this project and a GPL-3 one. **The sharing only works in that direction:** +BSD-3 code can be linked into a GPL-3 program, so nothing under `rcdict/` may +ever acquire a GPL header, or a dependency on anything outside this tree. + +`remimu.h` is a third-party regex engine released under CC0, i.e. placed in the +public domain. CC0 imposes no conditions, so it adds nothing to what a +redistributor of this project has to do; it is listed for completeness and +because attribution is polite rather than required. ## Original project code — BSD-3-Clause diff --git a/doubletalk/Makefile b/doubletalk/Makefile index 4b9d1e1..1415116 100644 --- a/doubletalk/Makefile +++ b/doubletalk/Makefile @@ -3,9 +3,15 @@ # Standalone DoubleTalk PC emulator (vendored MAME 80C188EB core + shim) CXX ?= g++ CXXFLAGS ?= -O2 -g -CXXFLAGS += -std=c++20 -Ishim -Imame -Wall -Wno-unused-variable +CXXFLAGS += -std=c++20 -Ishim -Imame -I. -Wall -Wno-unused-variable +CC ?= cc +CFLAGS ?= -O2 -g -OBJS = build/emu.o build/i86.o build/i186.o build/doubletalk_board.o build/dtalk.o +# rcdict is mirrored verbatim from its upstream project (BSD-3-Clause, libc +# only); it is what makes dtalk_say apply a pronunciation dictionary. Do not +# edit it here - the sync runs upstream, and its check belongs in CI. +OBJS = build/emu.o build/i86.o build/i186.o build/doubletalk_board.o build/dtalk.o \ + build/rcdict.o build/rcdict_regex.o all: build/dtalk_cli build/libdtalk.a @@ -27,6 +33,16 @@ build/doubletalk_board.o: doubletalk_board.cpp doubletalk_board.h shim/emu.h | b build/dtalk.o: dtalk.cpp dtalk.h doubletalk_board.h shim/emu.h | build $(CXX) $(CXXFLAGS) -c $< -o $@ +# rcdict is C, not C++, and builds with nothing but its own directory on the +# include path -- that self-containment is the whole point of it. +build/rcdict.o: rcdict/rcdict.c rcdict/rcdict.h | build + $(CC) $(CFLAGS) -std=c99 -Wall -Wextra -Ircdict -c $< -o $@ + +# remimu.h is vendored (CC0) and does not compile clean under -Wall; the noise +# is all in its disabled debug paths, so quiet just this object. +build/rcdict_regex.o: rcdict/rcdict_regex.c rcdict/remimu.h | build + $(CC) $(CFLAGS) -std=c99 -w -Ircdict -c $< -o $@ + build/libdtalk.a: $(OBJS) | build ar rcs $@ $(OBJS) @@ -52,23 +68,44 @@ clean: # rather than probing which alternative the builder happens to have selected. # This project's own code uses no threads; the dependency comes entirely # from libstdc++'s posix build. -MINGW_FLAGS = -std=c++20 -O2 -Ishim -Imame -Wall -DDTALK_DLL -DDTALK_BUILD \ +MINGW_FLAGS = -std=c++20 -O2 -Ishim -Imame -I. -Wall -DDTALK_DLL -DDTALK_BUILD \ -static -static-libgcc -static-libstdc++ WIN_SRCS = shim/emu.cpp mame/i86.cpp mame/i186.cpp doubletalk_board.cpp dtalk.cpp -WIN_DEPS = shim/emu.h dtalk.h doubletalk_board.h $(WIN_SRCS) +WIN_DEPS = shim/emu.h dtalk.h doubletalk_board.h rcdict/rcdict.h $(WIN_SRCS) + +# rcdict is C and has to be COMPILED as C: handing rcdict.c to g++ compiles it +# as C++, where its designated initializers and implicit void* conversions are +# errors. So each DLL is two compilers' output linked together - the C++ +# emulator, and rcdict built by the matching gcc. Warnings are off for +# rcdict_regex.c alone, because vendored remimu.h is not clean under -Wall. +# +# Leaving rcdict out of these two rules was a silent failure worth naming: the +# DLL linked, dtalk_set_dictionary was exported, and the driver simply had no +# way to build a dictionary to hand it. +MINGW_CFLAGS = -std=c99 -O2 -Ircdict win32: build/win32/dtalk.dll win64: build/win64/dtalk64.dll windows: win32 win64 -build/win32/dtalk.dll: $(WIN_DEPS) +build/win32/dtalk.dll: $(WIN_DEPS) rcdict/rcdict.c rcdict/rcdict_regex.c mkdir -p build/win32 - i686-w64-mingw32-g++ $(MINGW_FLAGS) -shared \ - -o $@ $(WIN_SRCS) -Wl,--out-implib,build/win32/dtalk.lib - -build/win64/dtalk64.dll: $(WIN_DEPS) + i686-w64-mingw32-gcc $(MINGW_CFLAGS) -Wall -Wextra -c rcdict/rcdict.c \ + -o build/win32/rcdict.o + i686-w64-mingw32-gcc $(MINGW_CFLAGS) -w -c rcdict/rcdict_regex.c \ + -o build/win32/rcdict_regex.o + i686-w64-mingw32-g++ $(MINGW_FLAGS) -shared -o $@ $(WIN_SRCS) \ + build/win32/rcdict.o build/win32/rcdict_regex.o \ + -Wl,--out-implib,build/win32/dtalk.lib + +build/win64/dtalk64.dll: $(WIN_DEPS) rcdict/rcdict.c rcdict/rcdict_regex.c mkdir -p build/win64 - x86_64-w64-mingw32-g++ $(MINGW_FLAGS) -shared \ - -o $@ $(WIN_SRCS) -Wl,--out-implib,build/win64/dtalk64.lib + x86_64-w64-mingw32-gcc $(MINGW_CFLAGS) -Wall -Wextra -c rcdict/rcdict.c \ + -o build/win64/rcdict.o + x86_64-w64-mingw32-gcc $(MINGW_CFLAGS) -w -c rcdict/rcdict_regex.c \ + -o build/win64/rcdict_regex.o + x86_64-w64-mingw32-g++ $(MINGW_FLAGS) -shared -o $@ $(WIN_SRCS) \ + build/win64/rcdict.o build/win64/rcdict_regex.o \ + -Wl,--out-implib,build/win64/dtalk64.lib .PHONY: all clean win32 win64 windows diff --git a/doubletalk/dtalk.cpp b/doubletalk/dtalk.cpp index b04428f..d3ebd33 100644 --- a/doubletalk/dtalk.cpp +++ b/doubletalk/dtalk.cpp @@ -6,6 +6,7 @@ #include "dtalk.h" #include "doubletalk_board.h" +#include "rcdict/rcdict.h" #include #include @@ -224,6 +225,7 @@ struct dtalk u64 samples_dropped = 0; // grid samples pulled/carried but never delivered (dtalk_stop) s64 idle_stable = 0; // consecutive idle cycles observed int rate_boost = 0; // current boost level (0 = authentic) + rcdict_options dict{}; // pronunciation rules; profile always set u16 rate_orig[RATE_REGION_HI - RATE_REGION_LO + 1]; // pristine period words bool rate_saved = false; @@ -352,6 +354,7 @@ extern "C" { dtalk *dtalk_create(const void *rom, size_t rom_size) { dtalk *dt = new dtalk; + rcdict_options_init(&dt->dict, &rcdict_doubletalk_pc); if (!dt->board.load_rom(static_cast(rom), rom_size)) { delete dt; @@ -417,10 +420,75 @@ void dtalk_queue(dtalk *dt, const void *bytes, size_t len) dt->idle_stable = 0; } +/* --- pronunciation dictionary --------------------------------------------- + * + * rcdict/ is mirrored verbatim from upstream and must not be edited here; it + * is BSD-3-Clause and depends on nothing but libc, which is what lets the same + * sources serve more than one engine. All that happens here is that dtalk_say + * runs its text through the rules first. + * + * The RC8650 datasheet's Table 5 is captioned "DoubleTalk Phoneme Symbols", so + * the phoneme set, the Table 6 modifiers, the D/T/C mode commands, the Ctrl-A + * command character and the nI index markers are the same on this card; what + * differs is carried by rcdict_doubletalk_pc. + */ + +void dtalk_set_dictionary(dtalk *dt, const rcdict *d, int inline_phonemes) +{ + if (!dt) return; + dt->dict.dict = d; // borrowed; the caller keeps ownership + dt->dict.inline_phonemes = inline_phonemes ? 1 : 0; +} + +size_t dtalk_expand(dtalk *dt, const char *in, size_t inlen, + char *out, size_t outcap) +{ + if (!dt || !in) return 0; + return rcdict_expand(&dt->dict, in, inlen, out, outcap, nullptr, nullptr); +} + +/* Construction, re-exported for the DLL. See the comment in dtalk.h for why + * these exist at all -- nothing here is more than a forwarding call. */ + +rcdict *dtalk_dict_new(void) +{ + return rcdict_new(&rcdict_doubletalk_pc); +} + +void dtalk_dict_free(rcdict *d) +{ + rcdict_free(d); +} + +int dtalk_dict_add_file(rcdict *d, const char *path) +{ + if (!d || !path) return 0; + return rcdict_add_file(d, path, nullptr, nullptr); +} + +size_t dtalk_dict_rule_count(const rcdict *d) +{ + return d ? rcdict_rule_count(d) : 0; +} + void dtalk_say(dtalk *dt, const char *text) { - dtalk_queue(dt, text, std::strlen(text)); + const size_t len = std::strlen(text); const u8 cr = 0x0d; + + // No dictionary and no escape is the common case, and stays allocation + // free. + if (!dt->dict.dict && !dt->dict.inline_phonemes) { + dtalk_queue(dt, text, len); + dtalk_queue(dt, &cr, 1); + return; + } + + const size_t need = rcdict_expand(&dt->dict, text, len, nullptr, 0, + nullptr, nullptr); + std::vector buf(need + 1); + rcdict_expand(&dt->dict, text, len, buf.data(), need + 1, nullptr, nullptr); + dtalk_queue(dt, buf.data(), need); dtalk_queue(dt, &cr, 1); } diff --git a/doubletalk/dtalk.h b/doubletalk/dtalk.h index a18bef2..2f0c7c9 100644 --- a/doubletalk/dtalk.h +++ b/doubletalk/dtalk.h @@ -78,9 +78,67 @@ DTALK_API uint8_t dtalk_lpc_status(dtalk *dt); * card RDY-gated as it accepts them while dtalk_synth() runs. */ DTALK_API void dtalk_queue(dtalk *dt, const void *bytes, size_t len); -/* Convenience: queue text followed by CR. */ +/* Convenience: queue text followed by CR. The pronunciation dictionary, if + * one is set, is applied first. */ DTALK_API void dtalk_say(dtalk *dt, const char *text); +/* --- pronunciation dictionary -------------------------------------------- */ + +/* Substitute pronunciations into text on its way to the card: respellings, + * phonemes through the card's own phoneme mode (Ctrl-A D), or embedded + * commands. The rules, the file format and the loaders belong to rcdict, which + * is mirrored verbatim from upstream -- include rcdict/rcdict.h to build one. + * All of it is optional and off until asked for. + * + * The dictionary is BORROWED: it must outlive the instance, or be replaced + * with NULL before it is freed. inline_phonemes switches on the "[[K AE T]]" + * escape in ordinary text, which is off by default because "[[" is wiki link + * syntax and a user reading a wiki must not lose text to it. */ +struct rcdict; +DTALK_API void dtalk_set_dictionary(dtalk *dt, const struct rcdict *d, + int inline_phonemes); + +/* Run the dictionary over text WITHOUT queueing it, returning the number of + * bytes the result needs (not counting a terminating NUL); a value >= outcap + * means it was truncated and the call should be repeated bigger. + * + * For callers that split long text into utterances themselves: one word can + * expand into forty characters of phonemes plus the mode switches, so + * splitting first and expanding after can push a piece past what the card will + * take. Expand, then split. */ +DTALK_API size_t dtalk_expand(dtalk *dt, const char *in, size_t inlen, + char *out, size_t outcap); + +/* Building a dictionary, for callers that cannot reach rcdict's own symbols. + * + * A C caller that links libdtalk.a should just include rcdict/rcdict.h and use + * rcdict_new/rcdict_add_file directly -- these add nothing. They exist for the + * DLL: dtalk.h marks its entry points __declspec(dllexport), which turns off + * mingw's export-everything default, so rcdict's own symbols are not in the + * DLL's export table and a ctypes caller cannot see them. Re-exporting the + * four calls a host actually needs is a great deal less crude than exporting + * every symbol in the image, and it keeps the export marker out of rcdict -- + * which has to stay free of anything Windows- or project-specific if it is to + * go on being shared verbatim with its other host. + * + * dtalk_dict_new also settles the profile (rcdict_doubletalk_pc) here rather + * than making the caller name it. That is the one piece of the construction a + * host has no business choosing. + * + * The result is BORROWED by dtalk_set_dictionary: detach it with NULL before + * dtalk_dict_free. */ +DTALK_API struct rcdict *dtalk_dict_new(void); +DTALK_API void dtalk_dict_free(struct rcdict *d); + +/* Append one file's rules, returning how many were added (0 on any failure: + * unreadable file, or a file whose every line was rejected). Rules are + * first-match-wins in load order across all files, so a later file can only + * add -- to override an earlier one it has to be loaded first. */ +DTALK_API int dtalk_dict_add_file(struct rcdict *d, const char *path); + +/* Total rules held, across every file added. */ +DTALK_API size_t dtalk_dict_rule_count(const struct rcdict *d); + /* Immediate stop (Ctrl-X / DTLK_CLEAR, written un-gated per the manual): * drops the host-side queue, stops speech, flushes the card's buffer, and * discards pending audio and index marks. Synthesizer settings persist. */ diff --git a/doubletalk/nvda/README.md b/doubletalk/nvda/README.md index 41816cf..d120cdc 100644 --- a/doubletalk/nvda/README.md +++ b/doubletalk/nvda/README.md @@ -57,3 +57,72 @@ Index commands become Ctrl-A nI markers (rolling 0–99 mapped back to NVDA's index values); the emulator reports each marker with its exact output-sample position, which the driver converts to `synthIndexReached` notifications as playback passes it. Cancel writes the card's own Ctrl-X clear command. + +## Pronunciation dictionaries + +The driver applies `rcdict`, a pronunciation layer shared byte-for-byte with +another engine: respellings, phonemes through the card's own phoneme mode (Ctrl-A +D), or embedded commands, matched by word, substring or regular expression. +The format is documented in `../rcdict/rcdict.h`, with a worked example in +`../rcdict/example.dict`. + +**[The dictionary format is documented in `DICTIONARY-GUIDE.md`](../rcdict/DICTIONARY-GUIDE.md)** - a self-contained guide written for users rather than for this repository, and the thing to hand to anyone who asks how to write one. + +Dictionaries are `.dict` files. They are arranged from **NVDA menu > +Preferences > Settings > DoubleTalk PC dictionaries**, which has a list of the +files in load order and five buttons: + +| Button | Key | What it does | +|---|---|---| +| Add... | `Alt+D` | Adds a `.dict` file to the list, wherever it is | +| Remove | `Alt+R` | Takes it out of the list | +| Move up | `Alt+U` | Raises its priority | +| Move down | `Alt+N` | Lowers it | +| Reload dictionaries | `Alt+L` | Re-reads the files, so editing one does not mean restarting the synthesizer | + +`Alt+F` moves to the list itself. + +**Files are read where they are.** Add records where a file is; it does not copy +it. So the dictionary you edit is the one the synthesizer reads: save it in your +editor, press Reload, and the change is live. Keep them wherever suits - a +folder of your own, a synced drive, a checkout under version control. + +A reference to a file that is not there right now (an unplugged drive, a share +not mounted yet) stays in the list, marked *not found*, and is skipped until it +comes back. Removing it is your decision, not the add-on's. + +**Order is the whole point of the list.** Rules are tried from the top down and +the first match wins, so a file higher in the list overrides one below it - a +later file can only *add*. The order lives in NVDA's configuration, not in the +filenames. + +**There is also a folder, for anyone who just wants somewhere to put one.** +`%APPDATA%\nvda\doubletalkpc\` is scanned as well: a `.dict` file dropped in +there works with no configuration at all, appended after everything the list +names, in sorted order. It is how dictionaries worked before there was a panel +and it still works that way - the panel is how you take control of what beats +what. Its files show in the list under their bare names, and because the folder +is scanned rather than listed, Remove on one of them has to delete it; the panel +says so and asks first. Remove on anything else only unlists it. + +The list is applied when the Settings dialog is closed with OK or Apply, or +immediately by the Reload button. + +The substitution is made before the utterance is queued, never inside +`dtalk_say` - this driver queues its own bytes with `dtalk_queue`, so anything +applied further down would never fire for NVDA. + +## Testing + + make -C ../.. # the tests build a shared libdtalk from build/*.o + tests/run.sh + +`tests/` stubs NVDA's API (`nvdastub.py`), stages a copy of the driver package +with the host's shared library standing in for `dtalk64.dll`, and drives the +real `speak()`. It covers the dictionary path: that files are found and loaded +in the right order, that a substitution reaches the queued bytes, that the +Ctrl-A prefix and any index markers come through untouched, that the utterance +still ends in the CR without which the card says nothing at all, and - the +check the others rest on - that the card really renders different audio with a +dictionary loaded than without. `test_dictfiles.py` covers the load-order rules +on their own, with no emulator involved. diff --git a/doubletalk/nvda/build_addon.sh b/doubletalk/nvda/build_addon.sh index 99f7405..ea9f291 100755 --- a/doubletalk/nvda/build_addon.sh +++ b/doubletalk/nvda/build_addon.sh @@ -14,18 +14,40 @@ cd "$(dirname "${BASH_SOURCE[0]}")" WITH_ROM=0 [[ "${1:-}" == "--with-rom" ]] && WITH_ROM=1 +# Refresh the DLLs from the build tree if they differ from what is here. +# Copying them in by hand is easy to forget, and forgetting it produces the +# worst kind of bug: a package that builds, installs and runs, but ships the +# previous library. That is not hypothetical - it happened during the +# dictionary work, where the symptom was a driver that simply never applied a +# dictionary. If the build tree has no copy, whatever is here already is used. +for pair in "build/win32/dtalk.dll:dtalk.dll" "build/win64/dtalk64.dll:dtalk64.dll"; do + src="../${pair%%:*}" + dst="synthDrivers/doubletalkpc/${pair##*:}" + if [[ -f "$src" ]] && ! cmp -s "$src" "$dst"; then + cp "$src" "$dst" + echo "refreshed $dst from $src" + fi +done + for f in synthDrivers/doubletalkpc/dtalk.dll synthDrivers/doubletalkpc/dtalk64.dll; do - [[ -f "$f" ]] || { echo "missing $f - see README.md" >&2; exit 1; } + [[ -f "$f" ]] || { echo "missing $f - run 'make -C .. windows' first, or see README.md" >&2; exit 1; } done cp ../NOTICE synthDrivers/doubletalkpc/NOTICE.txt +# globalPlugins/ carries the dictionaries settings category. It has to be a +# global plugin rather than part of the driver - a driver module comes and goes +# with the synthesizer, and a settings category cannot - so it is a second +# directory in the package and easy to leave out of this line. rm -f doubletalkpc.nvda-addon if [[ $WITH_ROM == 1 ]]; then [[ -f synthDrivers/doubletalkpc/doubletalkpc.bin ]] || { echo "missing ROM for --with-rom build" >&2; exit 1; } - zip -r -q doubletalkpc.nvda-addon manifest.ini synthDrivers + zip -r -q doubletalkpc.nvda-addon manifest.ini synthDrivers globalPlugins \ + -x "*/__pycache__/*" echo "wrote doubletalkpc.nvda-addon (PRIVATE build - bundles the proprietary ROM, do not distribute)" else - zip -r -q doubletalkpc.nvda-addon manifest.ini synthDrivers -x "synthDrivers/doubletalkpc/doubletalkpc.bin" + zip -r -q doubletalkpc.nvda-addon manifest.ini synthDrivers globalPlugins \ + -x "synthDrivers/doubletalkpc/doubletalkpc.bin" -x "*/__pycache__/*" echo "wrote doubletalkpc.nvda-addon (public-safe, no ROM - user must supply doubletalkpc.bin)" fi +unzip -l doubletalkpc.nvda-addon diff --git a/doubletalk/nvda/globalPlugins/doubletalkpcDictionaries.py b/doubletalk/nvda/globalPlugins/doubletalkpcDictionaries.py new file mode 100644 index 0000000..59430cf --- /dev/null +++ b/doubletalk/nvda/globalPlugins/doubletalkpcDictionaries.py @@ -0,0 +1,359 @@ +# license: BSD-3-Clause +# copyright-holders: David Sexton +# (NVDA's GPL-2 license includes an explicit exception permitting non-GPL +# drivers and plugins; BSD-3 is additionally GPL-compatible regardless.) +# +# "DoubleTalk PC dictionaries" in NVDA's Settings dialog: which pronunciation +# dictionary files load, in what order, and a way to reload them without +# restarting the synthesizer. +# +# Why a global plugin and not the synth driver. A driver module is loaded when +# its synthesizer is selected and unloaded when it is not, so a settings +# category registered there would appear and disappear with the synthesizer - +# and worse, could be pulled out from under an open Settings dialog. Global +# plugins live for the whole session, which is what a settings category needs. +# +# What the panel is actually for. rcdict matches first-match-wins in load order, +# so a rule in a later file cannot override an earlier one - it can only add. +# The single thing a user needs control of, then, is the ORDER, and there is no +# way to express it in a folder of files except by naming them so they sort the +# way you want. Hence a list with Move up / Move down, and hence the order +# living in NVDA's configuration rather than in the filenames. +# +# Add does not copy. It records where the file is and leaves it there, so the +# file the user edits is the file the synthesizer reads - press Reload and the +# edit is live. Copying would have made every dictionary two files, one of them +# quietly stale, and "why did my change do nothing" the commonest question about +# this panel. Remove is the mirror of that: a reference is dropped from the +# list, and only a file in the add-on's own folder is deleted, because that +# folder is scanned and unlisting a file in it would not stick. +# +# Everything here is keyboard-first: the buttons keep the focus when they act +# (so Move up can be pressed four times without chasing the selection around), +# and anything that happens without focus moving is spoken with ui.message, +# because a wx selection changed in code raises no event for NVDA to announce. + +import os + +import wx + +import globalPluginHandler +import globalVars +import gui +import synthDriverHandler +import ui +from gui import guiHelper +from gui.settingsDialogs import NVDASettingsDialog, SettingsPanel +from logHandler import log + +# The driver package owns the rule about which files load and in what order, +# because the driver has to work whether or not this plugin does - and never +# the other way round. Caught rather than allowed to propagate so that a +# half-installed add-on costs a log line and this one category, instead of +# taking the whole global plugin down with a traceback at import time. +try: + from synthDrivers.doubletalkpc import dictfiles +except ImportError: + dictfiles = None + +#: The synthesizer these dictionaries belong to (SynthDriver.name). +SYNTH_NAME = "doubletalkpc" + +#: What NVDA's own settings panels wrap their descriptions at. +PANEL_DESCRIPTION_WIDTH = 544 + + +def _count(n, singular, plural): + """'1 file' / '2 files', because "1 files" reads as a bug.""" + return "%d %s" % (n, singular if n == 1 else plural) + + +class DictionariesPanel(SettingsPanel): + # The name of this category in NVDA's Settings dialog. + title = "DoubleTalk PC dictionaries" + + panelDescription = ( + "Pronunciation dictionaries for the DoubleTalk PC (emulated) " + "synthesizer. Rules are tried from the top of the list downwards and " + "the first match wins, so a file higher in the list overrides one " + "below it. Files are read where they are, so you can edit one and " + "press Reload dictionaries." + ) + + def makeSettings(self, settingsSizer): + sHelper = guiHelper.BoxSizerHelper(self, sizer=settingsSizer) + + # The same sentence as panelDescription, which is only ever an + # accessible description; this is the one a sighted user reads. + description = sHelper.addItem( + wx.StaticText(self, label=self.panelDescription)) + description.Wrap(self.scaleSize(PANEL_DESCRIPTION_WIDTH)) + + #: The working order, as dictfiles entries. Held here rather than read + #: back out of the list control so that a move is a list operation and + #: not a screen-scrape - and because what the list SHOWS is a label, + #: which is not what goes in the configuration. + self._entries = dictfiles.orderedEntries() + + self._listBox = sHelper.addLabeledControl( + "Dictionary &files, in the order they are loaded", + wx.ListBox, + choices=self._labels(), + ) + self._listBox.Bind(wx.EVT_LISTBOX, self.onSelectionChanged) + + bHelper = guiHelper.ButtonHelper(wx.HORIZONTAL) + # Mnemonics chosen to clear the dialog's own OK / Cancel / Apply, which + # take O, C and A. + self._addButton = bHelper.addButton(self, label="A&dd...") + self._removeButton = bHelper.addButton(self, label="&Remove") + self._upButton = bHelper.addButton(self, label="Move &up") + self._downButton = bHelper.addButton(self, label="Move dow&n") + self._reloadButton = bHelper.addButton(self, label="Re&load dictionaries") + sHelper.addItem(bHelper) + + self._addButton.Bind(wx.EVT_BUTTON, self.onAdd) + self._removeButton.Bind(wx.EVT_BUTTON, self.onRemove) + self._upButton.Bind(wx.EVT_BUTTON, lambda evt: self._move(-1)) + self._downButton.Bind(wx.EVT_BUTTON, lambda evt: self._move(1)) + self._reloadButton.Bind(wx.EVT_BUTTON, self.onReload) + + # Add can reach a file anywhere, so this is no longer where dictionaries + # have to live - but it is still scanned, and it is still the answer for + # someone who just wants somewhere to put one. + sHelper.addItem(wx.StaticText( + self, + label="Files added here stay where they are. Anything placed in " + "%s is loaded as well." % dictfiles.folderPath(), + )) + + self._selectIndex(0) + + def onPanelActivated(self): + # The folder can have changed since the panel was built - a file dropped + # in by hand, or an editor saving a new one - and the panel may well + # have been built at the start of the session. Re-merge rather than + # reload, so an order the user has arranged but not yet applied is not + # thrown away by walking out of the category and back in. It also + # refreshes the "not found" marks, which is the other thing that can + # change under a panel left open. + # + # The selection is restored by ENTRY rather than by index, because the + # merge can insert above it. + index = self._listBox.GetSelection() + current = self._entries[index] \ + if 0 <= index < len(self._entries) else None + self._entries = dictfiles.mergeOrder(self._entries) + self._rebuild( + self._entries.index(current) if current in self._entries else 0) + super(DictionariesPanel, self).onPanelActivated() + + # --- the list ------------------------------------------------------------- + + def _labels(self): + return [dictfiles.label(e) for e in self._entries] + + def _find(self, entry): + """Where C{entry} is in the working list, or -1. + + Same file, not same spelling, and dictfiles is the one that decides + which is which - the panel must not grow a second opinion about it. + """ + wanted = dictfiles.key(dictfiles.normalize(entry)) + keys = [dictfiles.key(e) for e in self._entries] + try: + return keys.index(wanted) + except ValueError: + return -1 + + def _rebuild(self, select=0): + self._listBox.Set(self._labels()) + self._selectIndex(select) + + def _selectIndex(self, index): + if self._entries: + index = max(0, min(index, len(self._entries) - 1)) + self._listBox.SetSelection(index) + self._updateButtons() + + def _updateButtons(self): + index = self._listBox.GetSelection() + has = index != wx.NOT_FOUND + self._removeButton.Enable(has) + self._upButton.Enable(has and index > 0) + self._downButton.Enable(has and index < len(self._entries) - 1) + + def onSelectionChanged(self, evt): + self._updateButtons() + + def _move(self, delta): + index = self._listBox.GetSelection() + if index == wx.NOT_FOUND: + return + target = index + delta + if target < 0 or target >= len(self._entries): + return + self._entries[index], self._entries[target] = \ + self._entries[target], self._entries[index] + self._rebuild(target) + # Focus stays on the button so the move can be repeated, which means + # nothing announces the new position by itself. + ui.message("%s, %d of %d" + % (dictfiles.label(self._entries[target]), + target + 1, len(self._entries))) + + # --- adding and removing -------------------------------------------------- + + def onAdd(self, evt): + """Point the list at files, wherever they are. Nothing is copied. + + The dialog opens in the dictionary folder when there is one, because + that is where a user with no opinion about where to keep dictionaries + will have put them - but it is only a starting point, and anywhere is + as good. + """ + with wx.FileDialog( + self, + message="Select pronunciation dictionaries to add", + defaultDir=dictfiles.folderPath() + if os.path.isdir(dictfiles.folderPath()) else "", + wildcard="Pronunciation dictionaries (*%s)|*%s|All files (*.*)|*.*" + % (dictfiles.EXTENSION, dictfiles.EXTENSION), + style=wx.FD_OPEN | wx.FD_FILE_MUST_EXIST | wx.FD_MULTIPLE, + ) as dialog: + if dialog.ShowModal() != wx.ID_OK: + return + chosen = dialog.GetPaths() + + added = [] + already = [] + for source in chosen: + entry = dictfiles.normalize(source) + if not entry: + continue + if self._find(entry) != -1: + # Already listed, under whatever spelling. Adding it again would + # be a second entry for one file, and the second could never + # match anything the first had not already matched. + already.append(entry) + continue + self._entries.append(entry) + added.append(entry) + + if added: + self._rebuild(self._find(added[-1])) + ui.message("Added %s" + % _count(len(added), "dictionary", "dictionaries")) + elif already: + index = self._find(already[-1]) + self._rebuild(index) + ui.message("%s is already in the list" + % dictfiles.label(self._entries[index])) + + def onRemove(self, evt): + """Take the selected dictionary out of the list. + + For a file in the add-on's own folder that means deleting it, and there + is no way round that: the folder is scanned for anything the saved order + has not seen, so a file merely dropped from the list would be picked + straight back up. For a file anywhere else it means exactly what it + says, and nothing on disk is touched - which is why only the first case + asks. + """ + index = self._listBox.GetSelection() + if index == wx.NOT_FOUND: + return + entry = self._entries[index] + name = dictfiles.label(entry) + if dictfiles.inFolder(entry): + if gui.messageBox( + "Remove %s?\n\nThe file will be deleted from %s." + % (entry, dictfiles.folderPath()), + "Remove dictionary", + wx.YES_NO | wx.NO_DEFAULT | wx.ICON_WARNING, self, + ) != wx.YES: + return + try: + os.remove(dictfiles.fullPath(entry)) + except OSError: + log.exception("doubletalkpc: could not delete %s" % entry) + gui.messageBox("Could not delete %s." % entry, + "Error", wx.OK | wx.ICON_ERROR, self) + return + del self._entries[index] + self._rebuild(index) + ui.message("Removed %s" % name) + + # --- applying ------------------------------------------------------------- + + def onSave(self): + dictfiles.setOrder(self._entries) + self._reloadSynth() + + def onReload(self, evt): + """Re-read the files, so editing one does not mean restarting the synth. + + This is the button the whole no-copying arrangement is for: the file the + user just saved in their editor is the file the synthesizer reads, so + re-reading it is the entire round trip. + + The folder is re-scanned first, so this also picks up a file written + since the panel was opened; and the order is saved first, because a user + who presses this expects the arrangement in front of them to be the one + that takes effect, and applying a different order from the one shown + would be the worst of both. + """ + self._entries = dictfiles.mergeOrder(self._entries) + self._rebuild(self._listBox.GetSelection()) + dictfiles.setOrder(self._entries) + result = self._reloadSynth() + if result is None: + ui.message( + "Dictionaries will be loaded when the DoubleTalk PC " + "synthesizer is next started.") + return + rules, files = result + ui.message("Loaded %s from %s" + % (_count(rules, "rule", "rules"), + _count(files, "file", "files"))) + + def _reloadSynth(self): + """Ask the running synthesizer to re-read its dictionaries. + + Returns (rules, files) - (0, 0) is a perfectly good answer, meaning the + folder is empty - or None if this synthesizer is not the one speaking, + which is an ordinary thing for it not to be and no reason to complain: + the files are read when it next starts. + """ + synth = synthDriverHandler.getSynth() + if synth is None or getattr(synth, "name", None) != SYNTH_NAME: + return None + try: + return synth.reloadDictionaries() + except Exception: + log.exception("doubletalkpc: reloading dictionaries failed") + return None + + +class GlobalPlugin(globalPluginHandler.GlobalPlugin): + def __init__(self): + super(GlobalPlugin, self).__init__() + # On the secure desktop there is nothing to configure and nowhere to + # save it, and a file picker there is a hole rather than a feature. + if globalVars.appArgs.secure: + return + if dictfiles is None: + log.error("doubletalkpc: the synth driver's dictfiles module is " + "missing, so the dictionaries settings category is not " + "available. Reinstall the add-on.") + return + dictfiles.initConfig() + if DictionariesPanel not in NVDASettingsDialog.categoryClasses: + NVDASettingsDialog.categoryClasses.append(DictionariesPanel) + + def terminate(self): + try: + NVDASettingsDialog.categoryClasses.remove(DictionariesPanel) + except ValueError: + pass + super(GlobalPlugin, self).terminate() diff --git a/doubletalk/nvda/synthDrivers/doubletalkpc/__init__.py b/doubletalk/nvda/synthDrivers/doubletalkpc/__init__.py index 8824c4e..adabdbf 100644 --- a/doubletalk/nvda/synthDrivers/doubletalkpc/__init__.py +++ b/doubletalk/nvda/synthDrivers/doubletalkpc/__init__.py @@ -16,6 +16,11 @@ # distributed with the add-on). Supply your own dump; # verify CRC32 66685631 / SHA1 # bf7e78d6381c76d291ee069971873347a314ffff. +# +# Pronunciation dictionaries are read where the user keeps them - the paths in +# NVDA's configuration, plus whatever is in /doubletalkpc/ - and +# arranged from the "DoubleTalk PC dictionaries" category in NVDA's Settings. +# See dictfiles.py for the load-order rule and rcdict/rcdict.h for the format. import ctypes # Unmaps the DLL on terminate - ctypes itself never does, so without this each @@ -106,6 +111,36 @@ def __init__(self): self.lib.dtalk_read_index_marks.restype = ctypes.c_size_t self.lib.dtalk_read_index_marks.argtypes = [ ctypes.c_void_p, ctypes.POINTER(_DtalkIndexMark), ctypes.c_size_t] + # Pronunciation dictionary. The rules live in rcdict, a BSD-3 module + # mirrored verbatim from upstream, but its own symbols are not in the + # DLL's export table - dtalk.h marks its entry points dllexport, which + # turns off mingw's export-everything default - so the DLL re-exports + # the four construction calls as dtalk_dict_*. They also settle the + # profile, so there is nothing to get wrong from here. + # + # Guarded because a DLL from before this existed is a real possibility + # (an add-on packaged with a stale one; a hand-copied dtalk.dll), and + # looking up a missing export raises. Speech without dictionaries is a + # reasonable synth; no synth at all is not. + self.has_dictionary = False + try: + self.lib.dtalk_dict_new.restype = ctypes.c_void_p + self.lib.dtalk_dict_new.argtypes = [] + self.lib.dtalk_dict_free.argtypes = [ctypes.c_void_p] + self.lib.dtalk_dict_add_file.restype = ctypes.c_int + self.lib.dtalk_dict_add_file.argtypes = [ctypes.c_void_p, ctypes.c_char_p] + self.lib.dtalk_dict_rule_count.restype = ctypes.c_size_t + self.lib.dtalk_dict_rule_count.argtypes = [ctypes.c_void_p] + self.lib.dtalk_set_dictionary.argtypes = [ + ctypes.c_void_p, ctypes.c_void_p, ctypes.c_int] + self.lib.dtalk_expand.restype = ctypes.c_size_t + self.lib.dtalk_expand.argtypes = [ + ctypes.c_void_p, ctypes.c_char_p, ctypes.c_size_t, + ctypes.c_char_p, ctypes.c_size_t] + self.has_dictionary = True + except AttributeError: + log.warning("doubletalkpc: %s has no dictionary support; " + "pronunciation dictionaries will be ignored" % _dllName()) # Unload again if the ROM is missing or bad: this is the one failure # every user without a firmware dump hits, and it is retried on every @@ -307,6 +342,15 @@ def __init__(self): self._nextMark = 0 self._queue = queue.Queue() self._stopping = threading.Event() + # The pronunciation dictionary, borrowed by the card instance and freed + # in terminate(). None until something is loaded, which is the common + # case and the one that costs nothing. + self._dict = None + # The "[[K AE T]]" escape in ordinary text. Off, and it must stay off by + # default: "[[" is wiki link syntax, and a screen-reader user reading a + # wiki must not lose text to it. + self._inlinePhonemes = False + self._loadDictionaries() self._thread = threading.Thread(target=self._synthLoop, daemon=True) self._thread.start() @@ -325,8 +369,110 @@ def terminate(self): if not unload: log.warning("Synthesis thread still running; leaving dtalk DLL loaded") with self._libLock: + # The card instance borrows the dictionary, so detach it before it + # is freed. Skipped along with the unload when the synthesis thread + # outlived its join: it may still be inside the DLL, and a freed + # dictionary under a live handle is worse than a leaked one. + if self._dict and unload: + try: + self._dt.lib.dtalk_set_dictionary(self._dt.handle, None, 0) + self._dt.lib.dtalk_dict_free(self._dict) + except Exception: + log.exception("doubletalkpc: freeing the dictionary failed") + self._dict = None self._dt.close(unload=unload) + # --- pronunciation dictionaries ------------------------------------------ + + def _loadDictionaries(self): + """(Re)load the user's dictionary files into the card instance. + + Files and load order come from dictfiles; rules are first-match-wins in + that order, so the file at the top of the list wins. Format and a worked + example: rcdict/rcdict.h and rcdict/example.dict. + + Returns (rules, files) for a caller that wants to report what happened. + (0, 0) is a perfectly good answer and means there are none; None means + the attempt did not get far enough to say. Deliberately quiet either + way: having no dictionaries is the normal case, and every failure path + here leaves the synth speaking exactly as it did before dictionaries + existed. A dictionary must never be the reason a screen reader goes + silent. + """ + if not self._dt.has_dictionary: + return None + try: + from . import dictfiles + paths = dictfiles.orderedPaths() + except Exception: + log.exception("doubletalkpc: could not look for dictionaries") + return None + + lib = self._dt.lib + new = None + total = 0 + if paths or self._inlinePhonemes: + try: + new = lib.dtalk_dict_new() + for path in paths: + try: + # The card's own char* path, so the ANSI code page - the + # same encoding its fopen will decode it with. + total += lib.dtalk_dict_add_file(new, path.encode("mbcs")) + except Exception: + log.exception("doubletalkpc: could not load %s" % path) + except Exception: + log.exception("doubletalkpc: building the dictionary failed") + new = None + + # Attach the new one before freeing the old, so there is no moment at + # which the card holds a dangling pointer - even though both this and + # _expand run on NVDA's main thread and cannot overlap today. + old, self._dict = self._dict, new + try: + with self._libLock: + lib.dtalk_set_dictionary( + self._dt.handle, new, 1 if self._inlinePhonemes else 0) + except Exception: + log.exception("doubletalkpc: attaching the dictionary failed") + if old: + try: + lib.dtalk_dict_free(old) + except Exception: + log.exception("doubletalkpc: freeing the old dictionary failed") + if new is not None: + log.info("doubletalkpc: loaded %d dictionary rules from %d file(s)" + % (total, len(paths))) + return (total, len(paths) if new is not None else 0) + + def reloadDictionaries(self): + """Re-read the dictionary files, for the settings panel. + + This is what makes editing a .dict file a matter of saving it rather + than restarting the synthesizer. + """ + return self._loadDictionaries() + + def _expand(self, text): + """Run the pronunciation dictionary over one utterance's bytes. + + Returns the text unchanged when nothing is loaded, which is the common + case, and on any failure. + """ + if not self._dict and not self._inlinePhonemes: + return text + try: + lib, h = self._dt.lib, self._dt.handle + need = lib.dtalk_expand(h, text, len(text), None, 0) + if need <= 0: + return text + buf = ctypes.create_string_buffer(need + 1) + lib.dtalk_expand(h, text, len(text), buf, need + 1) + return buf.raw[:need] + except Exception: + log.exception("doubletalkpc: dictionary expansion failed") + return text + def _clampSavedSettings(self): # Settings saved by add-on <= 0.1.9 can hold slider values below # CARD_MIN_PCT (the sliders then ran 0-100; reverb's old default was 0). @@ -632,8 +778,20 @@ def speak(self, speechSequence): else: eff = min(100, max(0, self._pitch + offset)) parts.append("\x01%dP" % self._mapPitch(eff)) + # Nothing is spoken until the CR arrives. parts.append("\r") - self._queue.put("".join(parts).encode("ascii", "replace")) + # The dictionary is applied HERE, not in dtalk_say: this driver queues + # its own bytes with dtalk_queue, which is the raw path, so a + # substitution made any further down would never happen for NVDA. + # + # It runs over the finished utterance, commands and all, because rcdict + # tokenizes the Ctrl-A commands as opaque atoms and will not match into + # or across one - so an index marker planted between two words cannot be + # swallowed by a rule, and a rule cannot corrupt a command into + # something the card would execute. The trailing CR is an utterance + # boundary to it, which matching never crosses either. + text = "".join(parts).encode("ascii", "replace") + self._queue.put(self._expand(text)) def cancel(self): self._stopping.set() diff --git a/doubletalk/nvda/synthDrivers/doubletalkpc/dictfiles.py b/doubletalk/nvda/synthDrivers/doubletalkpc/dictfiles.py new file mode 100644 index 0000000..3931a0e --- /dev/null +++ b/doubletalk/nvda/synthDrivers/doubletalkpc/dictfiles.py @@ -0,0 +1,260 @@ +# license: BSD-3-Clause +# copyright-holders: David Sexton +# +# Which pronunciation dictionaries load, and in what order. +# +# Two halves of the add-on need the same answer: the synth driver, which loads +# the files, and the settings panel, which lets the user arrange them. It lives +# here rather than with the panel because the driver has to work whether or not +# the panel ever loads - never the other way round. +# +# The rule in one sentence: the dictionaries are the files listed in NVDA's +# configuration, in that order, wherever on disk they live, plus every .dict +# file in /doubletalkpc/ that the list has not already named, +# appended in sorted order. +# +# Two sources, because they answer two different questions. The folder is the +# zero-configuration one: drop a file in and it works, which is how dictionaries +# worked before there was a panel and how a user who never opens the panel goes +# on using them. The list is for files kept ANYWHERE ELSE - a dictionary in a +# Dropbox folder, one under version control, one being edited in a text editor +# two windows away. Those are REFERENCED WHERE THEY LIE, never copied here: a +# copy is a second file that goes stale the moment the first is edited, and the +# whole point of the reload button is that editing a dictionary is a matter of +# saving it. +# +# So an entry is one of two things, and which one it is is decided by whether it +# has a directory part: +# +# "spanish.dict" a file in the dictionary folder; stored by +# name so that a portable copy of the NVDA +# configuration keeps working on another machine +# "D:\\dicts\\spanish.dict" a reference to a file somewhere else; stored +# absolute, because a relative path would be +# relative to whatever NVDA's working directory +# happens to be +# +# Everything else follows from rcdict's one hard rule: matching is +# first-match-wins in LOAD ORDER, so a later file cannot override an earlier one +# - it can only add. Order is therefore not a cosmetic preference, it is the +# only setting there is, which is why it is worth a config key and four buttons. + +import os + +import config +import globalVars +from logHandler import log + +#: Section in NVDA's configuration, and the folder under . Same +#: name for both, on purpose: one thing to tell a user to look for. +CONFIG_SECTION = "doubletalkpc" +FOLDER_NAME = "doubletalkpc" + +EXTENSION = ".dict" + +#: string_list rather than a single delimited string because a Windows path can +#: contain almost anything, including whatever separator would have been picked. +_CONFIG_SPEC = { + "dictionaries": "string_list(default=list())", +} + + +def initConfig(): + """Declare our configuration section. + + Idempotent and safe to call from anywhere; both the driver and the settings + panel call it before touching config, because whichever of them runs first + cannot know that the other has. + """ + if CONFIG_SECTION not in config.conf.spec: + config.conf.spec[CONFIG_SECTION] = _CONFIG_SPEC + + +def folderPath(create=False): + """The folder scanned for dictionaries, created on demand. + + Not created unless asked for: a user with no dictionaries should not find an + empty folder in their configuration wondering what it wants from them. + """ + path = os.path.join(globalVars.appArgs.configPath, FOLDER_NAME) + if create and not os.path.isdir(path): + os.makedirs(path) + return path + + +def folderNames(): + """Every dictionary file in the folder, sorted, whatever the order says.""" + try: + return sorted( + f for f in os.listdir(folderPath()) + if f.lower().endswith(EXTENSION) + ) + except OSError: + # No folder yet is the normal case, not a problem to report. + return [] + + +# --- entries ------------------------------------------------------------------ +# +# An entry is the string that goes in the configuration and comes back out of +# it: a bare name for a file in the folder, an absolute path for one anywhere +# else. This is the only section that understands that distinction; the load +# order below it, the driver and the settings panel all just pass entries +# around and ask here whenever they need to know something about one. + + +def _fold(path): + """One spelling of a path, for comparing two of them. + + normcase does the Windows part of this - it is what turns a forward slash + someone typed into a backslash - and lower() is on top of it rather than + left to normcase because this module is tested off Windows, where normcase + is the identity, and a case rule that only holds on the target platform is + a case rule nothing checks. + """ + return os.path.normcase(path).lower() + + +def normalize(entry): + """The canonical form of one entry, or "" for something unusable. + + A path that happens to point INTO the dictionary folder becomes a bare name, + so that adding a file already in the folder cannot produce a second listing + of the same file - and so that the entry survives the configuration being + carried to a machine where the folder is somewhere else. + """ + entry = entry.strip().strip('"') + if not entry: + return "" + if not os.path.dirname(entry): + return entry + path = os.path.abspath(entry) + if _fold(os.path.dirname(path)) == _fold(os.path.abspath(folderPath())): + return os.path.basename(path) + return path + + +def inFolder(entry): + """Is this entry one of the folder's own files rather than a reference?""" + return not os.path.dirname(entry) + + +def fullPath(entry): + """The file an entry names.""" + return os.path.join(folderPath(), entry) if inFolder(entry) else entry + + +def exists(entry): + """Is the file there? A reference is allowed not to be; see mergeOrder.""" + return os.path.isfile(fullPath(entry)) + + +def key(entry): + """What makes two entries the same file, for de-duplication. + + Case-folded, because Windows filesystems are, and the configuration may hold + whatever spelling the file was added under. Public because the settings + panel has to ask the same question of the list it is holding, and two + answers to "is this the same dictionary" would be one too many. + """ + return _fold(os.path.abspath(fullPath(entry))) + + +def label(entry): + """How one entry reads in the settings panel. + + A folder file is just its name; there is one folder and the panel says where + it is. A reference has to carry its directory or two files called + spanish.dict are indistinguishable - and where the file is IS the + interesting part of a reference. A missing one says so rather than sitting + in the list looking as though it were doing something. + """ + if inFolder(entry): + name = entry + else: + name = "%s (%s)" % (os.path.basename(entry), os.path.dirname(entry)) + return name if exists(entry) else "%s - not found" % name + + +# --- load order --------------------------------------------------------------- + + +def mergeOrder(preferred): + """The dictionaries to load, ordered by C{preferred}, folder files appended. + + Entries in C{preferred} are normalized and de-duplicated, then every .dict + file in the folder that none of them names is appended in sorted order. That + last clause is what keeps dropping a file into the folder working. + + The two kinds of entry are treated differently when the file is not there, + and deliberately: + + - a folder entry that has gone drops out, because the folder is scanned + and is therefore the authority on what is in it: a file deleted from it + is gone, and keeping the name would be keeping a ghost. + - a reference that has gone is KEPT, because it names a file this add-on + does not control - on a memory stick not plugged in, a network share not + mounted yet, a file its editor is halfway through rewriting. Dropping it + would mean the user silently loses the reference for good the next time + the panel saves. It is marked "not found" in the list and skipped at load + time; removing it is the user's decision to make. + + The folder's spelling of a name wins over the configuration's, because the + filesystem is the authority on that. + """ + folder = dict((key(n), n) for n in folderNames()) + ordered = [] + seen = set() + for entry in preferred: + entry = normalize(entry) + if not entry: + continue + k = key(entry) + if k in seen: + continue + if inFolder(entry): + entry = folder.get(k) + if entry is None: + continue + seen.add(k) + ordered.append(entry) + ordered.extend(sorted( + name for k, name in folder.items() if k not in seen)) + return ordered + + +def orderedEntries(): + """The dictionaries to load, in load order, as entries.""" + return mergeOrder(savedOrder()) + + +def orderedPaths(): + """The dictionary files to load, in load order, as full paths. + + Only the ones that are actually there: this is what the driver hands to + rcdict, and a reference to a file that is not currently reachable is a thing + to skip this time round, not an error. + """ + return [fullPath(e) for e in orderedEntries() if exists(e)] + + +def savedOrder(): + """The order stored in the configuration. Empty if it has never been set.""" + initConfig() + try: + return list(config.conf[CONFIG_SECTION]["dictionaries"]) + except Exception: + # A config written by some other version, or a section that somehow did + # not take the spec. Falling back to the sorted folder is exactly the + # behaviour this add-on had before the order was configurable. + log.debugWarning("doubletalkpc: could not read the dictionary order", + exc_info=True) + return [] + + +def setOrder(entries): + """Store the load order.""" + initConfig() + config.conf[CONFIG_SECTION]["dictionaries"] = [ + e for e in (normalize(x) for x in entries) if e + ] diff --git a/doubletalk/nvda/tests/nvdastub.py b/doubletalk/nvda/tests/nvdastub.py new file mode 100644 index 0000000..d14bdc1 --- /dev/null +++ b/doubletalk/nvda/tests/nvdastub.py @@ -0,0 +1,196 @@ +"""Just enough of NVDA's API to load a synth driver outside it. + +Only what the two drivers actually touch. The point is to run the real speak() +against the real emulator, so anything that would make the driver take a +different path is stubbed as thinly as possible. +""" +import sys +import types + + +def install(configPath): + # The drivers import FreeLibrary to unmap the DLL on terminate. It only + # exists on Windows; here the library is never unmapped, which is exactly + # what the driver does when it decides unloading is unsafe. + import _ctypes + if not hasattr(_ctypes, "FreeLibrary"): + _ctypes.FreeLibrary = lambda handle: None + + # The drivers encode dictionary paths with "mbcs" - the Windows ANSI code + # page, which is what the C runtime's fopen will decode a char* path with. + # There is no such codec off Windows, and without this the driver logs + # "could not load " for every dictionary and quietly loads none, which + # is a property of the harness and not of the driver. + import codecs + import encodings.cp1252 + try: + codecs.lookup("mbcs") + except LookupError: + codecs.register( + lambda name: encodings.cp1252.getregentry() if name == "mbcs" else None) + + config = types.ModuleType("config") + + class _Conf(dict): + def __init__(self): + dict.__init__(self) + self.spec = {} + self.profiles = [] + + config.conf = _Conf() + config.conf["audio"] = {"outputDevice": None} + sys.modules["config"] = config + + globalVars = types.ModuleType("globalVars") + globalVars.appArgs = types.SimpleNamespace(configPath=configPath, secure=False) + sys.modules["globalVars"] = globalVars + + logHandler = types.ModuleType("logHandler") + + class _Log: + def __init__(self): + self.lines = [] + + def _record(self, level): + def write(*args, **kwargs): + self.lines.append("%s: %s" % (level, args[0] if args else "")) + return write + + def __getattr__(self, name): + return self._record(name) + + logHandler.log = _Log() + sys.modules["logHandler"] = logHandler + + nvwave = types.ModuleType("nvwave") + + class WavePlayer: + """Records what was fed, so a test can look at the audio.""" + + def __init__(self, **kwargs): + self.fed = bytearray() + + def feed(self, data, onDone=None): + self.fed.extend(data) + if onDone: + onDone() + + def stop(self): + pass + + def idle(self): + pass + + def close(self): + pass + + def pause(self, switch): + pass + + nvwave.WavePlayer = WavePlayer + sys.modules["nvwave"] = nvwave + + class _Setting: + def __init__(self, id=None, displayName=None, **kwargs): + self.id = id + self.displayName = displayName + for key, value in kwargs.items(): + setattr(self, key, value) + + driverSetting = types.ModuleType("autoSettingsUtils.driverSetting") + driverSetting.DriverSetting = _Setting + driverSetting.NumericDriverSetting = _Setting + driverSetting.BooleanDriverSetting = _Setting + utils = types.ModuleType("autoSettingsUtils.utils") + + class StringParameterInfo: + def __init__(self, id, displayName): + self.id, self.displayName = id, displayName + + utils.StringParameterInfo = StringParameterInfo + autoSettingsUtils = types.ModuleType("autoSettingsUtils") + autoSettingsUtils.driverSetting = driverSetting + autoSettingsUtils.utils = utils + sys.modules["autoSettingsUtils"] = autoSettingsUtils + sys.modules["autoSettingsUtils.driverSetting"] = driverSetting + sys.modules["autoSettingsUtils.utils"] = utils + + sdh = types.ModuleType("synthDriverHandler") + + class SynthDriver(object): + @staticmethod + def VoiceSetting(**kwargs): + return _Setting("voice", "Voice", **kwargs) + + @staticmethod + def RateSetting(**kwargs): + return _Setting("rate", "Rate", **kwargs) + + @staticmethod + def RateBoostSetting(**kwargs): + return _Setting("rateBoost", "Rate boost", **kwargs) + + @staticmethod + def PitchSetting(**kwargs): + return _Setting("pitch", "Pitch", **kwargs) + + @staticmethod + def VolumeSetting(**kwargs): + return _Setting("volume", "Volume", **kwargs) + + @staticmethod + def InflectionSetting(**kwargs): + return _Setting("inflection", "Inflection", **kwargs) + + def loadSettings(self, onlyChanged=False): + pass + + class VoiceInfo(StringParameterInfo): + def __init__(self, id, displayName, language=None): + StringParameterInfo.__init__(self, id, displayName) + self.language = language + + class _Notification: + def __init__(self): + self.notified = [] + + def notify(self, **kwargs): + self.notified.append(kwargs) + + sdh.SynthDriver = SynthDriver + sdh.VoiceInfo = VoiceInfo + sdh.synthIndexReached = _Notification() + sdh.synthDoneSpeaking = _Notification() + sys.modules["synthDriverHandler"] = sdh + + commands = types.ModuleType("speech.commands") + + class IndexCommand: + def __init__(self, index): + self.index = index + + class _Offset: + def __init__(self, offset): + self.offset = offset + + class PitchCommand(_Offset): + pass + + class RateCommand(_Offset): + pass + + class VolumeCommand(_Offset): + pass + + commands.IndexCommand = IndexCommand + commands.PitchCommand = PitchCommand + commands.RateCommand = RateCommand + commands.VolumeCommand = VolumeCommand + speech = types.ModuleType("speech") + speech.commands = commands + sys.modules["speech"] = speech + sys.modules["speech.commands"] = commands + + return types.SimpleNamespace( + config=config, globalVars=globalVars, log=logHandler.log, + commands=commands, synthDriverHandler=sdh) diff --git a/doubletalk/nvda/tests/run.sh b/doubletalk/nvda/tests/run.sh new file mode 100755 index 0000000..2349915 --- /dev/null +++ b/doubletalk/nvda/tests/run.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# license:BSD-3-Clause +# +# Run the driver's dictionary tests outside NVDA, against the real emulator. +# +# ./run.sh # builds a shared libdtalk from ../../build/*.o +# ROM=path ./run.sh # a firmware ROM elsewhere +# +# The driver is Windows-only in production (it loads dtalk64.dll), but nothing +# about the dictionary path is: the tests stage a copy of the driver package +# with the host's shared library standing in for the DLL, which is why this +# works on a Mac or a Linux box with no NVDA in sight. +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +OBJ=../../build +[[ -d "$OBJ" ]] || { echo "run 'make -C ../..' first" >&2; exit 1; } + +# There is no shared-library target in the Makefile - the DLL rules cross +# compile, and libdtalk.a is what the CLI links - so build one here. +case "$(uname -s)" in + Darwin) SHARED=$OBJ/libdtalk.dylib; FLAGS=-dynamiclib ;; + *) SHARED=$OBJ/libdtalk.so; FLAGS=-shared ;; +esac +if [[ ! -f "$SHARED" ]] || [[ -n "$(find "$OBJ" -name '*.o' -newer "$SHARED" -print -quit)" ]]; then + ${CXX:-c++} $FLAGS -o "$SHARED" "$OBJ"/*.o + echo "built $SHARED" +fi + +ROM=${ROM:-} +if [[ -z "$ROM" ]]; then + for candidate in ../../../../doubletalkpc.bin ../synthDrivers/doubletalkpc/doubletalkpc.bin; do + [[ -f "$candidate" ]] && ROM=$candidate && break + done +fi +[[ -n "$ROM" && -f "$ROM" ]] || { + echo "no firmware ROM - set ROM=" >&2; exit 1; } + +echo "== dictionary file ordering" +python3 test_dictfiles.py ../synthDrivers/doubletalkpc + +echo +echo "== the driver, against the emulator" +python3 test_dictionaries.py ../synthDrivers/doubletalkpc "$SHARED" "$ROM" diff --git a/doubletalk/nvda/tests/test_dictfiles.py b/doubletalk/nvda/tests/test_dictfiles.py new file mode 100644 index 0000000..e9abb8c --- /dev/null +++ b/doubletalk/nvda/tests/test_dictfiles.py @@ -0,0 +1,200 @@ +"""Exercise dictfiles.py outside NVDA, with the modules it imports stubbed. + +The interesting behaviour is all in mergeOrder: what happens when the configured +order and the folder disagree. That is worth a test because every one of those +disagreements is a real situation - a file deleted by hand, a file dropped in by +hand, a config written under a different spelling on a case-insensitive +filesystem, a dictionary referenced on a drive that is not plugged in today. + +The two kinds of entry - a name in the add-on's folder, a path to a file +anywhere else - behave differently on purpose, and the difference is what the +second half of this file is about. + +usage: test_dictfiles.py DRIVER_DIR +""" +import os +import sys +import tempfile + +# The driver package is imported from the tree itself, so do not scatter +# __pycache__ directories through it - they would end up in the add-on. +sys.dont_write_bytecode = True + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import nvdastub # noqa: E402 + +configPath = tempfile.mkdtemp() +nvda = nvdastub.install(configPath) + +sys.path.insert(0, os.path.abspath(sys.argv[1])) +import dictfiles # noqa: E402 + +failures = [] + + +def check(name, got, want): + if got != want: + failures.append("%s\n got %r\n want %r" % (name, got, want)) + print("FAIL %s" % name) + else: + print("ok %s" % name) + + +def setOrder(entries): + # Written straight into the stub config: the point here is dictfiles' own + # merge, not NVDA's configobj validation. + nvda.config.conf.setdefault(dictfiles.CONFIG_SECTION, {})["dictionaries"] = list(entries) + + +folder = dictfiles.folderPath(create=True) + +#: Somewhere that is not the dictionary folder, standing in for the user's own +#: dictionaries - the ones this add-on must read where they lie. +elsewhere = tempfile.mkdtemp() + + +def touch(path): + open(path, "w").close() + return path + + +def inFolder(name): + return touch(os.path.join(folder, name)) + + +def outside(name, directory=None): + return touch(os.path.join(directory or elsewhere, name)) + + +# --- the folder, which works with no configuration at all --------------------- + +# No configuration: sorted folder order, which is what the add-on did before the +# order was configurable, and what a user who never opens the settings panel +# goes on getting. +inFolder("50-shared.dict") +inFolder("00-mine.dict") +inFolder("notes.txt") +check("unconfigured: sorted, non-.dict ignored", + dictfiles.orderedEntries(), ["00-mine.dict", "50-shared.dict"]) + +# A configured order wins, even against sorting. This is the whole point of the +# panel: rules are first-match-wins in load order, so the order IS the priority. +setOrder(["50-shared.dict", "00-mine.dict"]) +check("configured order wins", + dictfiles.orderedEntries(), ["50-shared.dict", "00-mine.dict"]) + +# A file dropped into the folder by hand is appended, not lost. +inFolder("zz-new.dict") +inFolder("aa-new.dict") +check("unlisted folder files appended in sorted order", + dictfiles.orderedEntries(), + ["50-shared.dict", "00-mine.dict", "aa-new.dict", "zz-new.dict"]) + +# A folder file named in the configuration but no longer there drops out: the +# folder is scanned, so it is the authority on what is in it. +setOrder(["gone.dict", "00-mine.dict", "50-shared.dict"]) +check("missing folder files drop out", + dictfiles.orderedEntries(), + ["00-mine.dict", "50-shared.dict", "aa-new.dict", "zz-new.dict"]) + +# Case: the config may hold whatever spelling the file was added under, but the +# folder is the authority on the real name. +setOrder(["00-MINE.DICT", "50-Shared.Dict"]) +check("case-insensitive match, folder spelling returned", + dictfiles.orderedEntries()[:2], ["00-mine.dict", "50-shared.dict"]) + +# A duplicated entry collapses onto the first, which is the one that would have +# won anyway. +setOrder(["zz-new.dict", "zz-new.dict", "00-mine.dict"]) +check("duplicates collapse", + dictfiles.orderedEntries(), + ["zz-new.dict", "00-mine.dict", "50-shared.dict", "aa-new.dict"]) + +# mergeOrder is what the settings panel calls to re-sync a list it is holding +# without discarding an arrangement the user has not applied yet. +check("mergeOrder keeps a working order and appends the rest", + dictfiles.mergeOrder(["aa-new.dict"]), + ["aa-new.dict", "00-mine.dict", "50-shared.dict", "zz-new.dict"]) + + +# --- references to files kept anywhere else ----------------------------------- + +# The reason for all of this: a dictionary the user keeps in their own folder is +# read where it lies, so editing it and pressing Reload is the whole workflow. +mine = outside("mine.dict") +setOrder([mine, "00-mine.dict"]) +check("a path outside the folder is kept as a path", + dictfiles.orderedEntries()[:2], [mine, "00-mine.dict"]) +check("and it is what gets loaded", + dictfiles.orderedPaths()[0], mine) + +# Order across the two kinds is one order: a reference can outrank a folder file +# or sit below it, because first-match-wins does not care where a file lives. +setOrder(["00-mine.dict", mine]) +check("references and folder files share one order", + dictfiles.orderedEntries()[:2], ["00-mine.dict", mine]) + +# A reference to a file that is not there today is KEPT - the drive may simply +# not be plugged in - but it is not handed to rcdict. +missing = os.path.join(elsewhere, "not-here.dict") +setOrder([missing, "00-mine.dict"]) +check("a missing reference stays in the list", + dictfiles.orderedEntries()[0], missing) +check("a missing reference is not loaded", + missing in dictfiles.orderedPaths(), False) +check("and the list says so", + dictfiles.label(missing).endswith("not found"), True) + +# A path that points into the dictionary folder is the same thing as the bare +# name, however it was written, so adding an already-installed file cannot +# produce two entries for one file. +setOrder([os.path.join(folder, "50-shared.dict"), "00-mine.dict"]) +check("a path into the folder normalizes to a name", + dictfiles.orderedEntries()[:2], ["50-shared.dict", "00-mine.dict"]) +setOrder([os.path.join(folder, "50-shared.dict"), "50-shared.dict"]) +check("the same file written two ways collapses", + dictfiles.orderedEntries().count("50-shared.dict"), 1) + +# Two files of the same name in different places are two dictionaries, and the +# list has to be able to hold both. +other = outside("mine.dict", tempfile.mkdtemp()) +setOrder([mine, other]) +check("same name, different directories, both kept", + dictfiles.orderedEntries()[:2], [mine, other]) +check("labels tell them apart", + dictfiles.label(mine) != dictfiles.label(other), True) + +# What goes back into the configuration is the normalized form, so the config +# does not accumulate the spellings of whatever file picker produced them. +dictfiles.setOrder([ + " %s " % os.path.join(folder, "00-mine.dict"), '"%s"' % mine, ""]) +check("setOrder stores normalized entries", + list(nvda.config.conf[dictfiles.CONFIG_SECTION]["dictionaries"]), + ["00-mine.dict", mine]) + +# A relative path is resolved against the working directory once, when it is +# stored, rather than meaning something different every time NVDA is started. +check("a relative path is stored absolute", + dictfiles.normalize(os.path.join(".", "sub", "mine.dict")), + os.path.abspath(os.path.join("sub", "mine.dict"))) + + +# --- nothing at all ----------------------------------------------------------- + +# No folder: not an error, and the commonest state there is. A reference still +# works without one, which is the point of references. +nvda.globalVars.appArgs.configPath = tempfile.mkdtemp() +setOrder([]) +check("no folder yet", dictfiles.orderedEntries(), []) +check("no folder yet, paths", dictfiles.orderedPaths(), []) +setOrder([mine]) +check("a reference works with no folder at all", + dictfiles.orderedPaths(), [mine]) + +print() +if failures: + print("\n".join(failures)) + print("%d failures" % len(failures)) + sys.exit(1) +print("0 failures") diff --git a/doubletalk/nvda/tests/test_dictionaries.py b/doubletalk/nvda/tests/test_dictionaries.py new file mode 100644 index 0000000..eb1e9b3 --- /dev/null +++ b/doubletalk/nvda/tests/test_dictionaries.py @@ -0,0 +1,176 @@ +"""Drive the real doubletalkpc NVDA driver against the real emulator. + +Checks the things that are only true if the whole path is wired up: that a +dictionary is found and loaded, that speak() substitutes through it, that the +utterance still ends in the CR without which the card says nothing at all, and +that reloading picks up an edited file. + +usage: test_dictionaries.py DRIVER_DIR SHARED_LIBRARY ROM + +Run it through run.sh, which finds all three. +""" +import os +import shutil +import sys +import tempfile + +# The driver package is imported from the tree itself, so do not scatter +# __pycache__ directories through it - they would end up in the add-on. +sys.dont_write_bytecode = True + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import nvdastub # noqa: E402 + +driverDir, dylib, rom = sys.argv[1:4] + +configPath = tempfile.mkdtemp() +dictFolder = os.path.join(configPath, "doubletalkpc") +os.makedirs(dictFolder) + +nvda = nvdastub.install(configPath) + +# A package the driver can be imported from, with the ROM and the native +# library beside it - the driver looks for both next to its __init__.py. +staging = tempfile.mkdtemp() +package = os.path.join(staging, "doubletalkpc") +shutil.copytree(driverDir, package) +shutil.copyfile(rom, os.path.join(package, "doubletalkpc.bin")) +shutil.copyfile(dylib, os.path.join(package, "dtalk64.dll")) +sys.path.insert(0, staging) + +import doubletalkpc # noqa: E402 + +failures = [] + + +def check(name, condition, detail=""): + if condition: + print("ok %s" % name) + else: + failures.append("%s %s" % (name, detail)) + print("FAIL %s %s" % (name, detail)) + + +def quiesce(driver): + """Stop the synthesis thread so the test owns the queue. + + The driver starts a daemon thread that blocks on _queue.get(), so a test + that inspects what speak() queued is racing it - and losing that race means + the emulator renders the whole utterance, which for the split test below is + minutes of speech nobody is listening to. A None on the queue is the + driver's own way of telling the thread to return. + """ + driver._queue.put(None) + driver._thread.join(timeout=5) + assert not driver._thread.is_alive(), "synthesis thread did not stop" + return driver + + +def render(driver, sequence): + """speak() and return the bytes it queued, without running the thread.""" + driver.speak(sequence) + return driver._queue.get_nowait() + + +def writeDict(name, body): + with open(os.path.join(dictFolder, name), "w") as f: + f.write(body) + + +IndexCommand = nvda.commands.IndexCommand + +# --- no dictionaries at all: the ordinary case, and nothing may change ------- + +driver = quiesce(doubletalkpc.SynthDriver()) +plain = render(driver, ["I use NVDA."]) +check("no dictionary: text passes through", b"I use NVDA." in plain, repr(plain)) +check("no dictionary: utterance ends in CR", plain.endswith(b"\r"), repr(plain[-8:])) +check("no dictionary: nothing loaded", driver._dict is None) +driver.terminate() + +# --- one dictionary --------------------------------------------------------- + +writeDict("50-shared.dict", "#!rcdict 1\n#!case insensitive\nword\t\tNVDA\tenn vee dee ay\n") + +driver = quiesce(doubletalkpc.SynthDriver()) +check("dictionary loaded", driver._dict is not None) +spoken = render(driver, ["I use NVDA."]) +check("substitution happens in speak()", b"enn vee dee ay" in spoken, repr(spoken)) +check("original word is gone", b"NVDA" not in spoken, repr(spoken)) +check("utterance still ends in CR", spoken.endswith(b"\r"), repr(spoken[-8:])) + +# The prefix commands must survive expansion untouched: they are what select +# the voice, and rcdict is supposed to treat them as opaque atoms. +check("Ctrl-A prefix intact", spoken.startswith(b"\x01"), repr(spoken[:12])) +check("number mode intact", b"\x0114B" in spoken, repr(spoken)) + +# An index marker planted between two words must come through whole. A mangled +# \x01nI would change the card's mode or eat the next character. +withIndex = render(driver, ["I use ", IndexCommand(7), "NVDA."]) +check("index marker survives", b"\x01%dI" % 0 in withIndex or b"\x011I" in withIndex, + repr(withIndex)) +check("substitution after a marker", b"enn vee dee ay" in withIndex, repr(withIndex)) + +# --- ordering: the earlier file wins ---------------------------------------- + +writeDict("00-mine.dict", "#!rcdict 1\n#!case insensitive\nword\t\tNVDA\tmy own version\n") + +result = driver.reloadDictionaries() +check("reload reports counts", result is not None and result[1] == 2, repr(result)) +spoken = render(driver, ["I use NVDA."]) +check("first file in load order wins", b"my own version" in spoken, repr(spoken)) + +# --- phonemes make it through as a mode switch ------------------------------ + +writeDict("00-mine.dict", + "#!rcdict 1\n#!case insensitive\nword\t\tNVDA\t[EH N V IY D IY EY]\n") +driver.reloadDictionaries() +spoken = render(driver, ["I use NVDA."]) +check("phoneme span is wrapped in the mode switches", + b"\x01D" in spoken and b"\x01T" in spoken, repr(spoken)) + +# --- the audio actually differs --------------------------------------------- +# +# The strongest check available here: run the emulator and require the +# substituted utterance to sound different from the plain one. Without this +# everything above could pass with a dictionary the card never receives. + +import ctypes # noqa: E402 + + +def samples(driver, sequence): + driver.speak(sequence) + utterance = driver._queue.get_nowait() + lib, h = driver._dt.lib, driver._dt.handle + lib.dtalk_queue(h, utterance, len(utterance)) + buf = ctypes.create_string_buffer(2048 * 2) + buf16 = ctypes.cast(buf, ctypes.POINTER(ctypes.c_int16)) + out = bytearray() + while True: + n = lib.dtalk_synth16(h, buf16, 2048) + if n == 0: + break + out.extend(buf.raw[:n * 2]) + return bytes(out) + + +withDict = samples(driver, ["I use NVDA."]) +driver.terminate() + +os.remove(os.path.join(dictFolder, "00-mine.dict")) +os.remove(os.path.join(dictFolder, "50-shared.dict")) +driver = quiesce(doubletalkpc.SynthDriver()) +check("dictionaries gone after removing the files", driver._dict is None) +without = samples(driver, ["I use NVDA."]) +driver.terminate() + +check("the card really speaks something different", + len(withDict) > 1000 and len(without) > 1000 and withDict != without, + "%d vs %d samples" % (len(withDict) // 2, len(without) // 2)) + +print() +if failures: + print("%d failures" % len(failures)) + sys.exit(1) +print("0 failures") diff --git a/doubletalk/rcdict/DICTIONARY-GUIDE.md b/doubletalk/rcdict/DICTIONARY-GUIDE.md new file mode 100644 index 0000000..5774adf --- /dev/null +++ b/doubletalk/rcdict/DICTIONARY-GUIDE.md @@ -0,0 +1,606 @@ +# Writing pronunciation dictionaries + +A pronunciation dictionary is a plain text file that rewrites words on their way +to the synthesizer. You can use it to fix a name it says wrongly, expand an +abbreviation, spell something out phonetically, or slow it down for an acronym +that goes past too fast. + +Dictionary files are named with a `.dict` extension. Where you put them, and how +you tell the program about them, depends on which program you are using — see +its own documentation for that. Everything else about them is the same +everywhere, and is what this guide covers. + +--- + +## A first dictionary + +``` +#!rcdict 1 +#!case insensitive + +; My dictionary. Lines starting with ';' are comments. +; The columns are separated by TAB characters. + +word Sean Shon +word Mbps megabits per second +word NVDA [EH N V IY D IY EY] +``` + +Three rules. The first two are **respellings** — you write the word the way it +should sound, and it goes back through the synthesizer's own letter-to-sound +rules. The third is **phonemes**, for when respelling cannot get you there. + +Respelling first, always. It is easier to write, easier to read six months +later, and it keeps all of the synthesizer's natural rhythm and intonation. +Reach for phonemes only when no spelling gives you the sound you want. + +--- + +## The shape of a file + +A dictionary is line-oriented UTF-8 text. There are three kinds of line. + +### Comments + +A line whose first non-blank character is `;` is a comment. Comments must be +lines of their own — you cannot put one at the end of a rule — which means a +`;` inside a rule needs no escaping. + +``` +; This is a comment. +``` + +### Directives + +Lines beginning `#!` set something for the whole file. All of them are +optional, but the first two are worth writing down every time. + +``` +#!rcdict 1 the format version +#!case insensitive the file's default: ignore capitals (this is the default) +#!case sensitive ...or match capitals exactly +#!lang en the language, recorded for humans; nothing acts on it yet +``` + +### Rules + +Everything else is a rule, and a rule is a row of columns: + +``` +type flags pattern output +``` + +**The columns are separated by tab characters, not spaces.** That is the one +thing to get right. Because a tab can never appear inside a column, your pattern +and your output can contain spaces, `=`, `;` and anything else without needing +to be escaped. + +The **flags** column may be left empty, or left out altogether for a +three-column line: + +``` +word US [Y UW EH S] four columns, flags column empty +word Sean Shon three columns, no flags column +word (sic) two columns: matches, says nothing +``` + +A trailing tab is a real empty last column and does change the meaning of the +line. No other whitespace matters. + +Two things follow from that, and both catch people out: + +- **One tab per column.** Two tabs in a row is an *empty column*, not spacing. + Lining your rules up prettily with extra tabs gives you too many columns and + the line is rejected. Use one tab, or use the empty flags column deliberately. +- **There are no end-of-line comments.** Anything after the output column's tab + is still part of the output and will be spoken. Put remarks on their own `;` + line above the rule. + +### If your editor eats tabs + +Tabs are invisible, and text boxes, web forms and some editors turn them into +spaces without telling you. So a line **with no tabs anywhere in it** is read a +second way, and means exactly the same thing: + +``` +type:flags pattern = output +``` + +for example: + +``` +word Sean = Shon +word:C US = [Y UW EH S] +``` + +Here the pattern ends at an `=` with a space on either side, and `\=` is a +literal equals sign. Both spellings work, and you can mix them in one file. Use +tabs when you can. + +--- + +## Matching: what to look for + +The first column says what kind of pattern it is. + +### `word` — whole words only + +``` +word cat kat +``` + +Matches `cat`, and `cat.` and `(cat)`, but not `cats`, `bobcat` or `cat5`. A +word is bounded by anything that is not a letter or a digit. + +This is what you want almost every time. + +### `text` — any run of characters + +``` +text Mbps megabits per second +``` + +Matches anywhere, including in the middle of a word. Useful for units and +suffixes, and dangerous for anything short: a `text` rule for `it` will also +fire inside `bitmap` and `politics`. + +### `regex` — a regular expression + +``` +regex \bv([0-9]+)\.([0-9]+)\b version \1 point \2 +regex ([0-9]+)% \1 percent +``` + +For patterns you cannot express as a fixed string. The pieces you captured in +round brackets come back in the output as `\1` to `\9`. + +Supported: capture groups, alternation `|`, character classes `[a-z]`, anchors +`^` and `$`, word boundaries `\b` and `\B`, the quantifiers `*` `+` `?` and +`{n,m}`, and lazy (`*?`) and possessive (`*+`) forms. + +Two things to know about regular expressions here: + +- **They always match capitals exactly**, whatever `#!case` says, and whatever + flags you write. There is no case-insensitive mode. Write a character class + instead: `[Vv]ersion`. +- A pattern that takes too long to try is abandoned, reported, and switched off + for the rest of that piece of text, so a runaway expression cannot make the + synthesizer stall. If a regex rule seems to work sometimes and not others, + this is the first thing to suspect — simplify it. + +### `rule` — the chip's own exception syntax + +``` +rule C(O)N [AA] +rule $R(H) +rule (5) [S I NG K O] +``` + +This is the syntax the RC8650 datasheet uses for its own exception +dictionaries: `L(F)R`, read as *"the text fragment F, occurring with left +context L and right context R, gets this pronunciation."* Only what is inside +the parentheses is consumed and replaced. The contexts are looked at and left +alone, so `C(O)N` rewrites just the `o` of `icon`, and the `n` is still there +for the next rule to see. + +The first example gives `o` between `c` and `n` the sound in *cot*. The second +says an `h` after an initial `r` is silent, as in *rhyme* — no output at all. +The third is a whole Spanish `5`. + +Use it when a pronunciation depends on the letters around it rather than on the +word, and when you are converting a dictionary RC Systems wrote — those are in +exactly this form already, one line each. + +**Context tokens.** The characters in `L` and `R` are matched literally, except +for these fifteen: + +| | | +|---|---| +| `#` | a vowel | +| `+` | a front vowel: e, i, y | +| `^` | a consonant | +| `*` | one or more consonants | +| `:` | zero or more consonants | +| `?` | a voiced consonant | +| `@` | one of d, j, l, n, r, s, t, z, ch, sh, th | +| `!` | one of b, c, d, f, g, p, t | +| `%` | a suffix (`-ing`, `-ed`, `-ely`…), followed by a non-letter | +| `&` | a sibilant: c, g, j, s, x, z, ch, sh | +| `$` | a nonalphabetic character — a space, a digit, or the edge of the text | +| `~` | one or more spaces or control characters | +| `\` | a digit | +| `\|` | one or more digits, ignoring commas | +| `` ` `` | any character at all | + +Inside the fragment they are all literal except `` ` ``, which is still a +wildcard there. + +A fragment may be empty. `rule` with a pattern of `()` and no output matches +anything nothing else did and silences it, which is how the datasheet suggests +ending a dictionary meant to replace the chip's built-in rules entirely. + +### Writing rules for another language + +Those classes are English. Left as they are, an accented letter is not a letter +at all — so `$`, which means "not a letter", matches *inside* a word, and rules +written for single letters start firing in the middle of one. Spanish `más` +comes out spelled aloud as "e-eme-e a e-ese-e". + +So declare the alphabet: + +``` +#!class # a e i o u y á é í ó ú ü +#!class ^ b c d f g h j k l m n ñ p q r s t v w x y z +#!class + e i y é í +#!class ? b d g j l m n ñ r v w z +``` + +Members are separated by spaces, which is what lets a class hold the +two-character members of `@` and `&`, or a list of suffixes. + +You do not declare `$`, `*` or `:` and cannot: `$` is "not in `#` and not in +`^`", and `*` and `:` follow `^`. Declaring the two alphabets is enough to make +all of them right. + +A declaration applies to every rule *after* it, and rules keep the classes they +were loaded under — so two dictionaries for two languages can be loaded at once +and neither disturbs the other. + +### Converting a dictionary RC Systems wrote + +If you have one of their `.dic` files — the ones RCStudio edits, and the samples +that ship with it — you do not have to retype it. `dic2rcdict.py`, in this same +directory, converts one: + +``` +python3 dic2rcdict.py Spanish.dic -o spanish.dict +``` + +It reports what it did on stderr: how many rules it wrote, how many were +silent, and any line it could not read. Nothing is written silently — a line +that is not an exception, or a pronunciation using something that is not a +phoneme, is named with its line number and left out rather than guessed at. + +The rules themselves come across unchanged; `L(F)R` is `L(F)R` in both. What the +converter does is the surrounding work: + +- **The encoding.** RC Systems' files are ISO-8859-1, dictionaries here are + UTF-8. Use `-e` if yours is something else. +- **The pronunciations** are wrapped in the `[ ]` of a phoneme span, and checked + a symbol at a time against the phoneme table. +- **The alphabet.** `--lang es` writes the Spanish `#!class` declarations above + before the rules. It is the default, because Spanish is the sample everyone + has; `--lang en` writes none, which is right for English. For any other + language it says it has none to write, and you add your own — the section + above is what to write. +- **Character mode.** A `.dic` file may have a second half, after a line + containing just `C`, for how the chip should read letters out one at a time. + There is no character mode here, and the text-mode rules already name every + letter, so that half is dropped and counted. `--char-mode keep` folds it in + with the rest if you want to look at it. + +The result is an ordinary dictionary file. Edit it, reorder it, put your own +rules above it — but if you edit the `.dic` and convert again, whatever you +changed here goes. Keep your own additions in a separate file loaded before it; +that is what load order is for. + +### What a rule dictionary costs + +A dictionary that covers every letter turns the whole utterance into phonemes. +That is fine — consecutive rules are joined into a single phoneme span rather +than one span each — but the text roughly doubles in length, so a piece that +fitted the chip's input buffer before may not afterwards. Expand first, then +split. The synthesizer drivers already do this in the right order. + +--- + +## Flags: capitals + +The second column takes any of three letters. + +| Flag | Meaning | +|---|---| +| `i` | Ignore capitals. The default, unless the file says `#!case sensitive`. | +| `c` | Match capitals exactly. | +| `C` | Ignore the capitals in your *pattern*, but match only where the *text* is in all capitals. | + +`c` and `C` both keep an acronym from swallowing an ordinary word: + +``` +word C US [Y UW EH S] +word C IT [AY T IY] +``` + +`US` and `IT` are spelled out; `us`, `it` and `Us` at the start of a sentence +are left alone. Written with `c` and an all-capitals pattern these two rules +would behave the same way. The difference is only in what you have to type +correctly: with `C` the pattern's own capitals do not matter, so `word C us` and +`word C Us` are the same rule. + +--- + +## The output: what to say instead + +The last column is a template. Most of it is ordinary text, and ordinary text is +spoken through the synthesizer's own letter-to-sound rules — which is why a +respelling is just a word. + +Inside it, four things are special. + +### `[ ... ]` — phonemes + +``` +word arthritis [AA R TH R AY DX IX S] +``` + +Everything between the brackets is a phoneme string (see the table below). The +synthesizer is switched into phoneme mode for it and back out afterwards; you do +not write the switches yourself. + +Every symbol is checked when the file is loaded. A typo is reported with its +line number and the rule is skipped — it can never reach the synthesizer as +something unintelligible. + +### `{ ... }` — a command + +``` +word ASAP {2S}[EY EH S EY P IY]{5S} +``` + +Whatever is inside the braces is sent to the synthesizer as one of its own +commands. The example drops to speed 2 for the acronym, so a string of letters +that flies past at normal speed is read deliberately. + +Note what the second command is **not** doing. `{5S}` sets the speed to 5; it +does not restore whatever the speed was before. A command changes the +synthesizer for everything that follows, so a rule that changes a setting has to +name the value it wants to come back to — and only makes sense where you know +what that value is. + +Commands are passed through as written and are **not** checked, so this is the +one part of the format where a mistake can produce something odd. Look up the +command letters in your synthesizer's documentation. + +### `\0` to `\9` — what was matched + +`\0` is the whole of the matched text. `\1` to `\9` are the round-bracket groups +of a `regex` rule, numbered left to right, and are only available there. + +``` +regex ([0-9]+)% \1 percent +word kg \0 ilograms +``` + +### `\` — literals + +`\[ \] \{ \} \\` give you those characters as themselves, and `\=` a literal +equals sign in the untabbed spelling. Anything else after a backslash is simply +the character that follows it. + +### Mixing them + +An entry has one output field, not two, so text and phonemes go in together: + +``` +word Dr [D AA K T ER] +text approx approximately +``` + +### An empty output + +Leave the output off entirely and the match is **silent** — the matched text is +removed and nothing is said in its place: + +``` +text (sic) +``` + +--- + +## Phonemes + +Phoneme symbols are written in capitals, separated by spaces, between `[` and +`]`. + +### Vowels + +| Symbol | As in | | Symbol | As in | +|---|---|---|---|---| +| `AA` | f**a**ther, h**o**t | | `IY` | b**ee**t, s**ee** | +| `AE` | c**a**t, b**a**d | | `OW` | b**oa**t, g**o** | +| `AH` | b**u**t, c**u**p | | `OY` | b**oy**, c**oi**n | +| `AW` | **ou**t, h**ow** | | `UH` | b**oo**k, p**u**t | +| `AX` | **a**bout (unstressed) | | `UW` | b**oo**t, f**oo**d | +| `AY` | b**i**te, m**y** | | `EW` | same sound as `UW` | +| `EH` | b**e**t, s**ai**d | | `EI` | same sound as `EY` | +| `ER` | b**ir**d, h**er** | | `IH` | b**i**t, s**i**t | +| `EY` | b**a**ke, d**ay** | | `IX` | ros**e**s (unstressed) | + +### Consonants + +| Symbol | As in | | Symbol | As in | +|---|---|---|---|---| +| `B` `D` `F` `G` | as spelled | | `NG` | si**ng** | +| `H` `K` `L` `M` | as spelled | | `NY` | o**ni**on | +| `N` `P` `R` `S` | as spelled | | `SH` | **sh**oe | +| `T` `V` `W` `Z` | as spelled | | `ZH` | mea**s**ure | +| `CH` | **ch**urch | | `TH` | **th**in | +| `J` | **j**udge | | `DH` | **th**is, **th**em | +| `YY` | **y**es | | `WH` | **wh**ich | +| `DX` | bu**tt**er (a flap) | | `RR` | a rolled `R` | +| `KX` `PX` `TX` | variants of `K`, `P` and `T` — the synthesizer normally picks these itself | + +### Single letters + +`A` `E` `I` `O` `U` and `Y` are **also** phoneme symbols, and they are *not* the +letters' names: + +| Written | Is the same sound as | +|---|---| +| `A` | `AA` (f**a**ther) | +| `E` | `EH` (b**e**t) | +| `I` | `IY` (b**ee**t) | +| `O` | `OW` (b**oa**t) | +| `U` | `UW` (b**oo**t) | +| `Y` | `YY` (**y**es) | + +So `[K A T]` says "cot", not "cat". You want `[K AE T]`. When in doubt, use the +two-letter symbols — they are unambiguous. + +Several other symbols are also the same sound as each other, which means +choosing between them changes nothing: `AH` and `AX`; `I`, `IY`; `IH` and `IX`; +`EW`, `U` and `UW`; `O` and `OW`; `A` and `AA`; `E` and `EH`; `EI` and `EY`. + +### Pauses + +`.` and `,` may be written inside a phoneme string as pauses — `,` a short one, +`.` a longer one. + +> On the RC8650 only, `'` is available as a very short pause. The DoubleTalk PC +> does not have it, and a dictionary using it will report an error there. Leave +> it out of any dictionary you intend to share. + +### Stress and pitch + +Six characters — `/` `\` `+` `-` `>` `<` — are accepted inside a phoneme string +as attribute modifiers, along with a number from 0 to 99 which sets the pitch at +that point. + +These attach directly to what follows them rather than standing as symbols of +their own, and are written with no space, so `-/D>/EH R` is `-`, `/`, `D`, `>`, +`/`, `EH`, space, `R`. + +`/` places a rising-pitch marker exactly where you write it, which is the one of +the six with a documented use here: it is how you would rebuild the rise on a +question by hand. For what the others do, see the attribute modifier table in +your synthesizer's own documentation — they change pitch and stress rather than +timing, and are fiddly enough that they are worth reading up on rather than +guessing at. + +**Most dictionaries need none of this.** The synthesizer applies its own stress +and intonation to a phoneme string exactly as it does to ordinary text, so +leaving them out is normally the right answer and is what these examples do. + +--- + +## Order decides everything + +**Rules are tried from the top of the file downwards, and the first one that +matches wins.** Nothing looks for the longest or the best match — it takes the +first. + +That makes ordering the single most important thing about a dictionary, and it +catches people out in one particular way: a short pattern placed above a longer +one that starts the same way means the longer one is never reached. + +``` +text RAT rodent +text RATING score +``` + +`RATING` never fires: scanning reaches the `R`, `RAT` matches there, and that is +the end of it — the result is "rodentING". Turn them round: + +``` +text RATING score +text RAT rodent +``` + +**Put your specific rules above your general ones.** If a rule is not firing, +this is almost always why. + +The same applies when you use more than one dictionary file at once: they are +all read into a single list, in the order they are loaded, and the first match +in that combined list wins. A file loaded later can therefore only *add* rules — +it can never override one in a file loaded before it. To beat an existing rule, +your file has to be loaded first. + +Two more things follow from this model: + +- **Scanning goes left to right through the text**, and once a rule has fired, + scanning carries on *after* the text it matched. +- **Output is never looked at again.** A rule cannot match something another + rule produced, so rules cannot trigger each other, and there is no way to + write a loop. + +--- + +## Things worth knowing + +**Your text is not the only thing in the stream.** By the time a dictionary +runs, the program may have put its own commands into the text. A match will +never run across one of these, and never inside one — so a rule cannot corrupt a +command, and in exchange, a rule cannot match a phrase that happens to have one +in the middle of it. In practice they fall between chunks of text and you will +not notice. + +**A word cannot be substituted across a line ending.** Matching stops at the end +of each piece of text the program sends. + +**Mind the `!` `?` `;` and `:`.** A phoneme string that ends immediately before +one of these loses the rising intonation that a question would normally get, +because there is no pause symbol to carry it. The substitution is still made — +the right word said flatly beats the wrong word — but if a phrase matters, a +respelling will keep the intonation where phonemes will not. + +**One bad line costs one rule.** A line that does not parse, or a phoneme +symbol that does not exist, is reported with its file name and line number and +skipped; the rest of the file loads normally. Check your program's log if a rule +is not doing anything. + +**Nothing here can silence the synthesizer.** If a dictionary cannot be read, or +a rule cannot be applied, the text is spoken unchanged. + +--- + +## Cheat sheet + +This is a working dictionary. Every line uses exactly one tab between columns, +and the remarks are on their own comment lines because the format has no +end-of-line comments. + +``` +#!rcdict 1 +#!case insensitive + +; columns are TAB separated: type flags pattern output +; (the flags column may be empty, or left out entirely) + +; respell a name +word Sean Shon +; expand an abbreviation +word Mbps megabits per second +; phonemes +word NVDA [EH N V IY D IY EY] +; only when written in capitals +word C US [Y UW EH S] +; slow down for an acronym, then put the speed back +word ASAP {2S}[EY EH S EY P IY]{5S} +; text matches inside words too +text Kbps kilobits per second +; no output at all: matched and said silently +text (sic) +; keep what was captured +regex ([0-9]+)% \1 percent +; the chip's own exception syntax: o between c and n, as in icon +rule C(O)N [AA] +; h after an initial r is silent, as in rhyme +rule $R(H) + +; no tabs available? this means the same as the Mbps line above: +word Mbps = megabits per second +``` + +| | | +|---|---| +| `word` | whole words only | +| `text` | any substring | +| `regex` | regular expression, `\1`–`\9` for the groups, always case-sensitive | +| `rule` | the chip's `L(F)R` exception syntax; only `F` is replaced | +| `#!class # …` | redefine a context token's characters, for another language | +| `i` `c` `C` | ignore capitals / match them exactly / match only all-capitals text | +| `[ ]` | phonemes | +| `{ }` | a synthesizer command | +| `\0` | the text that matched | +| `\[ \] \{ \} \\` | those characters, literally | +| first match wins | so put specific rules above general ones | diff --git a/doubletalk/rcdict/dic2rcdict.py b/doubletalk/rcdict/dic2rcdict.py new file mode 100755 index 0000000..3244bf6 --- /dev/null +++ b/doubletalk/rcdict/dic2rcdict.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +"""dic2rcdict --- convert an RC Systems .dic exception dictionary to rcdict. + +The source is the text form RCStudio compiles: one exception per line, in the +RC8650 datasheet's L(F)R=P syntax, ISO-8859-1, with Table 5 phoneme mnemonics +on the right-hand side. The target is an rcdict tab-separated dictionary using +the `rule` matcher, in UTF-8: + + rule L(F)R [P] + +The mapping is one-to-one and needs no editing of the rules themselves -- the +pattern column carries the exception verbatim, and the pronunciation only has +to be wrapped in the phoneme-span brackets rcdict already understands. What +changes is the encoding (ISO-8859-1 to UTF-8) and the split of the file at the +Character-mode marker, which rcdict has no equivalent for. + + python3 rcdict/dic2rcdict.py RCStudio/Samples/Dictionaries/Spanish.dic \ + -o spanish.dict + +This lives beside rcdict itself, and is mirrored into doubletalk-pc by +sync-to-doubletalk-pc.sh, because the dictionaries it produces are read by both +engines: a converter kept next to only one of them would be a converter one of +them did not know it had. +""" + +import argparse +import re +import sys + +RULE_RE = re.compile(r"^(.*?)\((.*?)\)(.*?)=(.*)$") + +# Table 5 letter mnemonics, plus the pause symbols and Table 6 modifiers that +# may appear in a pronunciation. Kept in step with rcdict.c's table5[]. +TABLE5 = set("""AA AE AH AW AX AY CH DH DX EH EI ER EW EY IH IX IY KX NG NY +OW OY PX RR SH TH TX UH UW WH YY ZH A B D E F G H I J K L M N O P R S T U V W +Y Z""".split()) +PAUSES = ".,'" +MODIFIERS = "/\\+-><" + + +# Table 22's classes are English, and that is the one part of the rule language +# that does not travel: with the stock vowel list an accented letter is not a +# letter at all, so '$' matches inside a word and the single-letter rules start +# firing -- "mas" comes out spelled aloud. rcdict lets a dictionary declare its +# own alphabet; these are the declarations each language needs. +CLASSES = { + "es": [ + ("#", "a e i o u y á é í ó ú ü"), + ("+", "e i y é í"), + ("^", "b c d f g h j k l m n ñ p q r s t v w x y z"), + ("?", "b d g j l m n ñ r v w z"), + ], +} + + +def convert(text, name): + """Yield (mode, kind, payload) for every line of a .dic file. + + mode is 'text' or 'char'; kind is 'rule', 'comment' or 'error'. + """ + mode = "text" + for lineno, raw in enumerate(text.replace("\r\n", "\n").replace("\r", "\n").split("\n"), 1): + s = raw.strip() + if not s: + yield mode, "blank", "" + continue + if s.startswith(";"): + yield mode, "comment", s[1:].strip() + continue + if s in ("C", "c"): + mode = "char" + continue + m = RULE_RE.match(s) + if not m: + yield mode, "error", "%s:%d: not an exception: %r" % (name, lineno, s) + continue + left, frag, right, pron = m.groups() + pattern = "%s(%s)%s" % (left, frag, right) + if "\t" in pattern: + yield mode, "error", "%s:%d: tab in pattern" % (name, lineno) + continue + yield mode, "rule", (lineno, pattern, pron.split()) + + +def check_phonemes(syms): + return [s for s in syms if s.upper() not in TABLE5 + and not (len(s) == 1 and (s in PAUSES or s in MODIFIERS))] + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("source") + ap.add_argument("-o", "--output", default="-") + ap.add_argument("-e", "--encoding", default="iso-8859-1", + help="source encoding (default iso-8859-1, per RC Systems)") + ap.add_argument("--lang", default="es") + ap.add_argument("--char-mode", choices=("drop", "keep"), default="drop", + help="what to do with Character-mode exceptions; rcdict has " + "no character mode, and the text-mode $(x)$ rules " + "already name every letter (default: drop)") + args = ap.parse_args() + + text = open(args.source, "rb").read().decode(args.encoding) + out = [] + stats = {"text": 0, "char": 0, "silent": 0, "bad": 0} + + out.append("; Generated by rcdict/dic2rcdict.py from %s" % args.source) + out.append("; Do not edit here; edit the source dictionary and re-run.") + out.append(";") + out.append("#!rcdict 1") + out.append("#!case insensitive") + out.append("#!lang %s" % args.lang) + classes = CLASSES.get(args.lang) + if classes: + out.append("") + out.append("; The alphabet these rules are written against. Without it") + out.append("; the accented letters are not letters, '$' matches inside") + out.append("; a word, and the letter-naming rules fire in the middle of") + out.append("; one. '$' follows '#' and '^'; '*' and ':' follow '^'.") + for tok, members in classes: + out.append("#!class %s %s" % (tok, members)) + elif args.lang != "en": + print("warning: no character classes known for --lang %s; the rules " + "will be matched against Table 22's English alphabet" + % args.lang, file=sys.stderr) + out.append("") + + for mode, kind, payload in convert(text, args.source): + if mode == "char" and args.char_mode == "drop": + if kind == "rule": + stats["char"] += 1 + continue + if kind == "blank": + if out and out[-1] != "": + out.append("") + continue + if kind == "comment": + # RCStudio buries editor anchors (#Hnn) in comments; they carry no + # meaning outside its dictionary editor. + if re.fullmatch(r"#H\d+", payload, re.I): + continue + out.append("; %s" % payload if payload else ";") + continue + if kind == "error": + print(payload, file=sys.stderr) + stats["bad"] += 1 + continue + + lineno, pattern, syms = payload + unknown = check_phonemes(syms) + if unknown: + print("%s:%d: not Table 5 symbols: %s" + % (args.source, lineno, " ".join(unknown)), file=sys.stderr) + stats["bad"] += 1 + continue + stats[mode] += 1 + if not syms: + stats["silent"] += 1 + out.append("rule\t\t%s\t" % pattern) + else: + out.append("rule\t\t%s\t[%s]" % (pattern, " ".join(syms))) + + body = "\n".join(out).rstrip("\n") + "\n" + if args.output == "-": + sys.stdout.write(body) + else: + open(args.output, "w", encoding="utf-8").write(body) + + print("%d text-mode rules (%d of them silent), %d character-mode rules %s, " + "%d rejected" + % (stats["text"], stats["silent"], stats["char"], + "dropped" if args.char_mode == "drop" else "kept", stats["bad"]), + file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/doubletalk/rcdict/example.dict b/doubletalk/rcdict/example.dict new file mode 100644 index 0000000..6b5f2f8 --- /dev/null +++ b/doubletalk/rcdict/example.dict @@ -0,0 +1,117 @@ +; example.dict --- a worked example of the rcdict format. +; +; COLUMNS ARE SEPARATED BY TABS. Four of them: +; +; type flags pattern output +; +; The flags column may be empty, and may be left out entirely for a +; three-column line. A tab cannot appear inside any field, so the pattern may +; contain spaces, '=' and ';' with nothing to escape. +; +; (Where tabs cannot survive -- a config dialog, a web form, a paste that +; expanded them -- the same rule can be written without any tabs as +; "type[:flags] pattern = output". Both spellings load.) +; +; Comment lines start with ';' and must be lines of their own. That is the +; chip datasheet's own convention for its exception dictionaries. + +#!rcdict 1 +#!case insensitive +#!lang en + +; --------------------------------------------------------------------------- +; Respellings. The simplest thing that works, and often the best: the output is +; ordinary text, so it goes back through the chip's letter-to-sound stage and +; keeps all of its prosody. The datasheet suggests exactly this as the first +; technique to reach for. +; --------------------------------------------------------------------------- +word Sean Shawn +word chauffeur show fur +word baseball base ball +word Adam atom + +; --------------------------------------------------------------------------- +; Phonemes, for when respelling cannot get there. Symbols are RC8650 datasheet +; Table 5, validated at load, so a typo is reported with its line number rather +; than sent to the chip. +; +; Several Table 5 symbols share one internal code -- AH and AX, I and IY, +; EW/U/UW, O and OW (see dict-lab/phoneme-codes.txt) -- so choosing between +; those is choosing nothing. +; --------------------------------------------------------------------------- +word arthritis [AA R TH R AY DX IX S] +word Ubuntu [UW B UH N T UW] +word Alex [AE L IX K S] + +; --------------------------------------------------------------------------- +; Case. 'c' matches exactly what is written, so a 'c' rule for "US" already +; leaves "us" and "Us" alone. 'C' asks a different question: it ignores the +; pattern's own case and fires only where the TEXT is all capitals. Reach for +; it when the pattern's case is not something you want to have to get right. +; --------------------------------------------------------------------------- +word C US [Y UW EH S] +word c iOS [AY OW EH S] + +; --------------------------------------------------------------------------- +; Substrings, for what is not a whole word. Rules are tried in load order and +; the first match wins, so put the specific ones first. This is the trap the +; datasheet warns about for its own dictionaries, where (RAT) placed before +; (RATING) means (RATING) is never reached. +; +; Commented out rather than deleted: whether an abbreviation is better expanded +; or left alone is a matter of taste, and the format is the point here. +; --------------------------------------------------------------------------- +; text Mbps megabits per second +; text Kbps kilobits per second + +; --------------------------------------------------------------------------- +; Commands can be mixed in: {2S} drops the chip to speed 2 for an acronym that +; is hard to catch at speed. Note what the second command is NOT: {5S} sets the +; speed to 5, it does not restore whatever the speed was before. A command +; leaves the chip changed for the rest of the utterance, so a rule that changes +; a setting has to name the value it wants back, and only fits where that value +; is known. An empty output makes the match silent. +; --------------------------------------------------------------------------- +; word ASAP {2S}[EY EH S EY P IY]{5S} +; word approx approximately + +; --------------------------------------------------------------------------- +; Regular expressions, with \1-\9 for the captured groups. Note that regex +; patterns always match CASE-SENSITIVELY, whatever #!case says: the engine has +; no case-insensitive mode, and lowercasing a pattern would turn \W into \w. +; Write a class such as [Vv] instead. +; --------------------------------------------------------------------------- + +; regex \bv([0-9]+)\.([0-9]+)\b version \1 point \2 +; regex ([0-9]+)% \1 percent + +; --------------------------------------------------------------------------- +; The chip's own exception syntax, L(F)R: "the text fragment F, occurring with +; left context L and right context R, gets this pronunciation". Only what is +; inside the parentheses is consumed -- the contexts are looked at and left for +; the next rule to see. Both of these are the datasheet's own examples. +; +; Fifteen context tokens: # a vowel, + a front vowel, ^ a consonant, * one or +; more consonants, : zero or more, ? a voiced consonant, @ one of d j l n r s t +; z ch sh th, ! one of b c d f g p t, % a suffix, & a sibilant, $ a +; nonalphabetic character (or the edge of the text), ~ one or more spaces or +; controls, \ a digit, | one or more digits, ` anything. +; --------------------------------------------------------------------------- + +; o between c and n, the o-sound in cot, as in icon and economy +; rule C(O)N [AA] +; an h after an initial r is silent, as in rhyme: no output at all +; rule $R(H) + +; --------------------------------------------------------------------------- +; Those classes are English, which is the one part of the rule language that +; does not travel. Declare the alphabet before the rules that need it and every +; class follows: $ is "not in # and not in ^", and * and : follow ^. Without +; this an accented letter is not a letter, $ matches inside a word, and rules +; written for single letters fire in the middle of one -- Spanish "mas" comes +; out spelled aloud. Members are separated by spaces, which is what lets a +; class hold the two-character members of @ and &. +; --------------------------------------------------------------------------- + +; #!class # a e i o u y á é í ó ú ü +; #!class ^ b c d f g h j k l m n ñ p q r s t v w x y z diff --git a/doubletalk/rcdict/rcdict.c b/doubletalk/rcdict/rcdict.c new file mode 100644 index 0000000..e6a56c6 --- /dev/null +++ b/doubletalk/rcdict/rcdict.c @@ -0,0 +1,2524 @@ +/* SPDX-License-Identifier: BSD-3-Clause */ +/* rcdict --- see rcdict.h. + * + * The substitution rules below are not guesses. Each one is a measurement, + * recorded in dict-lab/RESULTS.md and reproducible with dict-lab's scripts + * against the letter-to-sound stage's own output buffer at 0x1036. + */ + +#include +#include +#include +#include + +#include "rcdict.h" +#include "rcdict_regex.h" + +#define RCDICT_VERSION "0.2" + +/* Long enough for any sane entry; a longer line is reported and skipped rather + than silently truncated into something that means something else. */ +#define RCDICT_MAX_LINE 4096 + +/* --- profiles ------------------------------------------------------------- */ + +/* Table 5, letter mnemonics. 55 of them, and the calibration in + dict-lab/phoneme-codes.txt shows they land on 42 internal codes: nine groups + share one (AH/AX, I/IY, EW/U/UW and friends), and five are compounds the + chip expands itself (CH -> T SH, J -> D ZH, RR -> DX R DX). */ +static const char *const table5[] = { + "AA", "AE", "AH", "AW", "AX", "AY", "CH", "DH", "DX", "EH", "EI", "ER", + "EW", "EY", "IH", "IX", "IY", "KX", "NG", "NY", "OW", "OY", "PX", "RR", + "SH", "TH", "TX", "UH", "UW", "WH", "YY", "ZH", + "A", "B", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", "O", + "P", "R", "S", "T", "U", "V", "W", "Y", "Z", + NULL +}; + +const rcdict_profile rcdict_rc8650 = { + 0x01, + "\x01" "D", + "\x01" "T", + table5, + ".,'", /* the RC8650's table adds ' (short pause) */ + "/\\+-><", + 1900, /* what the NVDA driver already splits at */ +}; + +const rcdict_profile rcdict_doubletalk_pc = { + 0x01, + "\x01" "D", + "\x01" "T", + table5, + ".,", /* the PC manual's Table 5 has no ' */ + "/\\+-><", + 1900, +}; + +void +rcdict_options_init (rcdict_options *o, const rcdict_profile *p) +{ + if (!o) + return; + o->profile = p ? p : &rcdict_rc8650; + o->dict = NULL; + o->inline_phonemes = 0; + o->open = "[["; + o->close = "]]"; +} + +const char * +rcdict_version (void) +{ + return RCDICT_VERSION; +} + +/* --- small helpers -------------------------------------------------------- */ + +static int +is_space (int c) +{ + return c == ' ' || c == '\t' || c == '\n' || c == '\v' || c == '\f'; +} + +static int +is_digit (int c) +{ + return c >= '0' && c <= '9'; +} + +static int +is_alpha (int c) +{ + return ((unsigned) c | 32) >= 'a' && ((unsigned) c | 32) <= 'z'; +} + +static int +upper (int c) +{ + return (c >= 'a' && c <= 'z') ? c - 32 : c; +} + +static int +is_word_char (int c) +{ + return is_alpha (c) || is_digit (c); +} + +int +rcdict_is_phoneme (const rcdict_profile *p, const char *sym, size_t len) +{ + size_t i; + const char *const *t; + if (!p || !sym || !len) + return 0; + for (t = p->phonemes; *t; t++) + { + if (strlen (*t) != len) + continue; + for (i = 0; i < len; i++) + if (upper ((unsigned char) sym[i]) != (*t)[i]) + break; + if (i == len) + return 1; + } + return 0; +} + +/* --- output sink ---------------------------------------------------------- */ + +/* Writes what fits and counts what would have been needed, so one code path + serves both the sizing call and the real one. */ +typedef struct +{ + char *buf; + size_t cap, len, need; +} sink; + +/* Invariant, and what makes unput below correct: len is always + min(need, cap-1). need is the answer the caller wants; len is how much of it + fitted. */ +static void +put (sink * s, const char *p, size_t n) +{ + size_t room; + s->need += n; + if (!s->buf || s->len + 1 >= s->cap) + return; + room = s->cap - s->len - 1; /* keep one for the NUL */ + if (n > room) + n = room; + memcpy (s->buf + s->len, p, n); + s->len += n; +} + +/* Take back the last n bytes. Needed because whether the article before a + span is absorbed is only known once the span is reached, by which time the + article has been emitted as ordinary text. Restoring the invariant is all + that is required, so this works whether or not the buffer has already + overflowed. */ +static void +unput (sink * s, size_t n) +{ + if (n > s->need) + n = s->need; + s->need -= n; + if (s->buf && s->cap) + s->len = s->need < s->cap - 1 ? s->need : s->cap - 1; +} + +static void +puts_ (sink * s, const char *p) +{ + put (s, p, strlen (p)); +} + +static void +putc_ (sink * s, char c) +{ + put (s, &c, 1); +} + +static void +report_at (rcdict_report r, void *ctx, size_t off, const char *msg) +{ + if (r) + r (ctx, off, msg); +} + +/* --- phoneme strings ------------------------------------------------------ */ + +/* A phoneme string is not a list of space-delimited words. Table 6's + modifiers attach directly to phonemes -- the manual's own example is + "70H AW -/D>/EH R", where -/D>/EH is -, /, D, >, /, EH. So it is scanned a + character at a time: letter runs are mnemonics, digit runs are pitch values, + and everything else is either a modifier, a pause or an error. */ +int +rcdict_check_phonemes (const rcdict_profile *p, const char *s, size_t len, + rcdict_report report, void *ctx) +{ + size_t i = 0; + int ok = 1; + char msg[96]; + + if (!p || !s) + return 0; + + while (i < len) + { + unsigned char c = (unsigned char) s[i]; + + if (is_space (c)) + { + i++; + } + else if (is_alpha (c)) + { + size_t start = i; + while (i < len && is_alpha ((unsigned char) s[i])) + i++; + if (!rcdict_is_phoneme (p, s + start, i - start)) + { + size_t n = i - start; + char *q = msg; + if (n > 32) + n = 32; + strcpy (q, "unknown phoneme symbol '"); + q += strlen (q); + memcpy (q, s + start, n); + q += n; + *q++ = '\''; + *q = 0; + report_at (report, ctx, start, msg); + ok = 0; + } + } + else if (is_digit (c)) + { + /* Table 6: nn sets the pitch, 0-99. */ + size_t start = i; + while (i < len && is_digit ((unsigned char) s[i])) + i++; + if (i - start > 2) + { + report_at (report, ctx, start, + "pitch value has more than two digits"); + ok = 0; + } + } + else if (c && (strchr (p->modifiers, c) || strchr (p->pauses, c))) + { + i++; + } + else + { + char *q = msg; + strcpy (q, "not a phoneme, modifier or pause: '"); + q += strlen (q); + *q++ = (char) c; + *q++ = '\''; + *q = 0; + report_at (report, ctx, i, msg); + ok = 0; + i++; + } + } + return ok; +} + +/* --- dictionaries --------------------------------------------------------- */ + +typedef enum +{ + SEG_TEXT, /* literal, goes back through letter-to-sound */ + SEG_PHONEME, /* [ ... ] */ + SEG_COMMAND, /* { ... } */ + SEG_BACKREF /* \0 - \9 */ +} seg_kind; + +typedef struct +{ + seg_kind kind; + char *s; /* owned; NULL for SEG_BACKREF */ + size_t len; + int n; /* backref number */ +} seg; + +/* Captures handed to the output template. Slot 0 is the whole match, so \0 + works for every matcher and \1-\9 only ever mean something after a regex. */ +#define RCDICT_NCAPS 10 +typedef struct +{ + size_t start, len; +} rcap; + +typedef enum +{ + M_WORD, + M_TEXT, + M_REGEX, + M_RULE +} match_kind; + +/* --- Table 22 context tokens ---------------------------------------------- */ + +/* The fifteen tokens, in the datasheet's own order, which is also the order of + the class table below. Nine of them are sets of characters and can be + redefined per dictionary; the rest are structural and cannot. */ +static const char rc_tokens[] = "#+^*:?@!%&$~\\|`"; + +#define RC_NTOK 15 +#define RC_HASH 0 /* # a vowel */ +#define RC_PLUS 1 /* + a front vowel */ +#define RC_HAT 2 /* ^ a consonant */ +#define RC_STAR 3 /* * one or more consonants */ +#define RC_COLON 4 /* : zero or more consonants */ +#define RC_QUERY 5 /* ? a voiced consonant */ +#define RC_AT 6 /* @ one of d j l n r s t z ch sh th */ +#define RC_BANG 7 /* ! one of b c d f g p t */ +#define RC_PCT 8 /* % a suffix */ +#define RC_AMP 9 /* & a sibilant */ +#define RC_DOLLAR 10 /* $ a nonalphabetic character */ +#define RC_TILDE 11 /* ~ one or more non-printing characters */ +#define RC_BSLASH 12 /* \ a digit */ +#define RC_BAR 13 /* | one or more digits, commas ignored */ +#define RC_WILD 14 /* ` any character */ + +/* Class members, as a list of NUL-terminated strings ending in an empty one. + Strings rather than bytes because three of the classes have two-character + members (ch, sh, th) and the suffix class has whole suffixes, and because a + redefined class may contain characters outside ASCII -- these are compared + as UTF-8, so a member is however many bytes its characters need. + + Table 22's defaults are English, which is the one thing about the rule + language that does not travel: the vowels of a Spanish dictionary include + the accented ones, and a rule engine that does not know that reads "mas" + correctly and spells "mas" out letter by letter. So they are defaults and + not constants -- see rc_class_set and the #!class directive. */ +static const char def_vowel[] = "a\0e\0i\0o\0u\0y\0"; +static const char def_front[] = "e\0i\0y\0"; +static const char def_cons[] = + "b\0c\0d\0f\0g\0h\0j\0k\0l\0m\0n\0p\0q\0r\0s\0t\0v\0w\0x\0z\0"; +static const char def_voiced[] = "b\0d\0g\0j\0l\0m\0n\0r\0v\0w\0z\0"; +static const char def_at[] = "d\0j\0l\0n\0r\0s\0t\0z\0ch\0sh\0th\0"; +static const char def_bang[] = "b\0c\0d\0f\0g\0p\0t\0"; +static const char def_amp[] = "c\0g\0j\0s\0x\0z\0ch\0sh\0"; +static const char def_digit[] = "0\0" "1\0" "2\0" "3\0" "4\0" "5\0" "6\0" + "7\0" "8\0" "9\0"; +/* Table 22's suffix list. The parenthesised plurals are spelled out, and the + "must also be followed by a non-alphabetic character" part is in the + matcher rather than here. */ +static const char def_suffix[] = + "ables\0able\0ably\0edly\0eless\0elements\0element\0ements\0ement\0" + "eness\0ely\0ers\0er\0ed\0es\0e\0ingly\0ings\0ing\0"; + +/* Which token each default belongs to; NULL for the structural ones, whose + meaning is not a set of characters and so cannot be redefined. */ +static const char *const rc_defaults[RC_NTOK] = { + def_vowel, def_front, def_cons, NULL, NULL, def_voiced, def_at, def_bang, + def_suffix, def_amp, NULL, NULL, def_digit, NULL, NULL +}; + +/* A class table. Shared between the rules that were loaded under it, and + copied on write, so that two dictionaries with different alphabets can be + loaded into one rcdict and each keep its own -- which is the whole point of + putting the classes in the file rather than in the profile. */ +typedef struct rclasses +{ + int refs; /* -1 on the shared default, never freed */ + char *set[RC_NTOK]; /* NULL: use rc_defaults[i] */ +} rclasses; + +static rclasses rc_default_classes = { -1, { NULL } }; + +static const char * +rc_class (const rclasses * c, int tok) +{ + if (c && c->set[tok]) + return c->set[tok]; + return rc_defaults[tok]; +} + +static void +rc_classes_unref (rclasses * c) +{ + int i; + if (!c || c->refs < 0) + return; + if (--c->refs > 0) + return; + for (i = 0; i < RC_NTOK; i++) + free (c->set[i]); + free (c); +} + +/* Copy before writing, so rules already loaded keep the table they were + compiled against. Returns NULL only on allocation failure. */ +static rclasses * +rc_classes_own (rclasses * c) +{ + rclasses *n; + int i; + if (c && c->refs == 1) + return c; /* nobody else is looking at it */ + n = calloc (1, sizeof *n); + if (!n) + return NULL; + n->refs = 1; + for (i = 0; i < RC_NTOK; i++) + if (c && c->set[i]) + { + const char *p = c->set[i]; + size_t len = 0; + while (p[len]) + len += strlen (p + len) + 1; + len++; /* the terminating empty string */ + n->set[i] = malloc (len); + if (!n->set[i]) + { + rc_classes_unref (n); + return NULL; + } + memcpy (n->set[i], c->set[i], len); + } + rc_classes_unref (c); + return n; +} + +/* --- UTF-8, only as much of it as the classes need ------------------------ */ + +/* A dictionary is UTF-8 and the text it rewrites is too, so "the character + before the fragment" is not "the byte before the fragment". Everything here + walks characters; nothing decodes further than it has to. */ +static size_t +uclen (const char *s, size_t avail) +{ + unsigned char c = (unsigned char) s[0]; + size_t n = 1; + if (c >= 0xf0) + n = 4; + else if (c >= 0xe0) + n = 3; + else if (c >= 0xc0) + n = 2; + if (n > avail) + n = 1; /* malformed: treat the byte as itself */ + return n; +} + +/* Start of the character ending at i, not going below lo. */ +static size_t +uprev (const char *s, size_t lo, size_t i) +{ + size_t j = i; + while (j > lo && ((unsigned char) s[j - 1] & 0xc0) == 0x80) + j--; + return j > lo ? j - 1 : lo; +} + +/* Case-fold one character to a comparison key. ASCII, and the Latin-1 + supplement, which is as far as a Table 5 target's own character set went and + covers every accented letter a Spanish or French dictionary needs. Anything + beyond that compares unfolded rather than wrongly. */ +static unsigned +fold_cp (const char *s, size_t n) +{ + unsigned cp; + if (n == 1) + cp = (unsigned char) s[0]; + else if (n == 2) + cp = (unsigned) (((unsigned char) s[0] & 0x1f) << 6 + | ((unsigned char) s[1] & 0x3f)); + else if (n == 3) + cp = (unsigned) (((unsigned char) s[0] & 0x0f) << 12 + | ((unsigned char) s[1] & 0x3f) << 6 + | ((unsigned char) s[2] & 0x3f)); + else + return 0xfffd; + if (cp >= 'A' && cp <= 'Z') + return cp + 32; + /* C0-DE are the accented capitals; D7 is the multiplication sign, which is + in the middle of them and is not a letter. */ + if (cp >= 0xc0 && cp <= 0xde && cp != 0xd7) + return cp + 32; + return cp; +} + +/* Do the len bytes at a equal the NUL-terminated member b, case-folded? */ +static int +same_chars (const char *a, size_t alen, const char *b) +{ + size_t i = 0, j = 0, blen = strlen (b); + while (i < alen && j < blen) + { + size_t na = uclen (a + i, alen - i), nb = uclen (b + j, blen - j); + if (fold_cp (a + i, na) != fold_cp (b + j, nb)) + return 0; + i += na; + j += nb; + } + return i == alen && j == blen; +} + +/* Is the character (or characters) at s[0..avail) a member of the token's + class? Returns the number of BYTES matched, or 0 for no. Longest member + wins, which is what makes "ch" beat "c" in the @ and & classes. */ +static size_t +rc_class_match (const rclasses * c, int tok, const char *s, size_t avail) +{ + const char *list = rc_class (c, tok); + size_t best = 0; + if (!list || !avail) + return 0; + for (; *list; list += strlen (list) + 1) + { + /* Compare member characters against the input, both folded. */ + size_t i = 0, j = 0, mlen = strlen (list); + while (i < avail && j < mlen) + { + size_t na = uclen (s + i, avail - i), nb = uclen (list + j, + mlen - j); + if (fold_cp (s + i, na) != fold_cp (list + j, nb)) + break; + i += na; + j += nb; + } + if (j == mlen && i > best) + best = i; + } + return best; +} + +/* A letter is anything in the vowel class or the consonant class, and '$' is + anything that is not. Deriving it means a dictionary that adds the accented + vowels to '#' gets '$' right for free -- and getting '$' wrong is not a + small mistake: it is what makes a letter-naming rule such as $(m)$ fire in + the middle of a word. */ +static int +rc_is_letter (const rclasses * c, const char *s, size_t avail) +{ + return rc_class_match (c, RC_HASH, s, avail) != 0 + || rc_class_match (c, RC_HAT, s, avail) != 0; +} + +/* One element of a context or fragment: a token, or a literal character. */ +typedef struct +{ + unsigned char tok; /* a Table 22 token, or 0 for a literal */ + unsigned char len; /* literal length in bytes */ + char lit[5]; /* NUL-terminated: a UTF-8 character is <= 4 */ +} ratom; + +typedef struct rule +{ + match_kind kind; + int nocase; /* compare case-insensitively */ + int caps_only; /* and require the input to be all capitals */ + char *pat; + size_t patlen; + /* M_RULE only: the compiled L(F)R, and the class table it was compiled + against. left is matched right-to-left from the fragment, right is + matched left-to-right from the end of it. */ + ratom *left, *frag, *right; + size_t nleft, nfrag, nright; + rclasses *cls; + int anypos; /* fragment has no literal first byte */ + rcdict_regex *re; /* M_REGEX only; compiled once, at load */ + size_t rseq; /* ordinal among the regex rules */ + size_t seq; /* load order, so first-match-wins can be + decided between the literal and regex + lists, which are searched separately */ + seg *segs; + size_t nsegs; + char *origin; /* file name, for diagnostics */ + size_t line; + struct rule *next; /* next rule in the same first-byte bucket */ + struct rule *rnext; /* next regex rule, in load order */ + struct rule *anext; /* next any-position rule, in load order */ +} rule; + +struct rcdict +{ + const rcdict_profile *profile; + rule **rules; /* every rule, in load order, for freeing */ + size_t n, cap; + /* Indexed by the lowercased first byte of the pattern. Every rule that + could match at a position is in bucket[tolower(input byte)], and within a + bucket they stay in load order -- which is what keeps first-match-wins + meaning what the file says it means. (An Aho-Corasick automaton would be + the textbook answer for many literals at once, but it reports matches in + leftmost-longest order, and reconciling that with "the earlier rule wins" + costs more than the scan it saves at these sizes.) */ + rule *head[256], *tail[256]; + /* Regex rules cannot be indexed by a first byte -- a pattern may begin with + a class, an anchor or an alternation -- so they get their own list and are + tried at every position. Both lists are in load order, which is what lets + match_at decide between them by sequence number. */ + rule *rhead, *rtail; + size_t nregex; + int has_regex; + /* Rule matchers whose text fragment does not begin with a literal character + -- an empty fragment, or one starting with the wildcard -- cannot be + bucketed either, for the same reason. The datasheet's ()= idiom, which + ends a dictionary that is meant to replace the built in rules entirely, + lands here. */ + rule *ahead, *atail; + int has_anypos; + int has_rules; /* any M_RULE at all: see the lossy_punct note */ + /* The class table the next rules loaded will compile against. Rules hold + their own reference, so this only ever moves forward. */ + rclasses *classes; +}; + +rcdict * +rcdict_new (const rcdict_profile *p) +{ + rcdict *d = calloc (1, sizeof *d); + if (!d) + return NULL; + d->profile = p ? p : &rcdict_rc8650; + d->classes = &rc_default_classes; + return d; +} + +static void +free_rule (rule *r) +{ + size_t i; + if (!r) + return; + for (i = 0; i < r->nsegs; i++) + free (r->segs[i].s); + free (r->segs); + free (r->pat); + free (r->origin); + free (r->left); + free (r->frag); + free (r->right); + rc_classes_unref (r->cls); + rcdict_regex_free (r->re); + free (r); +} + +void +rcdict_clear (rcdict *d) +{ + size_t i; + if (!d) + return; + for (i = 0; i < d->n; i++) + free_rule (d->rules[i]); + free (d->rules); + d->rules = NULL; + d->n = d->cap = 0; + memset (d->head, 0, sizeof d->head); + memset (d->tail, 0, sizeof d->tail); + d->rhead = d->rtail = NULL; + d->nregex = 0; + d->has_regex = 0; + d->ahead = d->atail = NULL; + d->has_anypos = 0; + d->has_rules = 0; + rc_classes_unref (d->classes); + d->classes = &rc_default_classes; +} + +void +rcdict_free (rcdict *d) +{ + if (!d) + return; + rcdict_clear (d); + free (d); +} + +size_t +rcdict_rule_count (const rcdict *d) +{ + return d ? d->n : 0; +} + +/* --- the format ----------------------------------------------------------- */ + +/* Diagnostics name the file and line, because a dictionary is edited by hand + and "something is wrong somewhere" is not an actionable thing to be told. */ +static void +report_line (rcdict_report r, void *ctx, size_t off, const char *name, + size_t line, const char *msg) +{ + char buf[256]; + size_t n; + if (!r) + return; + n = (size_t) snprintf (buf, sizeof buf, "%s:%lu: %s", + name ? name : "", (unsigned long) line, msg); + if (n >= sizeof buf) + buf[sizeof buf - 1] = 0; + r (ctx, off, buf); +} + +/* The pattern ends at an '=' with whitespace on both sides (or at end of + line). Anything else is part of the pattern, so "a = b" and "x=y" and ";" + all mean what they look like. A literal separator is written "\=". */ +static const char * +find_separator (const char *s, size_t len) +{ + size_t i; + for (i = 0; i < len; i++) + { + if (s[i] == '\\') + { + i++; + continue; + } + if (s[i] == '=' && i > 0 && is_space ((unsigned char) s[i - 1]) + && (i + 1 == len || is_space ((unsigned char) s[i + 1]))) + return s + i; + } + return NULL; +} + +static void +trim (const char **s, size_t *len) +{ + while (*len && is_space ((unsigned char) **s)) + { + (*s)++; + (*len)--; + } + while (*len && is_space ((unsigned char) (*s)[*len - 1])) + (*len)--; +} + +/* Copy a pattern. The ONLY escape it has is "\\=", and only in the untabbed + spelling, where '=' separates the pattern from the output; tab-separated + columns need no escaping at all. Every other backslash survives untouched, + which is not a nicety -- a regex is mostly backslashes, and collapsing them + here would quietly turn \\b into b and \\. into any character. */ +static char * +pattern_dup (const char *s, size_t len, int allow_eq_escape, size_t *outlen) +{ + char *o = malloc (len + 1); + size_t i, j = 0; + if (!o) + return NULL; + for (i = 0; i < len; i++) + { + if (allow_eq_escape && s[i] == '\\' && i + 1 < len && s[i + 1] == '=') + o[j++] = s[++i]; + else + o[j++] = s[i]; + } + o[j] = 0; + *outlen = j; + return o; +} + +static int +push_seg (rule *r, seg_kind kind, const char *s, size_t len, int n) +{ + seg *ns = realloc (r->segs, (r->nsegs + 1) * sizeof *ns); + if (!ns) + return 0; + r->segs = ns; + ns += r->nsegs; + ns->kind = kind; + ns->n = n; + ns->len = len; + ns->s = NULL; + if (s) + { + ns->s = malloc (len + 1); + if (!ns->s) + return 0; + memcpy (ns->s, s, len); + ns->s[len] = 0; + } + r->nsegs++; + return 1; +} + +/* Parse the output template into segments. Doing it once at load means a + mistake is reported when the file is read rather than in the middle of + speaking, and that expansion does no parsing at all. */ +static int +parse_template (rule *r, const rcdict_profile *p, const char *s, size_t len, + const char *name, size_t line, size_t off, + rcdict_report report, void *ctx) +{ + size_t i = 0, lit = 0; + char litbuf[RCDICT_MAX_LINE]; + +#define FLUSH_LIT() \ + do { \ + if (lit && !push_seg (r, SEG_TEXT, litbuf, lit, 0)) return 0; \ + lit = 0; \ + } while (0) + + while (i < len) + { + char c = s[i]; + + if (c == '\\' && i + 1 < len) + { + char e = s[i + 1]; + if (e >= '0' && e <= '9') + { + FLUSH_LIT (); + if (!push_seg (r, SEG_BACKREF, NULL, 0, e - '0')) + return 0; + if (e != '0' && r->kind != M_REGEX) + report_line (report, ctx, off + i, name, line, + "\\1-\\9 need a matcher with capture groups; " + "only \\0 is available here"); + i += 2; + continue; + } + if (lit + 1 < sizeof litbuf) + litbuf[lit++] = e; + i += 2; + continue; + } + + if (c == '[' || c == '{') + { + char close = (c == '[') ? ']' : '}'; + size_t body = i + 1, e = body; + while (e < len && s[e] != close) + { + if (s[e] == '\\' && e + 1 < len) + e++; + e++; + } + if (e >= len) + { + report_line (report, ctx, off + i, name, line, + c == '[' ? "unclosed [ in output" + : "unclosed { in output"); + return 0; + } + FLUSH_LIT (); + if (c == '[') + { + if (!rcdict_check_phonemes (p, s + body, e - body, NULL, NULL)) + { + report_line (report, ctx, off + body, name, line, + "output contains a symbol that is not a " + "phoneme, modifier or pause"); + return 0; + } + if (!push_seg (r, SEG_PHONEME, s + body, e - body, 0)) + return 0; + } + else + { + if (e == body) + { + report_line (report, ctx, off + i, name, line, + "empty {} in output"); + return 0; + } + if (!push_seg (r, SEG_COMMAND, s + body, e - body, 0)) + return 0; + } + i = e + 1; + continue; + } + + if (lit + 1 < sizeof litbuf) + litbuf[lit++] = c; + i++; + } + FLUSH_LIT (); +#undef FLUSH_LIT + return 1; +} + +/* --- the rule matcher: L(F)R, RC8650 datasheet "Exception Syntax" --------- */ + +/* Compile one side of an exception into atoms. In a context every Table 22 + token is a token; in the text fragment only the wildcard is, and the + datasheet is explicit that any other token there is a literal. */ +static int +compile_atoms (const char *s, size_t len, int in_fragment, + ratom ** out, size_t *nout) +{ + size_t i = 0, n = 0, cap = 0; + ratom *a = NULL; + + while (i < len) + { + size_t cl = uclen (s + i, len - i); + ratom one; + memset (&one, 0, sizeof one); + + if (cl == 1 && (in_fragment ? s[i] == '`' : strchr (rc_tokens, s[i]) + && s[i])) + one.tok = (unsigned char) s[i]; + else + { + if (cl > 4) + cl = 1; + one.len = (unsigned char) cl; + memcpy (one.lit, s + i, cl); + } + + if (n == cap) + { + ratom *na = realloc (a, (cap ? cap * 2 : 8) * sizeof *na); + if (!na) + { + free (a); + return 0; + } + a = na; + cap = cap ? cap * 2 : 8; + } + a[n++] = one; + i += cl; + } + *out = a; + *nout = n; + return 1; +} + +/* Split "L(F)R" and compile all three parts. The fragment runs from the first + '(' to the LAST ')', so a rule may pronounce a parenthesis: "(()" is a rule + for '(' and "())" one for ')'. */ +static int +compile_rule (rule *r, const char *pat, size_t len) +{ + const char *open = memchr (pat, '(', len), *close = NULL; + size_t i; + + if (!open) + return 0; + for (i = len; i > (size_t) (open - pat); i--) + if (pat[i - 1] == ')') + { + close = pat + i - 1; + break; + } + if (!close) + return 0; + + if (!compile_atoms (pat, (size_t) (open - pat), 0, &r->left, &r->nleft)) + return 0; + if (!compile_atoms (open + 1, (size_t) (close - open - 1), 1, + &r->frag, &r->nfrag)) + return 0; + if (!compile_atoms (close + 1, (size_t) (pat + len - close - 1), 0, + &r->right, &r->nright)) + return 0; + + /* Bucketable only if the fragment starts with a literal character. */ + r->anypos = !(r->nfrag && r->frag[0].len); + return 1; +} + +/* One character of input compared with one literal atom. */ +static int +atom_is (const rule *r, const ratom * a, const char *s, size_t n) +{ + if (!r->nocase) + return a->len == n && memcmp (a->lit, s, n) == 0; + return same_chars (s, n, a->lit); +} + +static int +is_nonprinting (unsigned char c) +{ + return c < 0x21; +} + +/* Which Table 22 class does this token name? -1 if it is not one. */ +static int +tok_class (unsigned char tok) +{ + const char *q = tok ? strchr (rc_tokens, tok) : NULL; + return q ? (int) (q - rc_tokens) : -1; +} + +/* How many bytes of the class's longest member end at i, looking backwards? + Classes may have two-character members (ch, sh, th), so this is not simply + the length of the previous character. */ +static size_t +class_match_back (const rclasses * c, int idx, const char *in, size_t lo, + size_t i) +{ + size_t take = 0, back; + for (back = 1; back <= 8 && back <= i - lo; back++) + if (rc_class_match (c, idx, in + i - back, back) == back) + take = back; + return take; +} + +/* The context matchers recurse rather than loop, because the variable-width + tokens have to be able to give a character back: "(a)*x" wants the run of + consonants to stop one short of the x, and a greedy pass that never + reconsidered would eat the x -- x being a consonant -- and then fail, + silently, for as long as the dictionary lived. + + The budget is the same idea as the regex engine's: a pattern with several + variable-width tokens in it can in principle be made to explode, and a + dictionary must not be able to make a screen reader go quiet. Contexts are + a handful of atoms long in practice, so nothing real comes near it. */ +#define RC_CTX_BUDGET 4000 + +static int +ctx_right (const rule *r, const char *in, size_t i, size_t end, size_t k, + int *budget) +{ + const rclasses *c = r->cls; + const ratom *a; + size_t avail, cl; + + if (k == r->nright) + return 1; + if (--*budget < 0) + return 0; + + a = &r->right[k]; + avail = end - i; + + switch (a->tok) + { + case 0: /* a literal character */ + if (!avail) + return 0; + cl = uclen (in + i, avail); + if (!atom_is (r, a, in + i, cl)) + return 0; + return ctx_right (r, in, i + cl, end, k + 1, budget); + + case '$': /* a nonalphabetic character, or the end */ + if (!avail) + return ctx_right (r, in, i, end, k + 1, budget); + if (rc_is_letter (c, in + i, avail)) + return 0; + return ctx_right (r, in, i + uclen (in + i, avail), end, k + 1, budget); + + case '`': /* wildcard */ + if (!avail) + return 0; + return ctx_right (r, in, i + uclen (in + i, avail), end, k + 1, budget); + + case '*': /* one or more consonants */ + case ':': /* zero or more */ + { + size_t stops[64], n = 0, j = i; + stops[n++] = j; /* having taken none */ + while (n < sizeof stops / sizeof *stops && j < end + && (cl = rc_class_match (c, RC_HAT, in + j, end - j)) != 0) + stops[n++] = j += cl; + /* Longest first, which is what makes the common case one step. */ + while (n-- > (a->tok == '*' ? 1u : 0u)) + if (ctx_right (r, in, stops[n], end, k + 1, budget)) + return 1; + return 0; + } + + case '~': /* one or more non-printing characters */ + { + size_t j = i, n = 0; + while (j < end && is_nonprinting ((unsigned char) in[j])) + { + j++; + n++; + } + for (; n; n--) + if (ctx_right (r, in, i + n, end, k + 1, budget)) + return 1; + return 0; + } + + case '|': /* one or more digits; commas are ignored */ + { + size_t stops[64], n = 0, j = i; + while (n < sizeof stops / sizeof *stops && j < end) + { + if (in[j] == ',') + { + j++; + continue; + } + cl = rc_class_match (c, RC_BSLASH, in + j, end - j); + if (!cl) + break; + j += cl; + stops[n++] = j; /* one stop per digit: '|' needs at least one */ + } + while (n--) + if (ctx_right (r, in, stops[n], end, k + 1, budget)) + return 1; + return 0; + } + + case '%': /* a suffix, and then a nonalphabetic */ + { + const char *list = rc_class (c, RC_PCT); + for (; list && *list; list += strlen (list) + 1) + { + size_t mlen = strlen (list); + if (mlen > avail || !same_chars (in + i, mlen, list)) + continue; + if (i + mlen < end && rc_is_letter (c, in + i + mlen, + end - i - mlen)) + continue; /* Table 22: and then a non-letter */ + if (ctx_right (r, in, i + mlen, end, k + 1, budget)) + return 1; + } + return 0; + } + + default: /* a character class */ + { + int idx = tok_class (a->tok); + if (idx < 0 || !avail) + return 0; + cl = rc_class_match (c, idx, in + i, avail); + if (!cl) + return 0; + return ctx_right (r, in, i + cl, end, k + 1, budget); + } + } +} + +/* The same atoms, matched right-to-left from the fragment. Separate from + ctx_right rather than one function with a direction flag: every step of it + differs, and a UTF-8 character is walked backwards by a different means + than forwards. */ +static int +ctx_left (const rule *r, const char *in, size_t lo, size_t i, size_t k, + int *budget) +{ + const rclasses *c = r->cls; + const ratom *a; + size_t avail, ps; + + if (!k) + return 1; + if (--*budget < 0) + return 0; + + a = &r->left[k - 1]; + avail = i - lo; + + switch (a->tok) + { + case 0: + if (!avail) + return 0; + ps = uprev (in, lo, i); + if (!atom_is (r, a, in + ps, i - ps)) + return 0; + return ctx_left (r, in, lo, ps, k - 1, budget); + + case '$': + if (!avail) /* the start of the run is nonalphabetic */ + return ctx_left (r, in, lo, i, k - 1, budget); + ps = uprev (in, lo, i); + if (rc_is_letter (c, in + ps, i - ps)) + return 0; + return ctx_left (r, in, lo, ps, k - 1, budget); + + case '`': + if (!avail) + return 0; + return ctx_left (r, in, lo, uprev (in, lo, i), k - 1, budget); + + case '*': + case ':': + { + size_t stops[64], n = 0, j = i, take; + stops[n++] = j; + while (n < sizeof stops / sizeof *stops && j > lo + && (take = class_match_back (c, RC_HAT, in, lo, j)) != 0) + stops[n++] = j -= take; + while (n-- > (a->tok == '*' ? 1u : 0u)) + if (ctx_left (r, in, lo, stops[n], k - 1, budget)) + return 1; + return 0; + } + + case '~': + { + size_t j = i, n = 0; + while (j > lo && is_nonprinting ((unsigned char) in[j - 1])) + { + j--; + n++; + } + for (; n; n--) + if (ctx_left (r, in, lo, i - n, k - 1, budget)) + return 1; + return 0; + } + + case '|': + { + size_t stops[64], n = 0, j = i, take; + while (n < sizeof stops / sizeof *stops && j > lo) + { + if (in[j - 1] == ',') + { + j--; + continue; + } + take = class_match_back (c, RC_BSLASH, in, lo, j); + if (!take) + break; + j -= take; + stops[n++] = j; + } + while (n--) + if (ctx_left (r, in, lo, stops[n], k - 1, budget)) + return 1; + return 0; + } + + case '%': + { + /* The suffix ends where the fragment begins, so the "and then a + non-letter" half of Table 22's definition is about the fragment and + is left to the fragment. */ + const char *list = rc_class (c, RC_PCT); + for (; list && *list; list += strlen (list) + 1) + { + size_t mlen = strlen (list); + if (mlen > avail || !same_chars (in + i - mlen, mlen, list)) + continue; + if (ctx_left (r, in, lo, i - mlen, k - 1, budget)) + return 1; + } + return 0; + } + + default: + { + int idx = tok_class (a->tok); + size_t take; + if (idx < 0 || !avail) + return 0; + take = class_match_back (c, idx, in, lo, i); + if (!take) + return 0; + return ctx_left (r, in, lo, i - take, k - 1, budget); + } + } +} + +/* The whole exception at in[i]. Sets *mlen to the fragment's length in bytes + -- which is what the scanner consumes; the contexts are only looked at. */ +static int +match_rule_at (const rule *r, const char *in, size_t lo, size_t i, + size_t end, size_t *mlen) +{ + size_t j = i, k; + + for (k = 0; k < r->nfrag; k++) + { + const ratom *a = &r->frag[k]; + size_t cl; + if (j >= end) + return 0; + cl = uclen (in + j, end - j); + if (a->tok == '`') /* the wildcard is legal inside a fragment */ + j += cl; + else if (atom_is (r, a, in + j, cl)) + j += cl; + else + return 0; + } + + if (r->nleft) + { + int budget = RC_CTX_BUDGET; + if (!ctx_left (r, in, lo, i, r->nleft, &budget)) + return 0; + } + if (r->nright) + { + int budget = RC_CTX_BUDGET; + if (!ctx_right (r, in, j, end, 0, &budget)) + return 0; + } + + *mlen = j - i; + return 1; +} + +static int +add_rule (rcdict *d, rule *r) +{ + unsigned char b; + if (d->n == d->cap) + { + size_t nc = d->cap ? d->cap * 2 : 32; + rule **nr = realloc (d->rules, nc * sizeof *nr); + if (!nr) + return 0; + d->rules = nr; + d->cap = nc; + } + r->seq = d->n; + d->rules[d->n++] = r; + if (r->kind == M_RULE) + d->has_rules = 1; + + if (r->kind == M_REGEX) + { + if (d->rtail) + d->rtail->rnext = r; + else + d->rhead = r; + d->rtail = r; + r->rseq = d->nregex++; + d->has_regex = 1; + return 1; + } + + if (r->kind == M_RULE && r->anypos) + { + if (d->atail) + d->atail->anext = r; + else + d->ahead = r; + d->atail = r; + d->has_anypos = 1; + return 1; + } + + /* An exception is bucketed by the first byte of its TEXT FRAGMENT, not of + the pattern -- "$(e)r^" is reached from an e in the input, not from a + dollar sign. */ + b = (unsigned char) (r->kind == M_RULE ? r->frag[0].lit[0] : r->pat[0]); + if (b >= 'A' && b <= 'Z') + b = (unsigned char) (b + 32); + if (d->tail[b]) + d->tail[b]->next = r; + else + d->head[b] = r; + d->tail[b] = r; + return 1; +} + +int +rcdict_add_text (rcdict *d, const char *src, size_t len, const char *name, + rcdict_report report, void *ctx) +{ + size_t i = 0, line = 0; + int added = 0, default_nocase = 1; + + if (!d || !src) + return 0; + + while (i < len) + { + size_t ls = i, le, off; + const char *p, *raw; + size_t plen, rawlen; + const char *sep; + + while (i < len && src[i] != '\n') + i++; + le = i; + if (le > ls && src[le - 1] == '\r') + le--; /* files written on Windows */ + if (i < len) + i++; + line++; + + off = ls; + + /* Two views of the line. raw keeps its tabs, including a trailing one, + because a trailing tab is an empty last column and trimming it away + would silently turn a four-column line into a three-column one -- and + so change which field is the pattern. p is fully trimmed, and is what + the emptiness, comment and directive tests use. */ + raw = src + ls; + rawlen = le - ls; + while (rawlen && raw[0] == ' ') + { + raw++; + rawlen--; + } + while (rawlen && raw[rawlen - 1] == ' ') + rawlen--; + + p = raw; + plen = rawlen; + trim (&p, &plen); + + if (!plen || *p == ';') + continue; + + if (plen > RCDICT_MAX_LINE) + { + report_line (report, ctx, off, name, line, "line too long, skipped"); + continue; + } + + /* Directives. */ + if (plen > 2 && p[0] == '#' && p[1] == '!') + { + const char *a = p + 2; + size_t alen = plen - 2; + trim (&a, &alen); + if (alen >= 4 && !memcmp (a, "case", 4)) + { + const char *v = a + 4; + size_t vlen = alen - 4; + trim (&v, &vlen); + if (vlen && (v[0] == 's' || v[0] == 'S')) + default_nocase = 0; + else if (vlen && (v[0] == 'i' || v[0] == 'I')) + default_nocase = 1; + else + report_line (report, ctx, off, name, line, + "#!case wants 'sensitive' or 'insensitive'"); + } + else if (alen >= 7 && !memcmp (a, "rcdict", 6)) + { + const char *v = a + 6; + size_t vlen = alen - 6; + trim (&v, &vlen); + if (vlen != 1 || v[0] != '1') + report_line (report, ctx, off, name, line, + "unknown #!rcdict version; reading it as 1"); + } + else if (alen >= 4 && !memcmp (a, "lang", 4)) + { + /* Recorded by convention, not yet acted on. */ + } + else if (alen > 5 && !memcmp (a, "class", 5) + && is_space ((unsigned char) a[5])) + { + /* #!class ... + * + * Redefines one Table 22 character class for every rule loaded + * after it. Members are separated by spaces so that a class can + * hold the two-character ones the datasheet gives @ and & -- and + * so that a suffix class is writable at all. + * + * This is what makes the rule language usable outside English. + * Table 22's own vowel list is "aeiouy", and a Spanish + * dictionary that inherits it does not merely misplace a stress: + * an accented letter falls out of the alphabet altogether, so $ + * matches inside a word and the rules meant for single letters + * start firing. "mas" comes out spelled aloud. */ + const char *v = a + 5; + size_t vlen = alen - 5; + const char *tokp; + trim (&v, &vlen); + if (!vlen || !(tokp = strchr (rc_tokens, v[0])) || !v[0]) + report_line (report, ctx, off, name, line, + "#!class wants a Table 22 token " + "(one of #+^*:?@!%&$~\\|`) and its members"); + else if (!rc_defaults[tokp - rc_tokens]) + report_line (report, ctx, off, name, line, + "that class is structural and cannot be " + "redefined; $ follows # and ^, and * and : " + "follow ^"); + else + { + size_t idx = (size_t) (tokp - rc_tokens); + const char *m = v + 1; + size_t mlen = vlen - 1, w = 0; + char *set; + trim (&m, &mlen); + /* The member list, rewritten as NUL-separated strings with + an empty one on the end. */ + set = malloc (mlen + 2); + if (!set) + return added; + { + size_t q = 0; + while (q < mlen) + { + size_t s0; + while (q < mlen && is_space ((unsigned char) m[q])) + q++; + s0 = q; + while (q < mlen && !is_space ((unsigned char) m[q])) + q++; + if (q > s0) + { + memcpy (set + w, m + s0, q - s0); + w += q - s0; + set[w++] = 0; + } + } + set[w] = 0; + } + if (!w) + { + free (set); + report_line (report, ctx, off, name, line, + "#!class with no members; a class may not " + "be empty"); + } + else + { + rclasses *nc = rc_classes_own (d->classes); + if (!nc) + { + free (set); + return added; + } + d->classes = nc; + free (nc->set[idx]); + nc->set[idx] = set; + } + } + } + else + report_line (report, ctx, off, name, line, "unknown directive"); + continue; + } + + /* A rule, in either of two spellings. + * + * Tab-separated columns, which is the preferred one because a tab cannot + * appear in any of the fields, so the pattern may contain spaces, '=' + * and anything else without escaping: + * + * type pattern output + * type flags pattern output + * + * Or, when the line has no tabs at all, the pattern and output are + * separated by an '=' with whitespace either side and the flags ride on + * the type after a colon: + * + * type[:flags] pattern = output + * + * The second exists because tabs are invisible and editors, config + * dialogs and web forms convert them to spaces without saying so. A + * dictionary that has been through one of those still loads. + */ + { + const char *t = p, *pat = NULL, *out = NULL, *flags = NULL; + size_t tlen = 0, patlen = 0, outlen = 0, flagslen = 0; + rule *r; + int nocase = default_nocase, caps = 0, asked_nocase = 0; + match_kind kind; + int tabbed = memchr (raw, '\t', rawlen) != NULL; + + if (tabbed) + { + const char *f[5]; + size_t fl[5]; + int nf = 0; + const char *q = raw, *e = raw + rawlen; + + while (nf < 5) + { + const char *tab = memchr (q, '\t', (size_t) (e - q)); + f[nf] = q; + fl[nf] = tab ? (size_t) (tab - q) : (size_t) (e - q); + trim (&f[nf], &fl[nf]); + nf++; + if (!tab) + break; + q = tab + 1; + } + + if (nf > 4) + { + report_line (report, ctx, off, name, line, + "too many tab-separated columns; want " + "type, [flags,] pattern, output"); + continue; + } + if (nf < 2) + { + report_line (report, ctx, off, name, line, + "a rule needs at least a type and a pattern"); + continue; + } + + t = f[0]; + tlen = fl[0]; + if (nf == 4) + { + flags = f[1]; + flagslen = fl[1]; + pat = f[2]; + patlen = fl[2]; + out = f[3]; + outlen = fl[3]; + /* Catch a stray tab inside what was meant to be the pattern, + which would otherwise be read as a column of nonsense + flags. */ + { + size_t k; + for (k = 0; k < flagslen; k++) + if (flags[k] != 'i' && flags[k] != 'c' && flags[k] != 'C') + break; + if (k < flagslen) + { + report_line (report, ctx, off, name, line, + "four columns, but the second is not " + "flags (want i, c or C, or leave it " + "empty) -- is there a tab in the " + "pattern?"); + continue; + } + } + } + else /* nf == 2 or 3 */ + { + pat = f[1]; + patlen = fl[1]; + if (nf == 3) + { + out = f[2]; + outlen = fl[2]; + } + else + { + out = pat + patlen; /* empty output: a silent match */ + outlen = 0; + } + } + } + else + while (tlen < plen && !is_space ((unsigned char) t[tlen]) + && t[tlen] != ':') + tlen++; + + /* Flags ride on the type after a colon in the untabbed spelling, and may + do so in the tabbed one too. */ + { + const char *colon = memchr (t, ':', tlen); + if (colon) + { + if (!flags) + { + flags = colon + 1; + flagslen = tlen - (size_t) (colon - t) - 1; + } + tlen = (size_t) (colon - t); + } + } + + if (tlen == 4 && !memcmp (t, "word", 4)) + kind = M_WORD; + else if (tlen == 4 && !memcmp (t, "text", 4)) + kind = M_TEXT; + else if (tlen == 5 && !memcmp (t, "regex", 5)) + kind = M_REGEX; + else if (tlen == 4 && !memcmp (t, "rule", 4)) + kind = M_RULE; + else + { + report_line (report, ctx, off, name, line, + "line does not start with a matcher " + "(word, text, regex or rule)"); + continue; + } + + + if (!tabbed) + { + const char *rest = t + tlen; + size_t restlen = plen - tlen; + + if (restlen && *rest == ':') + { /* type:flags -- the type scan stopped here */ + rest++; + restlen--; + flags = rest; + flagslen = 0; + while (flagslen < restlen + && !is_space ((unsigned char) rest[flagslen])) + flagslen++; + rest += flagslen; + restlen -= flagslen; + } + + sep = find_separator (rest, restlen); + if (!sep) + { + report_line (report, ctx, off, name, line, + "no ' = ' between pattern and output " + "(or separate the columns with tabs)"); + continue; + } + pat = rest; + patlen = (size_t) (sep - rest); + out = sep + 1; + outlen = (size_t) (rest + restlen - out); + trim (&pat, &patlen); + trim (&out, &outlen); + } + + { + size_t k; + for (k = 0; k < flagslen; k++) + switch (flags[k]) + { + case 'i': + nocase = 1; + caps = 0; + asked_nocase = 1; + break; + case 'c': + nocase = 0; + caps = 0; + break; + case 'C': + caps = 1; + nocase = 1; + break; + default: + report_line (report, ctx, off, name, line, + "unknown flag; want i, c or C"); + } + } + + if (!patlen) + { + report_line (report, ctx, off, name, line, + "empty pattern (columns are type, [flags,] " + "pattern, output)"); + continue; + } + + r = calloc (1, sizeof *r); + if (!r) + return added; + r->kind = kind; + r->nocase = nocase; + r->caps_only = caps; + r->line = line; + if (name) + { + r->origin = malloc (strlen (name) + 1); + if (r->origin) + memcpy (r->origin, name, strlen (name) + 1); + } + r->pat = pattern_dup (pat, patlen, !tabbed, &r->patlen); + if (!r->pat || !r->patlen) + { + free_rule (r); + report_line (report, ctx, off, name, line, + "empty pattern (columns are type, [flags,] " + "pattern, output)"); + continue; + } + if (kind == M_RULE) + { + if (!compile_rule (r, r->pat, r->patlen)) + { + free_rule (r); + report_line (report, ctx, off, name, line, + "a rule wants the datasheet's L(F)R form, with " + "the text fragment in parentheses -- e.g. " + "$(e)r^ or (ch)"); + continue; + } + r->cls = d->classes; + if (r->cls->refs >= 0) + r->cls->refs++; + } + + if (kind == M_REGEX) + { + /* Remimu has no case-insensitive mode, and lowercasing a pattern + is not safe -- it would turn \\W into \\w and \\B into \\b. So + regex rules match case-sensitively whatever the file default + says, and only an explicit request gets a complaint. */ + if (asked_nocase) + report_line (report, ctx, off, name, line, + "regex patterns always match case-sensitively; " + "write a class such as [Nn] instead of :i"); + r->nocase = 0; + + r->re = rcdict_regex_compile (r->pat); + if (!r->re) + { + free_rule (r); + report_line (report, ctx, off, name, line, + "regex does not compile (or is too long)"); + continue; + } + } + + if (!parse_template (r, d->profile, out, outlen, name, line, + (size_t) (out - src), report, ctx) + || !add_rule (d, r)) + { + free_rule (r); + continue; + } + added++; + } + } + return added; +} + +int +rcdict_add_file (rcdict *d, const char *path, rcdict_report report, void *ctx) +{ + FILE *f; + char *buf; + size_t cap = 0, len = 0, got; + int added; + + if (!d || !path) + return 0; + f = fopen (path, "rb"); + if (!f) + { + if (report) + { + char msg[300]; + snprintf (msg, sizeof msg, "cannot open '%s'", path); + report (ctx, 0, msg); + } + return 0; + } + + cap = 8192; + buf = malloc (cap); + if (!buf) + { + fclose (f); + return 0; + } + while ((got = fread (buf + len, 1, cap - len, f)) > 0) + { + len += got; + if (len == cap) + { + char *nb = realloc (buf, cap * 2); + if (!nb) + break; + buf = nb; + cap *= 2; + } + } + fclose (f); + + added = rcdict_add_text (d, buf, len, path, report, ctx); + free (buf); + return added; +} + +int +rcdict_add_path_list (rcdict *d, const char *list, char sep, + rcdict_report report, void *ctx) +{ + int added = 0; + const char *p = list; + + if (!d || !list) + return 0; + while (*p) + { + const char *e = strchr (p, sep); + size_t n = e ? (size_t) (e - p) : strlen (p); + if (n) + { + char path[1024]; + if (n < sizeof path) + { + memcpy (path, p, n); + path[n] = 0; + added += rcdict_add_file (d, path, report, ctx); + } + } + if (!e) + break; + p = e + 1; + } + return added; +} + +/* --- the substitution rules ----------------------------------------------- */ + +/* A phoneme span breaks the letter-to-sound stage's context on both sides of + itself. That one fact is the whole of what follows. Measured in + dict-lab/RESULTS.md; with both rules applied the output is byte-identical to + what text mode produces, at 1S, 5S and 9S alike. + * + * Right edge: if '.' or ',' follows the span, the chip no longer gives the + * final word its sentence-final fall and rate halving (notes.md §12) -- 14709 + * samples become 12900. Moving the punctuation inside the span as its Table 5 + * pause phoneme restores it exactly. + * + * Left edge: the article "a" loses its reduction, coming out as 0x0c (EY) + * rather than 0x05 (AX). Absorbing it into the span restores that too. It is + * the only word of fifty tested that does this -- "an", "the", "to", "of" and + * forty-six others are untouched -- so this is a special case for one word and + * not a heuristic. + * + * '!' '?' ';' and ':' have no Table 5 pause phoneme, so there is nothing to + * move and the rising terminal (0x2f) becomes a falling one (0x5c). The + * substitution is still made -- a right word with a statement's intonation + * beats a wrong word -- and the loss is reported so that a caller can see it. + * Rebuilding the rise by hand is possible (Table 6's '/' emits the marker + * where you write it) but needs stress placement that is not derivable from an + * arbitrary phoneme string, so it is not done automatically. + */ + +static int +absorbable_punct (const rcdict_profile *p, int c) +{ + return (c == '.' || c == ',') && strchr (p->pauses, c) != NULL; +} + +static int +lossy_punct (int c) +{ + return c == '!' || c == '?' || c == ';' || c == ':'; +} + +/* Does in[ws..we) end with the article "a", standing as its own word? */ +static int +ends_with_article (const char *in, size_t ws, size_t we, size_t *word_start) +{ + size_t e = we, s; + while (e > ws && is_space ((unsigned char) in[e - 1])) + e--; + if (e == we) /* no whitespace between the word and the span */ + return 0; + s = e; + while (s > ws && !is_space ((unsigned char) in[s - 1])) + s--; + if (e - s != 1 || upper ((unsigned char) in[s]) != 'A') + return 0; + *word_start = s; + return 1; +} + +static void +emit_span (sink * out, const rcdict_options *opt, + const char *inner, size_t innerlen, + int absorb_article, int pause_char, int continuing) +{ + const rcdict_profile *p = opt->profile; + size_t s = 0, e = innerlen; + + while (s < e && is_space ((unsigned char) inner[s])) + s++; + while (e > s && is_space ((unsigned char) inner[e - 1])) + e--; + + /* Continuing the span that is already open: take back the " T" that + closed it instead of closing and reopening. A letter-to-sound dictionary + matches every character of every word, so without this the RCStudio + Spanish sample produces 572 one-phoneme spans where it wants about ten, + each paying for two mode switches. */ + if (continuing) + unput (out, strlen (p->leave_phoneme) + 1); + else + puts_ (out, p->enter_phoneme); + if (absorb_article) + puts_ (out, " AX"); + if (e > s) + { + putc_ (out, ' '); + put (out, inner + s, e - s); + } + if (pause_char) + { + putc_ (out, ' '); + putc_ (out, (char) pause_char); + } + putc_ (out, ' '); + puts_ (out, p->leave_phoneme); +} + +/* --- matching ------------------------------------------------------------- */ + +static int +all_capitals (const char *s, size_t len) +{ + size_t i; + int seen = 0; + for (i = 0; i < len; i++) + { + unsigned char c = (unsigned char) s[i]; + if (c >= 'a' && c <= 'z') + return 0; + if (c >= 'A' && c <= 'Z') + seen = 1; + } + return seen; +} + +static int +same (const char *a, const char *b, size_t n, int nocase) +{ + size_t i; + if (!nocase) + return memcmp (a, b, n) == 0; + for (i = 0; i < n; i++) + if (upper ((unsigned char) a[i]) != upper ((unsigned char) b[i])) + return 0; + return 1; +} + +/* First rule, in load order, that matches at in[i]. run_start and run_end + bound the text run: a match may not reach outside it, because what lies + beyond is a command atom or an utterance boundary. + + That bound is also what makes the word-boundary test right. A word rule + asks whether the byte before the match is alphanumeric, and at the start of + a run there is no such byte to ask about -- a command atom sitting there + ends the word as surely as a space does, even though its last byte ("I" of + an index marker) is a letter. */ +static const rule * +match_at (const rcdict *d, const char *in, size_t run_start, size_t i, + size_t run_end, const char *runz, unsigned char *giveup, + size_t *mlen, rcap *caps, rcdict_report report, void *ctx) +{ + unsigned char b = (unsigned char) in[i]; + const rule *best = NULL, *r; + size_t k; + + if (b >= 'A' && b <= 'Z') + b = (unsigned char) (b + 32); + + for (r = d->head[b]; r; r = r->next) + { + if (r->kind == M_RULE) + { + size_t n = 0; + if (!match_rule_at (r, in, run_start, i, run_end, &n) || !n) + continue; + if (r->caps_only && !all_capitals (in + i, n)) + continue; + best = r; + *mlen = n; + break; + } + if (r->patlen > run_end - i) + continue; + if (!same (in + i, r->pat, r->patlen, r->nocase)) + continue; + if (r->caps_only && !all_capitals (in + i, r->patlen)) + continue; + if (r->kind == M_WORD) + { + if (i > run_start && is_word_char ((unsigned char) in[i - 1])) + continue; + if (i + r->patlen < run_end + && is_word_char ((unsigned char) in[i + r->patlen])) + continue; + } + best = r; + *mlen = r->patlen; + break; /* the bucket is in load order: first wins */ + } + + /* Regex rules live in their own list, so first-match-wins has to be settled + between the two by load order. A regex later in the file than a literal + that already matched cannot win, so the search stops there. */ + if (d->has_regex && runz) + for (r = d->rhead; r; r = r->rnext) + { + long n; + size_t cs[RCDICT_NCAPS], cl[RCDICT_NCAPS]; + + if (best && r->seq > best->seq) + break; + if (giveup && (giveup[r->rseq >> 3] & (1u << (r->rseq & 7)))) + continue; /* this one already blew its budget */ + + n = rcdict_regex_match (r->re, runz, i - run_start, RCDICT_NCAPS, + cs, cl); + + /* Giving up is bounded per attempt, but the scanner tries every + position, so a catastrophic pattern would still cost the limit + times the length of the run -- 1.7 seconds of dead air on an + utterance-sized one, measured. Retire the rule for the rest of + this call instead: the damage is then one budget, once, and the + user is told which line to look at. */ + if (n == RCDICT_REGEX_GAVEUP) + { + if (giveup) + giveup[r->rseq >> 3] |= (unsigned char) (1u << (r->rseq & 7)); + if (report) + { + char msg[256]; + snprintf (msg, sizeof msg, + "%s:%lu: regex gave up (runaway backtracking); " + "the rule is ignored for the rest of this text", + r->origin ? r->origin : "", + (unsigned long) r->line); + report (ctx, i, msg); + } + continue; + } + + /* A zero-length match would leave the scanner where it was, so it is + refused rather than allowed to spin. */ + if (n <= 0) + continue; + if ((size_t) n > run_end - i) + continue; /* cannot happen: runz ends at run_end */ + if (r->caps_only && !all_capitals (in + i, (size_t) n)) + continue; + + best = r; + *mlen = (size_t) n; + if (caps) + for (k = 1; k < RCDICT_NCAPS; k++) + if (cs[k] != RCDICT_REGEX_UNSET) + { + caps[k].start = run_start + cs[k]; + caps[k].len = cl[k]; + } + break; + } + + /* Exceptions whose fragment has no literal first character are not in any + bucket and are tried everywhere, settled against the others by load order + exactly as the regex rules are. */ + if (d->has_anypos) + for (r = d->ahead; r; r = r->anext) + { + size_t n = 0; + if (best && r->seq > best->seq) + break; + if (!match_rule_at (r, in, run_start, i, run_end, &n)) + continue; + /* A null text fragment matches without consuming anything, and the + scanner cannot stand still. It takes one character instead, which + is the only reading under which the datasheet's own "()=" -- placed + last so that unmatched characters are ignored rather than falling + through to the built in rules -- does what it says. */ + if (!n) + n = uclen (in + i, run_end - i); + if (r->caps_only && !all_capitals (in + i, n)) + continue; + best = r; + *mlen = n; + break; + } + + if (best && caps) + { + caps[0].start = i; /* \\0 is always the whole match */ + caps[0].len = *mlen; + } + return best; +} + +/* --- emitting a rule's output --------------------------------------------- */ + +/* The absorption rules are about the phoneme span's edges, and a template can + put text either side of one, so they apply to the FIRST segment (does it + start with phonemes?) and the LAST (does it end with them?). A template + that is a single phoneme span -- which is what the inline escape is -- gets + both, which is how the two paths stay consistent. */ +static void +emit_rule (sink * out, const rcdict_options *opt, const rule *r, + const char *in, size_t run_start, size_t mstart, size_t mend, + size_t run_end, const rcap *caps, size_t *consumed_to, + int *span_open, int may_continue, + rcdict_report report, void *ctx) +{ + const rcdict_profile *p = opt->profile; + size_t k; + int absorb = 0, pause = 0; + size_t article_at = 0; + /* Does this output pick up a span the previous match left open? */ + int cont = may_continue && span_open && *span_open + && r->nsegs && r->segs[0].kind == SEG_PHONEME; + + *consumed_to = mend; + + /* Absorbing the article ahead of the span is about the span's LEFT EDGE + breaking the letter-to-sound context, and there is no such edge when the + span was already open. */ + if (!cont && r->nsegs && r->segs[0].kind == SEG_PHONEME + && ends_with_article (in, run_start, mstart, &article_at)) + { + absorb = 1; + unput (out, mstart - article_at); + } + + if (r->nsegs && r->segs[r->nsegs - 1].kind == SEG_PHONEME) + { + size_t k2 = mend; + while (k2 < run_end && is_space ((unsigned char) in[k2])) + k2++; + if (k2 < run_end && absorbable_punct (p, (unsigned char) in[k2])) + { + pause = (unsigned char) in[k2]; + *consumed_to = k2 + 1; + } + else if (k2 < run_end && lossy_punct ((unsigned char) in[k2]) + && !(opt->dict && opt->dict->has_rules)) + /* Not warned about when the dictionary has exception rules in it: one + of those covers every character by construction, punctuation + included, so the '?' about to be complained of is on its way to a + pause phoneme of the dictionary's own choosing. */ + report_at (report, ctx, k2, + "span ends before ! ? ; or : which has no Table 5 pause " + "phoneme: the rising terminal is lost"); + } + + for (k = 0; k < r->nsegs; k++) + { + const seg *sg = &r->segs[k]; + switch (sg->kind) + { + case SEG_TEXT: + put (out, sg->s, sg->len); + break; + case SEG_PHONEME: + emit_span (out, opt, sg->s, sg->len, + absorb && k == 0, k == r->nsegs - 1 ? pause : 0, + cont && k == 0); + break; + case SEG_COMMAND: + putc_ (out, (char) p->cmd_char); + put (out, sg->s, sg->len); + break; + case SEG_BACKREF: + /* \0 is the whole match, for any matcher. \1-\9 are the groups a + regex captured; after any other matcher there are none, and they + expand to nothing rather than to something invented (which the + loader has already said out loud). */ + if (caps && sg->n < (int) RCDICT_NCAPS && caps[sg->n].len) + put (out, in + caps[sg->n].start, caps[sg->n].len); + else if (sg->n == 0) + put (out, in + mstart, mend - mstart); + break; + } + } + + /* A rule with no output at all -- the silent (h)= of a Spanish dictionary -- + emits nothing, so whatever the span was doing it is still doing. + + An absorbed pause does not close the span either. What Experiment 1 + measured is that the punctuation must be INSIDE the span (C3/C4), and it + still is; and a dictionary whose own rules cover punctuation reaches the + same place by the other path, "(,)=," matching and coalescing. The two + paths agreeing matters more than the caution would have bought. */ + if (span_open && r->nsegs) + *span_open = r->segs[r->nsegs - 1].kind == SEG_PHONEME; +} + +/* --- the scanner ---------------------------------------------------------- */ + +/* Length of the command atom starting at in[i], or 0 if this is not one. + *newcmd is set when the atom changes the command character, *zap when it is + the Zap command, which stops the chip honouring commands at all -- after + which emitting a mode switch would have the chip speak "D" aloud. */ +static size_t +command_atom (const char *in, size_t i, size_t len, unsigned char cmd, + int *newcmd, int *zap) +{ + size_t j; + + if (i >= len || (unsigned char) in[i] != cmd) + return 0; + if (i + 1 >= len) + return 1; /* trailing lead-in byte; pass it through */ + + { + unsigned char c = (unsigned char) in[i + 1]; + + /* Doubled: the command character itself, to be spoken. */ + if (c == cmd) + return 2; + + /* A different control character re-arms the lead-in as that character. */ + if (c >= 0x01 && c <= 0x1a) + { + *newcmd = c; + return 2; + } + } + + /* . */ + j = i + 1; + while (j < len && is_digit ((unsigned char) in[j])) + j++; + if (j < len && is_alpha ((unsigned char) in[j])) + { + if (upper ((unsigned char) in[j]) == 'Z') + *zap = 1; + return j - i + 1; + } + if (j < len && ((unsigned char) in[j] == '?' || (unsigned char) in[j] == '*' + || (unsigned char) in[j] == '@')) + return j - i + 1; /* the query, DTMF and reinitialise commands */ + + return 1; /* malformed; hand the byte over untouched */ +} + +/* One run of ordinary text, bounded by whatever a rewrite must not cross. + Both the inline escape and the dictionary are applied here, and both go + through emit_rule, so the absorption rules cannot drift apart between + them. */ +static void +expand_run (sink * out, const rcdict_options *opt, const char *in, + size_t start, size_t end, unsigned char *giveup, + rcdict_report report, void *ctx) +{ + const rcdict_profile *p = opt->profile; + size_t i = start; + size_t openlen = opt->open ? strlen (opt->open) : 0; + size_t closelen = opt->close ? strlen (opt->close) : 0; + + /* The regex engine takes a NUL-terminated string and no length, so a run + has to be handed over as its own copy. That is the right shape anyway: + `^`, `$` and `\b` should see the edges of the speakable run and not the + command bytes on either side of it, and without the copy a pattern could + match straight through an index marker. + + Built once per run, and only when the dictionary actually has a regex in + it, so the common case allocates nothing. */ + char stackbuf[1024]; + char *runz = NULL; + const rcdict *dict = opt->dict; + /* Whether the output currently ends in an open phoneme span, and the input + offset at which it was left open. Consecutive matches that both produce + phonemes with nothing between them are joined into one span. */ + int span_open = 0; + size_t span_at = start; + + if (dict && dict->has_regex && end > start) + { + size_t n = end - start; + runz = (n < sizeof stackbuf) ? stackbuf : malloc (n + 1); + if (runz) + { + memcpy (runz, in + start, n); + runz[n] = 0; + } + } + + while (i < end) + { + if (opt->inline_phonemes && openlen && closelen + && i + openlen <= end && memcmp (in + i, opt->open, openlen) == 0) + { + size_t body = i + openlen, endq = body, consumed; + seg one; + rule tmp; + + while (endq < end + && !(endq + closelen <= end + && memcmp (in + endq, opt->close, closelen) == 0)) + endq++; + + if (endq + closelen > end) + { + /* Unterminated: not a span at all. Pass the delimiter through + as text rather than swallowing the rest of the utterance. */ + report_at (report, ctx, i, "unterminated phoneme escape"); + put (out, in + i, openlen); + i += openlen; + continue; + } + + /* Fail loudly on a bad symbol: speak the contents as ordinary text, + which is audible and obviously wrong, rather than sending the chip + into phoneme mode with something it will not say. */ + if (!rcdict_check_phonemes (p, in + body, endq - body, report, ctx)) + { + put (out, in + body, endq - body); + i = endq + closelen; + continue; + } + + /* A one-segment rule, so this takes exactly the same path a + dictionary entry would. emit_rule only reads the segment. */ + memset (&tmp, 0, sizeof tmp); + memset (&one, 0, sizeof one); + one.kind = SEG_PHONEME; + one.s = (char *) (size_t) (in + body); + one.len = endq - body; + tmp.segs = &one; + tmp.nsegs = 1; + + emit_rule (out, opt, &tmp, in, start, i, endq + closelen, end, + NULL, &consumed, &span_open, span_open && span_at == i, + report, ctx); + span_at = consumed; + i = consumed; + continue; + } + + if (dict && dict->n) + { + rcap caps[RCDICT_NCAPS]; + size_t mlen = 0, k; + const rule *r; + + for (k = 0; k < RCDICT_NCAPS; k++) + { + caps[k].start = 0; + caps[k].len = 0; + } + r = match_at (dict, in, start, i, end, runz, giveup, &mlen, caps, + report, ctx); + if (r && mlen) + { + size_t consumed; + emit_rule (out, opt, r, in, start, i, i + mlen, end, caps, + &consumed, &span_open, span_open && span_at == i, + report, ctx); + span_at = consumed; + i = consumed; + continue; + } + } + + /* Anything passed through as text closes the span. */ + span_open = 0; + putc_ (out, in[i]); + i++; + } + + if (runz && runz != stackbuf) + free (runz); +} + +size_t +rcdict_expand (const rcdict_options *opt, const char *in, size_t inlen, + char *out_buf, size_t outcap, rcdict_report report, void *ctx) +{ + sink out; + size_t i = 0; + unsigned char cmd; + int zapped = 0; + /* One bit per regex rule, marking the ones that have blown their + backtracking budget during this call; see match_at. */ + unsigned char stack_gu[64], *giveup = NULL; + + if (!opt || !opt->profile || (!in && inlen)) + return 0; + cmd = opt->profile->cmd_char; + + if (opt->dict && opt->dict->nregex) + { + size_t nb = (opt->dict->nregex + 7) / 8; + giveup = (nb <= sizeof stack_gu) ? stack_gu : calloc (nb, 1); + if (giveup == stack_gu) + memset (stack_gu, 0, nb); + } + + out.buf = (out_buf && outcap) ? out_buf : NULL; + out.cap = out.buf ? outcap : 0; + out.len = 0; + out.need = 0; + + while (i < inlen) + { + unsigned char c = (unsigned char) in[i]; + int newcmd = 0; + size_t atom, run_end; + + /* An opaque command atom: copied verbatim, never matched into, and never + matched across. A substitution that spanned one would not mispronounce + a word, it would corrupt a command -- a mangled index marker changes + the chip's mode or eats the next byte as a parameter. */ + atom = command_atom (in, i, inlen, cmd, &newcmd, &zapped); + if (atom) + { + put (&out, in + i, atom); + if (newcmd) + cmd = (unsigned char) newcmd; + i += atom; + continue; + } + + /* CTRL+^ restores command recognition after Zap. */ + if (c == 0x1e) + { + zapped = 0; + putc_ (&out, (char) c); + i++; + continue; + } + + /* An utterance boundary -- CR or NUL, the two bytes that make the chip + speak what it has. Nothing is rewritten across one. */ + if (c == '\r' || c == 0) + { + putc_ (&out, (char) c); + i++; + continue; + } + + /* A run of ordinary text, ending at the next thing a rewrite must not + cross. */ + run_end = i; + while (run_end < inlen) + { + unsigned char c2 = (unsigned char) in[run_end]; + int d1 = 0, d2 = 0; + if (c2 == '\r' || c2 == 0 || c2 == 0x1e) + break; + if (command_atom (in, run_end, inlen, cmd, &d1, &d2)) + break; + run_end++; + } + + /* Under Zap the chip speaks commands instead of obeying them, so a mode + switch would be read out. Rewriting stops entirely rather than being + filtered down to the entries that happen to be text-only: Zap is rare, + and "some of your dictionary applies" is a worse thing to explain than + "none of it does". */ + if (zapped) + put (&out, in + i, run_end - i); + else + expand_run (&out, opt, in, i, run_end, giveup, report, ctx); + i = run_end; + } + + if (giveup && giveup != stack_gu) + free (giveup); + if (out.buf && out.cap) + out.buf[out.len < out.cap ? out.len : out.cap - 1] = 0; + return out.need; +} diff --git a/doubletalk/rcdict/rcdict.h b/doubletalk/rcdict/rcdict.h new file mode 100644 index 0000000..b56ccd9 --- /dev/null +++ b/doubletalk/rcdict/rcdict.h @@ -0,0 +1,292 @@ +/* SPDX-License-Identifier: BSD-3-Clause */ +/* rcdict --- portable pronunciation dictionaries for the RC Systems + * text-to-speech engines. + * + * This module is deliberately free of any dependency on either emulator: it + * takes a byte stream on its way to the chip and returns the byte stream that + * should go instead, with pronunciations substituted. It is linked by a GPL-3 + * host and by a BSD-3 one, which is why it is BSD-3-Clause and must stay that + * way -- a GPL-3 file could never move in the other direction. + * + * Targets differ only in trivia. RC8650 datasheet Table 5 is captioned + * "DoubleTalk Phoneme Symbols", so the phoneme set, the Table 6 attribute + * modifiers, the D/T/C mode commands, the CTRL+A command character and the nI + * index markers are common to both; a profile carries the rest. + * + * rcdict_options o; + * rcdict_options_init (&o, &rcdict_rc8650); + * o.inline_phonemes = 1; + * + * size_t n = rcdict_expand (&o, text, len, NULL, 0, NULL, NULL); + * char *buf = malloc (n + 1); + * rcdict_expand (&o, text, len, buf, n + 1, NULL, NULL); + * + * The input is NOT plain text. By the time a screen reader's driver calls + * this, index markers and prosody commands are already embedded in it, and + * rewriting across one of those would corrupt a command rather than merely + * mispronounce a word. Everything here is built around that: command + * sequences are opaque atoms, a substitution never spans one, and the scanner + * tracks the two things that can change how the stream is read -- the command + * character itself, and Zap. + * + * --- the dictionary format ------------------------------------------------ + * + * Line-oriented UTF-8. Comment lines begin with ';', the convention the + * RC8650 datasheet sets for its own exception dictionaries, and they must be + * lines of their own -- so a ';' inside a pattern needs no escaping. + * + * COLUMNS ARE SEPARATED BY TABS: + * + * type flags pattern output + * type pattern output (no flags column) + * type pattern (and no output: a silent match) + * + * #!rcdict 1 + * #!case insensitive ; the file's default; 'sensitive' is the other + * #!lang en + * + * word NVDA [EH N V IY D IY EY] + * word Sean Shon + * word C US [Y UW EH S] + * text Mbps megabits per second + * rule C(O)N [AA] + * + * A tab cannot appear inside a field, so a pattern may contain spaces, '=' + * and ';' with nothing to escape. A trailing tab is an empty last column and + * is significant; no other whitespace is. + * + * Matchers: word whole words only, bounded by non-alphanumerics + * text any substring + * rule the chip's own exception syntax, L(F)R -- "the text + * fragment F, with left context L and right context R" + * + * --- the rule matcher ------------------------------------------------------ + * + * This is RC8650 datasheet "Exception Syntax", so RC Systems' own dictionaries + * convert to it a line at a time -- the pattern column takes L(F)R verbatim + * and the pronunciation goes in the output column as a phoneme span. Only the + * fragment is consumed; the contexts are looked at and not eaten: + * + * rule C(O)N [AA] o between c and n, as in icon + * rule $R(H) h after initial r is silent + * rule (5) [S I NG K O] five, in Spanish + * + * The fifteen Table 22 context tokens all work. Nine of them are sets of + * characters: + * + * # a vowel ? a voiced consonant $ a nonalphabetic character + * + a front vowel @ d j l n r s t z ch sh th + * ^ a consonant ! b c d f g p t \ a digit + * & a sibilant % a suffix (and then a non-letter) + * + * and six are structural: * one or more consonants, : zero or more, ~ one or + * more non-printing characters, | one or more digits with commas ignored, and + * ` a wildcard, which is the one token that also means something inside a + * fragment. Matching is first-match-wins in load order, and the fragment is + * what the scanner steps over -- so, exactly as the datasheet warns, (RAT) + * placed before (RATING) means (RATING) is never reached. + * + * Consecutive rules that both produce phonemes are joined into ONE span. They + * have to be: a dictionary that defines a whole letter-to-sound system matches + * every character, and a span per character would be a pair of mode switches + * per character. + * + * --- character classes, per dictionary ------------------------------------ + * + * #!class # a e i o u y á é í ó ú ü + * + * Table 22's classes are English, and that is the one part of the rule + * language that does not travel. Members are separated by spaces, so a class + * can hold the two-character members the datasheet gives @ and &, and a whole + * suffix list. A declaration replaces the class for every rule loaded after + * it, and rules keep the classes they were loaded under -- two dictionaries + * with different alphabets can be loaded into one rcdict and each keeps its + * own. + * + * '$' is not declarable and does not need to be: it is "not in # and not in + * ^", so declaring the alphabet gets it right for free. Getting '$' wrong is + * not a small mistake -- with the English vowel list an accented letter is not + * a letter, so '$' matches in the middle of a word and a letter-naming rule + * such as $(m)$ fires there. Spanish "más" comes out spelled aloud. + * '*' and ':' likewise follow '^'. + * + * Flags: i ignore case (the default) + * c match case exactly + * C ignore the pattern's case, but match only where the TEXT is + * all capitals -- "US" and not "us" or "Us", however the + * pattern itself was typed. ('c' with an all-capitals pattern + * reaches the same place; 'C' is for when the pattern's own + * case is not something you want to have to get right.) + * + * A LINE WITH NO TABS IN IT is read the other way round, because tabs are + * invisible and editors, config dialogs and web forms turn them into spaces + * without saying so. A dictionary that has been through one of those still + * loads: + * + * type[:flags] pattern = output e.g. word:C US = [Y UW EH S] + * + * There the pattern ends at an '=' with whitespace either side, and "\=" is a + * literal one. The two spellings mean exactly the same thing. + * + * The output is a template, and one field rather than two, so that text and + * phonemes can be mixed in a single entry: + * + * [ ... ] phonemes: Table 5 symbols, Table 6 modifiers, pauses + * { ... } a raw chip command, e.g. {9S} + * \0 the matched text; \1-\9 are groups, for regex later + * \[ \] \{ \} \\ \= literals + * + * Anything else is ordinary text, which goes back through the chip's + * letter-to-sound stage -- so a Sean/Shon entry is a respelling and needs no + * phonemes at all. An empty output makes the match silent. + */ + +#ifndef RCDICT_H +#define RCDICT_H + +#include + +#ifdef __cplusplus +extern "C" +{ +#endif + +/* --- target profiles ------------------------------------------------------ */ + +typedef struct rcdict_profile +{ + /* The command lead-in as the chip powers up. Both targets use CTRL+A; a + stream can change it, and rcdict_expand follows that when it happens. */ + unsigned char cmd_char; + + /* Mode commands, written with the profile's own command character. */ + const char *enter_phoneme; /* "\x01" "D" */ + const char *leave_phoneme; /* "\x01" "T" */ + + /* Table 5, uppercase, NULL-terminated. Letter mnemonics only: the pause + symbols are handled separately because they are punctuation. */ + const char *const *phonemes; + + /* Table 5's pause symbols, longest first. The RC8650 has ' (short), the + DoubleTalk PC's table does not list it. */ + const char *pauses; + + /* Table 6 attribute modifiers. These attach to phonemes rather than being + space-delimited -- the manual's own example is "-/D>/EH R", which is + -, /, D, >, /, EH, space, R. */ + const char *modifiers; + + /* Longest utterance the target's input buffer will take, for callers that + split. rcdict does not split; it only reports what it produced. */ + size_t max_utterance; +} rcdict_profile; + +extern const rcdict_profile rcdict_rc8650; +extern const rcdict_profile rcdict_doubletalk_pc; + +/* --- diagnostics ---------------------------------------------------------- */ + +/* Called for anything the caller would want to know about but which is not + fatal: an unparsable line, an unknown phoneme symbol, a span emitted where + the chip will not give it the right intonation. offset is a byte offset + into whatever was being read -- the input for rcdict_expand, the source text + for the loaders, which also name the file and line in the message. msg is + valid only for the duration of the call. + + Nothing reported here is ever dropped silently: a bad rule is skipped and + the rest of the file still loads, and a bad phoneme span is spoken as text. + A dictionary must not be able to make a screen reader go quiet. */ +typedef void (*rcdict_report) (void *ctx, size_t offset, const char *msg); + +/* --- dictionaries --------------------------------------------------------- */ + +typedef struct rcdict rcdict; + +rcdict *rcdict_new (const rcdict_profile *p); +void rcdict_free (rcdict *d); +void rcdict_clear (rcdict *d); +size_t rcdict_rule_count (const rcdict *d); + +/* Load rules. Each returns how many rules were added; a line that does not + parse is reported and skipped, so a typo costs one entry and not the file. + + Rules accumulate in load order across calls, and matching is first-match- + wins in that order -- so a later file cannot override an earlier one, it can + only add to it. Load the most specific dictionary FIRST. (This is the + datasheet's own rule for the chip's exception dictionaries, where (RATING) + must precede (RAT), and it applies here for the same reason.) */ +int rcdict_add_text (rcdict *d, const char *src, size_t len, + const char *name, rcdict_report report, void *ctx); +int rcdict_add_file (rcdict *d, const char *path, + rcdict_report report, void *ctx); + +/* Split a separator-delimited path list and load each file in turn. A file + that is not there is reported but is not an error: a search path naming + locations the user has not created yet is normal. + + Deliberately mechanism and not policy -- WHERE to look is the host's + business, and keeping it there is what stops this module needing anything + beyond stdio. Reloading is the host's business too, for the same reason: + watch the files however your platform prefers, then rcdict_clear and load + again. */ +int rcdict_add_path_list (rcdict *d, const char *list, char sep, + rcdict_report report, void *ctx); + +/* --- options -------------------------------------------------------------- */ + +typedef struct rcdict_options +{ + const rcdict_profile *profile; + + /* Rules to apply. Borrowed, not owned, and may be NULL for none. */ + const rcdict *dict; + + /* Recognise an inline phoneme escape in ordinary text, so that a + pronunciation can be written anywhere text can -- including NVDA's own + speech dictionaries, whose replacements are printable ASCII and would + otherwise have no way to reach phoneme mode. + + OFF BY DEFAULT, and it must stay that way: "[[" is wiki link syntax, and a + screen reader user reading a wiki page would otherwise find their text + silently eaten. Turning it on is a considered choice, not a default. */ + int inline_phonemes; + const char *open; /* default "[[" */ + const char *close; /* default "]]" */ +} rcdict_options; + +void rcdict_options_init (rcdict_options *o, const rcdict_profile *p); + +/* --- expansion ------------------------------------------------------------ */ + +/* Rewrite in[0..inlen) into out, returning the number of bytes the full result + needs, not counting the terminating NUL -- so a return value >= outcap means + the output was truncated, and the call can be repeated with a bigger buffer. + out may be NULL when outcap is 0, which is how you ask for the size. + + Where this must be called from, for a streaming caller: BEFORE any splitting + into utterance-sized pieces. A word can expand to forty characters of + phonemes plus the mode switches, so a piece that fitted the chip's input + buffer before expansion may not afterwards, and the overflow is dropped by + the input ring without a word. */ +size_t rcdict_expand (const rcdict_options *opt, + const char *in, size_t inlen, + char *out, size_t outcap, + rcdict_report report, void *ctx); + +/* --- helpers, for tooling ------------------------------------------------- */ + +/* Is sym[0..len) a Table 5 phoneme for this target? Case-insensitive. */ +int rcdict_is_phoneme (const rcdict_profile *p, const char *sym, size_t len); + +/* Check a phoneme string the way rcdict_expand would. Returns 1 if every + symbol in it is known, 0 otherwise, reporting each unknown one. */ +int rcdict_check_phonemes (const rcdict_profile *p, const char *s, size_t len, + rcdict_report report, void *ctx); + +const char *rcdict_version (void); + +#ifdef __cplusplus +} +#endif + +#endif /* RCDICT_H */ diff --git a/doubletalk/rcdict/rcdict_regex.c b/doubletalk/rcdict/rcdict_regex.c new file mode 100644 index 0000000..f7b257a --- /dev/null +++ b/doubletalk/rcdict/rcdict_regex.c @@ -0,0 +1,137 @@ +/* SPDX-License-Identifier: BSD-3-Clause */ +/* rcdict_regex --- see rcdict_regex.h. + * + * Wraps Remimu (https://github.com/wareya/Remimu), vendored here as remimu.h + * and released by its author into the public domain under CC0. It was chosen + * over the alternatives for reasons that are mostly not about regex: + * + * - CC0 vendors into a BSD-3 module, and into doubletalk-pc's BSD-3 tree, + * with no licence friction. That ruled out most candidates on its own. + * - No heap allocation and no recursion, so memory use is statically known + * and a pathological pattern cannot exhaust the stack. In a screen + * reader's synthesis thread that matters more than speed. + * - A separate parse step, so a rule compiles when the dictionary loads + * rather than once per utterance. + * + * tiny-regex-c has no capture groups at all; SubReg re-parses the pattern on + * every match, which is the wrong shape for a hot path with many rules. + */ + +#include +#include +#include + +/* Ceiling on backtracking steps for one match attempt. Remimu defaults this + to 0, meaning unlimited, and unlimited is not an option here: it is a + backtracking engine, so (a+)+b against a run of a's is exponential, and a + dictionary is a file the user edits. Measured: with the limit set that + pattern gives up immediately against thirty a's; without it, it hangs. + 200k steps is far more than any sane pattern needs on an utterance-sized + string, and costs well under a millisecond to burn through. */ +#define REMIMU_ITERATION_LIMIT 200000 + +/* The default is puts(), which would print to stdout from inside a synthesis + thread every time the limit above is reached. The caller is told through + the return value instead. */ +#define REMIMU_LOG_ERROR(x) ((void) (x)) + +#include "remimu.h" +#include "rcdict_regex.h" + +/* Enough for the patterns a pronunciation dictionary carries; a longer one is + refused at load with the rest of its line. */ +#define RCDICT_REGEX_MAX_TOKENS 256 + +struct rcdict_regex +{ + int16_t ntokens; + RegexToken tokens[1]; /* trailing, sized to ntokens */ +}; + +rcdict_regex * +rcdict_regex_compile (const char *pattern) +{ + RegexToken scratch[RCDICT_REGEX_MAX_TOKENS]; + int16_t n = RCDICT_REGEX_MAX_TOKENS; + rcdict_regex *re; + size_t bytes; + + if (!pattern) + return NULL; + if (regex_parse (pattern, scratch, &n, 0) != 0) + return NULL; + if (n < 1) + return NULL; + + /* A RegexToken is 40 bytes, so keeping the 256-entry scratch for every rule + would cost 10 KB each. Copy out just what was used. */ + bytes = sizeof *re + ((size_t) n - 1) * sizeof (RegexToken); + re = malloc (bytes); + if (!re) + return NULL; + re->ntokens = n; + memcpy (re->tokens, scratch, (size_t) n * sizeof (RegexToken)); + return re; +} + +void +rcdict_regex_free (rcdict_regex *re) +{ + free (re); +} + +long +rcdict_regex_match (const rcdict_regex *re, const char *text, size_t at, + size_t ncaps, size_t *cap_start, size_t *cap_len) +{ + int64_t pos[16], span[16]; + uint16_t slots; + int64_t r; + size_t i; + + if (!re || !text) + return RCDICT_REGEX_NOMATCH; + + slots = (uint16_t) (ncaps > 16 ? 16 : ncaps); + for (i = 0; i < slots; i++) + pos[i] = span[i] = -1; + + r = regex_match (re->tokens, text, at, slots, slots ? pos : NULL, + slots ? span : NULL); + + if (r == -2) + return RCDICT_REGEX_GAVEUP; /* iteration limit, or out of stack slots */ + if (r < 0) + return RCDICT_REGEX_NOMATCH; + + /* THE CORRECTION. regex_match returns the END INDEX, not the match length + its own header claims. At at==0 the two are equal, which is exactly why + the documentation has survived: every doc example matches at 0. Verified + against the engine -- /[0-9]+/ on "abc 123xy" returns 8 from both at=5 + and at=6, which is an end index and cannot be a length. */ + if ((size_t) r < at) + return RCDICT_REGEX_NOMATCH; /* cannot happen; do not underflow if it does */ + + for (i = 0; i < ncaps; i++) + { + if (cap_start) + cap_start[i] = RCDICT_REGEX_UNSET; + if (cap_len) + cap_len[i] = 0; + if (i < slots && pos[i] >= 0) + { + if (cap_start) + cap_start[i] = (size_t) pos[i]; + if (cap_len) + cap_len[i] = span[i] > 0 ? (size_t) span[i] : 0; + } + } + + return (long) ((size_t) r - at); +} + +const char * +rcdict_regex_engine (void) +{ + return "Remimu (https://github.com/wareya/Remimu), CC0 / public domain"; +} diff --git a/doubletalk/rcdict/rcdict_regex.h b/doubletalk/rcdict/rcdict_regex.h new file mode 100644 index 0000000..bc1d893 --- /dev/null +++ b/doubletalk/rcdict/rcdict_regex.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: BSD-3-Clause */ +/* rcdict_regex --- the regex matcher, kept behind a door. + * + * Everything about the vendored engine stops here: remimu.h is included by + * rcdict_regex.c and nowhere else, so its assumptions cannot leak into the + * rest of the module and it can be swapped without touching rcdict.c. + * + * The interface deliberately differs from the engine's in two places, because + * the engine's documentation is wrong about both and the corrections belong + * somewhere they cannot be forgotten: + * + * - It returns a match LENGTH here. regex_match returns the end index, + * despite its header saying "Returns match length"; the two agree only + * when matching at offset 0, which is why the mistake survives testing. + * - Matching is ANCHORED at `at` and never searches forward. That is what + * the engine does, it suits the caller (which tries every position itself, + * so that first-match-wins keeps meaning what the dictionary says), and it + * is worth saying out loud because "match" usually implies otherwise. + * + * `text` must be NUL-terminated, and the NUL must be the end of what may be + * matched: the engine has no length argument, so a caller with a bounded + * region has to hand over a copy. For rcdict that is the point rather than a + * nuisance -- the region is one run of speakable text, and `^`, `$` and `\b` + * should see its edges and not the command bytes on either side. + */ + +#ifndef RCDICT_REGEX_H +#define RCDICT_REGEX_H + +#include + +#ifdef __cplusplus +extern "C" +{ +#endif + +typedef struct rcdict_regex rcdict_regex; + +/* Compile a pattern. NULL if it will not parse. Compilation happens once, + when the dictionary loads, and never while speaking. */ +rcdict_regex *rcdict_regex_compile (const char *pattern); +void rcdict_regex_free (rcdict_regex *re); + +#define RCDICT_REGEX_NOMATCH (-1) +#define RCDICT_REGEX_GAVEUP (-2) + +/* Match at text[at], anchored. Returns the match length, or one of the two + negatives above. + + GAVEUP means the engine hit its iteration limit -- this is a backtracking + engine, and a pattern like (a+)+b against a run of a's is exponential. The + limit is what keeps a bad dictionary entry from wedging a screen reader's + synthesis thread, so callers must treat it as "no match, and say so" rather + than as an impossible case. + + cap_start/cap_len receive up to ncaps captures, as offsets into text. Slot + 0 is the whole match; slots 1..n are the parenthesised groups. Unset + captures come back with cap_start == RCDICT_REGEX_UNSET. */ +#define RCDICT_REGEX_UNSET ((size_t) -1) + +long rcdict_regex_match (const rcdict_regex *re, const char *text, size_t at, + size_t ncaps, size_t *cap_start, size_t *cap_len); + +/* Name of the engine and its licence, for an about box or a NOTICE file. */ +const char *rcdict_regex_engine (void); + +#ifdef __cplusplus +} +#endif + +#endif /* RCDICT_REGEX_H */ diff --git a/doubletalk/rcdict/remimu.h b/doubletalk/rcdict/remimu.h new file mode 100644 index 0000000..21080ee --- /dev/null +++ b/doubletalk/rcdict/remimu.h @@ -0,0 +1,1509 @@ +#ifndef INCLUDE_REMIMU +#define INCLUDE_REMIMU 1 + +#ifndef REMIMU_FUNC_VISIBILITY +#define REMIMU_FUNC_VISIBILITY static inline +#endif + +#ifndef REMIMU_CONST_VISIBILITY +#define REMIMU_CONST_VISIBILITY static const +#endif + +#ifndef REMIMU_LOG_ERROR +#define REMIMU_LOG_ERROR puts +#endif + +#ifndef REMIMU_ITERATION_LIMIT +#define REMIMU_ITERATION_LIMIT 0 // Set to non-zero to enable an interation limit +#endif + +#ifndef REMIMU_ASSERT +#define REMIMU_ASSERT(x) assert(x) +#endif + +/************ + + REMIMU: SINGLE HEADER C/C++ REGEX LIBRARY + + Compatible with C99 and C++11 and later standards. Uses backtracking and relatively standard regex syntax. + + #include "remimu.h" + +FUNCTIONS + + // Returns 0 on success, or -1 on invalid or unsupported regex, or -2 on not enough tokens given to parse regex. + int regex_parse( + const char * pattern, // Regex pattern to parse. + RegexToken * tokens, // Output buffer of token_count regex tokens. + int16_t * token_count, // Maximum allowed number of tokens to write + int32_t flags // Optional bitflags. + ) + + // Returns match length, or -1 on no match, or -2 on out of memory, or -3 if the regex is invalid. + int64_t regex_match( + const RegexToken * tokens, // Parsed regex to match against text. + const char * text, // Text to match against tokens. + size_t start_i, // index value to match at. + uint16_t cap_slots, // Number of allowed capture info output slots. + int64_t * cap_pos, // Capture position info output buffer. + int64_t * cap_span // Capture length info output buffer. + ) + + void print_regex_tokens( + RegexToken * tokens // Regex tokens to spew to stdout, for debugging. + ) + +PERFORMANCE + + On simple cases, Remimu's match speed is similar to PCRE2. Regex parsing/compilation is also much faster (around 4x to 10x), so single-shot regexes are often faster than PCRE2. + + HOWEVER: Remimu is a pure backtracking engine, and has `O(2^x)` complexity on regexes with catastrophic backtracking. It can be much, much, MUCH slower than PCRE2. Beware! + + Remimu uses length-checked fixed memory buffers with no recursion, so memory usage is statically known. + +FEATURES + + - Lowest-common-denominator common regex syntax + - Based on backtracking (slow in the worst case, but fast in the best case) + - 8-bit only, no utf-16 or utf-32 + - Statically known memory usage (no heap allocation or recursion) + - Groups with or without capture, and with or without quantifiers + - Supported escapes: + - - 2-digit hex: e.g. \x00, \xFF, or lowercase, or mixed case + - - \r, \n, \t, \v, \f (whitespace characters) + - - \d, \s, \w, \D, \S, \W (digit, space, and word character classes) + - - \b, \B word boundary and non-word-boundary anchors (not fully supported in zero-size quantified groups, but even then, usually supported) + - - Escaped literal characters: {}[]-()|^$*+?:./\ + - - - Escapes work in character classes, except for 'b' + - Character classes, including disjoint ranges, proper handling of bare [ and trailing -, etc + - - Dot (.) matches all characters, including newlines, unless REMIMU_FLAG_DOT_NO_NEWLINES is passed as a flag to regex_parse + - - Dot (.) only matches at most one byte at a time, so matching \r\n requires two dots (and not using REMIMU_FLAG_DOT_NO_NEWLINES) + - Anchors (^ and $) + - - Same support caveats as \b, \B apply + - Basic quantifiers (*, +, ?) + - - Quantifiers are greedy by default. + - Explicit quantifiers ({2}, {5}, {5,}, {5,7}) + - Alternation e.g. (asdf|foo) + - Lazy quantifiers e.g. (asdf)*? or \w+? + - Possessive greedy quantifiers e.g. (asdf)*+ or \w++ + - - NOTE: Capture groups for and inside of possessive groups return no capture information. + - Atomic groups e.g. (?>(asdf)) + - - NOTE: Capture groups inside of atomic groups return no capture information. + +NOT SUPPORTED + + - Strings with non-terminal null characters + - Unicode character classes (matching single utf-8 characters works regardless) + - Exact POSIX regex semantics (posix-style greediness etc) + - Backreferences + - Lookbehind/Lookahead + - Named groups + - Most other weird flavor-specific regex stuff + - Capture of or inside of possessive-quantified groups (still take up a capture slot, but no data is returned) + +USAGE + + // minimal: + + RegexToken tokens[1024]; + int16_t token_count = 1024; + int e = regex_parse("[0-9]+\\.[0-9]+", tokens, &token_count, 0); + assert(!e); + + int64_t match_len = regex_match(tokens, "23.53) ", 0, 0, 0, 0); + printf("########### return: %zd\n", match_len); + + // with captures: + + RegexToken tokens[256]; + int16_t token_count = sizeof(tokens)/sizeof(tokens[0]); + int e = regex_parse("((a)|(b))++", tokens, &token_count, 0); + assert(!e); + + int64_t cap_pos[5]; + int64_t cap_span[5]; + memset(cap_pos, 0xFF, sizeof(cap_pos)); + memset(cap_span, 0xFF, sizeof(cap_span)); + + int64_t matchlen = regex_match(tokens, "aaaaaabbbabaqa", 0, 5, cap_pos, cap_span); + printf("Match length: %zd\n", matchlen); + for (int i = 0; i < 5; i++) + printf("Capture %d: %zd plus %zd\n", i, cap_pos[i], cap_span[i]); + + // for debugging + print_regex_tokens(tokens); + +LICENSE + + Creative Commons Zero, public domain. + +*/ + +#include +#include +#include +#include +#include + +REMIMU_CONST_VISIBILITY int REMIMU_FLAG_DOT_NO_NEWLINES = 1; + +REMIMU_CONST_VISIBILITY uint8_t REMIMU_KIND_NORMAL = 0; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_KIND_OPEN = 1; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_KIND_NCOPEN = 2; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_KIND_CLOSE = 3; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_KIND_OR = 4; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_KIND_CARET = 5; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_KIND_DOLLAR = 6; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_KIND_BOUND = 7; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_KIND_NBOUND = 8; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_KIND_END = 9; + +REMIMU_CONST_VISIBILITY uint8_t REMIMU_MODE_POSSESSIVE = 1; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_MODE_LAZY = 2; +REMIMU_CONST_VISIBILITY uint8_t REMIMU_MODE_INVERTED = 128; // temporary; gets cleared later + +typedef struct _RegexToken { + uint8_t kind; + uint8_t mode; + uint16_t count_lo; + uint16_t count_hi; // 0 means no limit + uint16_t mask[16]; // for groups: mask 0 stores group-with-quantifier number (quantifiers are +, *, ?, {n}, {n,}, or {n,m}) + int16_t pair_offset; // from ( or ), offset in token list to matching paren. TODO: move into mask maybe +} RegexToken; + +static int remimu_nibble_hex_to_bin(char hex, uint8_t *bin) +{ + if (hex >= '0' && hex <= '9') + { + *bin = hex - '0'; + return 0; + } + if (hex >= 'A' && hex <= 'F') + { + *bin = hex - 'A' + 10; + return 0; + } + if (hex >= 'a' && hex <= 'f') + { + *bin = hex - 'a' + 10; + return 0; + } + return -1; // invalid hex digit +} + +/// Returns a negative number on failure: +/// -1: Regex string is invalid or using unsupported features or too long. +/// -2: Provided buffer not long enough. Give up, or reallocate with more length and retry. +/// Returns 0 on success. +/// On call, token_count pointer must point to the number of tokens that can be written to the tokens buffer. +/// On successful return, the number of actually used tokens is written to token_count. +/// Sets token_count to zero if a regex is not created but no error happened (e.g. empty pattern). +/// Flags: Not yet used. +/// SAFETY: Pattern must be null-terminated. +/// SAFETY: tokens buffer must have at least the input token_count number of RegexToken objects. They are allowed to be uninitialized. +REMIMU_FUNC_VISIBILITY int regex_parse(const char * pattern, RegexToken * tokens, int16_t * token_count, int32_t flags) +{ + int64_t tokens_len = *token_count; + uint64_t pattern_len = strlen(pattern); + if (token_count == 0) + return -2; + + // 0: normal + // 1: just saw a backslash + int esc_state = 0; + + // 0: init + // 1: normal + // 2: in char class, initial state + // 3: in char class, but possibly looking for a range marker + // 4: in char class, but just saw a range marker + // 5: immediately after quantifiable token + // 6: immediately after quantifier + + const int STATE_NORMAL = 1; + const int STATE_QUANT = 2; + const int STATE_MODE = 3; + const int STATE_CC_INIT = 4; + const int STATE_CC_NORMAL = 5; + const int STATE_CC_RANGE = 6; + int state = STATE_NORMAL; + + int char_class_mem = -1; + + RegexToken token; + + #define _REGEX_CLEAR_TOKEN() do { \ + memset(&token, 0, sizeof(RegexToken)); \ + token.count_lo = 1; \ + token.count_hi = 2; \ + } while(0) + + _REGEX_CLEAR_TOKEN(); + + #define _REGEX_DO_INVERT() do { \ + for (int n = 0; n < 16; n++) \ + token.mask[n] = ~token.mask[n]; \ + token.mode &= ~REMIMU_MODE_INVERTED; \ + } while (0) + + int16_t k = 0; + + #define _REGEX_PUSH_TOKEN() do { \ + if (k == 0 || tokens[k-1].kind != token.kind || (token.kind != REMIMU_KIND_BOUND && token.kind != REMIMU_KIND_NBOUND)) \ + { \ + if (token.mode & REMIMU_MODE_INVERTED) _REGEX_DO_INVERT(); \ + if (k >= tokens_len) \ + { \ + REMIMU_LOG_ERROR("buffer overflow"); \ + return -2; \ + } \ + tokens[k++] = token; \ + _REGEX_CLEAR_TOKEN(); \ + } \ + } while (0) + + #define _REGEX_SET_MASK(byte) do { token.mask[((uint8_t)(byte))>>4] |= 1 << ((uint8_t)(byte) & 0xF); } while (0) + #define _REGEX_SET_MASK_ALL() do { \ + for (int n = 0; n < 16; n++) \ + token.mask[n] = 0xFFFF; \ + } while (0) + + // start with an invisible group specifier + // (this allows the matcher to not need to have a special root-level alternation operator case) + token.kind = REMIMU_KIND_OPEN; + token.count_lo = 0; + token.count_hi = 0; + + int paren_count = 0; + + for (uint64_t i = 0; i < pattern_len; i++) + { + char c = pattern[i]; + if (state == STATE_QUANT) + { + state = STATE_MODE; + if (c == '?') + { + token.count_lo = 0; + token.count_hi = 2; // first non-allowed amount + continue; + } + else if (c == '+') + { + token.count_lo = 1; + token.count_hi = 0; // unlimited + continue; + } + else if (c == '*') + { + token.count_lo = 0; + token.count_hi = 0; // unlimited + continue; + } + else if (c == '{') + { + if (pattern[i+1] == 0 || pattern[i+1] < '0' || pattern[i+1] > '9') + state = STATE_NORMAL; + else + { + i += 1; + uint32_t val = 0; + while (pattern[i] >= '0' && pattern[i] <= '9') + { + val *= 10; + val += (uint32_t)(pattern[i] - '0'); + if (val > 0xFFFF) + { + REMIMU_LOG_ERROR("quantifier range too long"); + return -1; // unsupported length + } + i += 1; + } + token.count_lo = val; + token.count_hi = val + 1; + if (pattern[i] == ',') + { + token.count_hi = 0; // unlimited + i += 1; + + if (pattern[i] >= '0' && pattern[i] <= '9') + { + uint32_t val2 = 0; + while (pattern[i] >= '0' && pattern[i] <= '9') + { + val2 *= 10; + val2 += (uint32_t)(pattern[i] - '0'); + if (val2 > 0xFFFF) + { + REMIMU_LOG_ERROR("quantifier range too long"); + return -1; // unsupported length + } + i += 1; + } + if (val2 < val) + { + REMIMU_LOG_ERROR("quantifier range is backwards"); + return -1; // unsupported length + } + token.count_hi = val2 + 1; + } + } + + if (pattern[i] == '}') + { + // quantifier range parsed successfully + continue; + } + else + { + REMIMU_LOG_ERROR("quantifier range syntax broken (no terminator)"); + return -1; + } + } + } + } + + if (state == STATE_MODE) + { + state = STATE_NORMAL; + if (c == '?') + { + token.mode |= REMIMU_MODE_LAZY; + continue; + } + else if (c == '+') + { + token.mode |= REMIMU_MODE_POSSESSIVE; + continue; + } + } + + if (state == STATE_NORMAL) + { + if (esc_state == 1) + { + esc_state = 0; + if (c == 'n') + _REGEX_SET_MASK('\n'); + else if (c == 'r') + _REGEX_SET_MASK('\r'); + else if (c == 't') + _REGEX_SET_MASK('\t'); + else if (c == 'v') + _REGEX_SET_MASK('\v'); + else if (c == 'f') + _REGEX_SET_MASK('\f'); + else if (c == 'x') + { + if (pattern[i+1] == 0 || pattern[i+2] == 0) + return -1; // too-short hex pattern + uint8_t n0, n1; + if (remimu_nibble_hex_to_bin(pattern[i+1], &n0)) + return -1; // invalid hex + if (remimu_nibble_hex_to_bin(pattern[i+2], &n1)) + return -1; // invalid hex + _REGEX_SET_MASK((n0 << 4) | n1); + i += 2; + state = STATE_QUANT; + } + else if (c == '{' || c == '}' || + c == '[' || c == ']' || c == '-' || + c == '(' || c == ')' || + c == '|' || c == '^' || c == '$' || + c == '*' || c == '+' || c == '?' || c == ':' || + c == '.' || c == '/' || c == '\\') + { + _REGEX_SET_MASK(c); + state = STATE_QUANT; + } + else if (c == 'd' || c == 's' || c == 'w' || + c == 'D' || c == 'S' || c == 'W') + { + uint8_t is_upper = c <= 'Z'; + + uint16_t m[16]; + memset(m, 0, sizeof(m)); + + if (is_upper) + c += 0x20; + if (c == 'd' || c == 'w') + m[3] |= 0x03FF; // 0~7 + if (c == 's') + { + m[0] |= 0x3E00; // \t-\r (includes \n, \v, and \f in the middle. 5 enabled bits.) + m[2] |= 1; // ' ' + } + if (c == 'w') + { + m[4] |= 0xFFFE; // A-O + m[5] |= 0x87FF; // P-Z_ + m[6] |= 0xFFFE; // a-o + m[7] |= 0x07FF; // p-z + } + + for (int j = 0; j < 16; j++) + token.mask[j] |= is_upper ? ~m[j] : m[j]; + + token.kind = REMIMU_KIND_NORMAL; + state = STATE_QUANT; + } + else if (c == 'b') + { + token.kind = REMIMU_KIND_BOUND; + state = STATE_NORMAL; + } + else if (c == 'B') + { + token.kind = REMIMU_KIND_NBOUND; + state = STATE_NORMAL; + } + else + { + REMIMU_LOG_ERROR("unsupported escape sequence"); + return -1; // unknown/unsupported escape sequence + } + } + else + { + _REGEX_PUSH_TOKEN(); + if (c == '\\') + { + esc_state = 1; + } + else if (c == '[') + { + state = STATE_CC_INIT; + char_class_mem = -1; + token.kind = REMIMU_KIND_NORMAL; + if (pattern[i + 1] == '^') + { + token.mode |= REMIMU_MODE_INVERTED; + i += 1; + } + } + else if (c == '(') + { + paren_count += 1; + state = STATE_NORMAL; + token.kind = REMIMU_KIND_OPEN; + token.count_lo = 0; + token.count_hi = 1; + if (pattern[i + 1] == '?' && pattern[i + 2] == ':') + { + token.kind = REMIMU_KIND_NCOPEN; + i += 2; + } + else if (pattern[i + 1] == '?' && pattern[i + 2] == '>') + { + token.kind = REMIMU_KIND_NCOPEN; + _REGEX_PUSH_TOKEN(); + + state = STATE_NORMAL; + token.kind = REMIMU_KIND_NCOPEN; + token.mode = REMIMU_MODE_POSSESSIVE; + token.count_lo = 1; + token.count_hi = 2; + + i += 2; + } + } + else if (c == ')') + { + paren_count -= 1; + if (paren_count < 0 || k == 0) + return -1; // unbalanced parens + token.kind = REMIMU_KIND_CLOSE; + state = STATE_QUANT; + + int balance = 0; + ptrdiff_t found = -1; + for (ptrdiff_t l = k - 1; l >= 0; l--) + { + if (tokens[l].kind == REMIMU_KIND_NCOPEN || tokens[l].kind == REMIMU_KIND_OPEN) + { + if (balance == 0) + { + found = l; + break; + } + else + balance -= 1; + } + else if (tokens[l].kind == REMIMU_KIND_CLOSE) + balance += 1; + } + if (found == -1) + return -1; // unbalanced parens + ptrdiff_t diff = k - found; + if (diff > 32767) + return -1; // too long + token.pair_offset = -diff; + tokens[found].pair_offset = diff; + // phantom group for atomic group emulation + if (tokens[found].mode == REMIMU_MODE_POSSESSIVE) + { + _REGEX_PUSH_TOKEN(); + token.kind = REMIMU_KIND_CLOSE; + token.mode = REMIMU_MODE_POSSESSIVE; + token.pair_offset = -diff - 2; + tokens[found - 1].pair_offset = diff + 2; + } + } + else if (c == '?' || c == '+' || c == '*' || c == '{') + { + REMIMU_LOG_ERROR("quantifier in non-quantifier context"); + return -1; // quantifier in non-quantifier context + } + else if (c == '.') + { + //puts("setting ALL of mask..."); + _REGEX_SET_MASK_ALL(); + if (flags & REMIMU_FLAG_DOT_NO_NEWLINES) + { + token.mask[1] ^= 0x04; // \n + token.mask[1] ^= 0x20; // \r + } + state = STATE_QUANT; + } + else if (c == '^') + { + token.kind = REMIMU_KIND_CARET; + state = STATE_NORMAL; + } + else if (c == '$') + { + token.kind = REMIMU_KIND_DOLLAR; + state = STATE_NORMAL; + } + else if (c == '|') + { + token.kind = REMIMU_KIND_OR; + state = STATE_NORMAL; + } + else + { + _REGEX_SET_MASK(c); + state = STATE_QUANT; + } + } + } + else if (state == STATE_CC_INIT || state == STATE_CC_NORMAL || state == STATE_CC_RANGE) + { + if (c == '\\' && esc_state == 0) + { + esc_state = 1; + continue; + } + uint8_t esc_c = 0; + if (esc_state == 1) + { + esc_state = 0; + if (c == 'n') + esc_c = '\n'; + else if (c == 'r') + esc_c = '\r'; + else if (c == 't') + esc_c = '\t'; + else if (c == 'v') + esc_c = '\v'; + else if (c == 'f') + esc_c = '\f'; + else if (c == 'x') + { + if (pattern[i+1] == 0 || pattern[i+2] == 0) + return -1; // too-short hex pattern + uint8_t n0, n1; + if (remimu_nibble_hex_to_bin(pattern[i+1], &n0)) + return -1; // invalid hex + if (remimu_nibble_hex_to_bin(pattern[i+2], &n1)) + return -1; // invalid hex + esc_c = (n0 << 4) | n1; + i += 2; + } + else if (c == '{' || c == '}' || + c == '[' || c == ']' || c == '-' || + c == '(' || c == ')' || + c == '|' || c == '^' || c == '$' || + c == '*' || c == '+' || c == '?' || c == ':' || + c == '.' || c == '/' || c == '\\') + { + esc_c = c; + } + else if (c == 'd' || c == 's' || c == 'w' || + c == 'D' || c == 'S' || c == 'W') + { + if (state == STATE_CC_RANGE) + { + REMIMU_LOG_ERROR("tried to use a shorthand as part of a range"); + return -1; // range shorthands can't be part of a range + } + uint8_t is_upper = c <= 'Z'; + + uint16_t m[16]; + memset(m, 0, sizeof(m)); + + if (is_upper) + c += 0x20; + if (c == 'd' || c == 'w') + m[3] |= 0x03FF; // 0~7 + if (c == 's') + { + m[0] |= 0x3E00; // \t-\r (includes \n, \v, and \f in the middle. 5 enabled bits.) + m[2] |= 1; // ' ' + } + if (c == 'w') + { + m[4] |= 0xFFFE; // A-O + m[5] |= 0x87FF; // P-Z_ + m[6] |= 0xFFFE; // a-o + m[7] |= 0x07FF; // p-z + } + + for (int j = 0; j < 16; j++) + token.mask[j] |= is_upper ? ~m[j] : m[j]; + + char_class_mem = -1; // range shorthands can't be part of a range + continue; + } + else + { + printf("unknown/unsupported escape sequence in character class (\\%c)\n", c); + return -1; // unknown/unsupported escape sequence + } + } + if (state == STATE_CC_INIT) + { + uint8_t val = esc_c ? esc_c : (uint8_t)c; + char_class_mem = val; + _REGEX_SET_MASK(val); + state = STATE_CC_NORMAL; + } + else if (state == STATE_CC_NORMAL) + { + if (c == ']' && esc_c == 0) + { + char_class_mem = -1; + state = STATE_QUANT; + continue; + } + else if (c == '-' && esc_c == 0 && char_class_mem >= 0) + { + state = STATE_CC_RANGE; + continue; + } + else + { + uint8_t val = esc_c ? esc_c : (uint8_t)c; + char_class_mem = val; + _REGEX_SET_MASK(val); + state = STATE_CC_NORMAL; + } + } + else if (state == STATE_CC_RANGE) + { + if (c == ']' && esc_c == 0) + { + char_class_mem = -1; + _REGEX_SET_MASK('-'); + state = STATE_QUANT; + continue; + } + else + { + if (char_class_mem == -1) + { + REMIMU_LOG_ERROR("character class range is broken"); + return -1; // probably tried to use a character class shorthand as part of a range + } + uint8_t rhs = esc_c ? esc_c : (uint8_t)c; + if (rhs < (uint8_t)char_class_mem) + { + REMIMU_LOG_ERROR("character class range is misordered"); + return -1; // range is in wrong order + } + //printf("enabling char class from %d to %d...\n", char_class_mem, c); + for (uint8_t j = rhs; j > (uint8_t)char_class_mem; j--) + _REGEX_SET_MASK(j); + state = STATE_CC_NORMAL; + char_class_mem = -1; + } + } + } + else + REMIMU_ASSERT(0); + } + if (paren_count > 0) + { + REMIMU_LOG_ERROR("(paren_count > 0)"); + return -1; // unbalanced parens + } + if (esc_state != 0) + { + REMIMU_LOG_ERROR("(esc_state != 0)"); + return -1; // open escape sequence + } + if (state >= STATE_CC_INIT) + { + REMIMU_LOG_ERROR("(state >= STATE_CC_INIT)"); + return -1; // open character class + } + + _REGEX_PUSH_TOKEN(); + + // add invisible non-capturing group specifier + token.kind = REMIMU_KIND_CLOSE; + token.count_lo = 1; + token.count_hi = 2; + _REGEX_PUSH_TOKEN(); + + // add end token (tells matcher that it's done) + token.kind = REMIMU_KIND_END; + _REGEX_PUSH_TOKEN(); + + tokens[0].pair_offset = k - 2; + tokens[k-2].pair_offset = -(k - 2); + + *token_count = k; + + // copy quantifiers from )s to (s (so (s know whether they're optional) + // also take the opportunity to smuggle "quantified group index" into the mask field for the ) + uint64_t n = 0; + for (int16_t k2 = 0; k2 < k; k2++) + { + if (tokens[k2].kind == REMIMU_KIND_CLOSE) + { + tokens[k2].mask[0] = n++; + + int16_t k3 = k2 + tokens[k2].pair_offset; + tokens[k3].count_lo = tokens[k2].count_lo; + tokens[k3].count_hi = tokens[k2].count_hi; + tokens[k3].mask[0] = n++; + tokens[k3].mode = tokens[k2].mode; + + //if (n > 65535) + if (n > 1024) + return -1; // too many quantified groups + } + else if (tokens[k2].kind == REMIMU_KIND_OR || tokens[k2].kind == REMIMU_KIND_OPEN || tokens[k2].kind == REMIMU_KIND_NCOPEN) + { + // find next | or ) and how far away it is. store in token + int balance = 0; + ptrdiff_t found = -1; + for (ptrdiff_t l = k2 + 1; l < k; l++) + { + if (tokens[l].kind == REMIMU_KIND_OR && balance == 0) + { + found = l; + break; + } + else if (tokens[l].kind == REMIMU_KIND_CLOSE) + { + if (balance == 0) + { + found = l; + break; + } + else + balance -= 1; + } + else if (tokens[l].kind == REMIMU_KIND_NCOPEN || tokens[l].kind == REMIMU_KIND_OPEN) + balance += 1; + } + if (found == -1) + { + REMIMU_LOG_ERROR("unbalanced parens..."); + return -1; // unbalanced parens + } + ptrdiff_t diff = found - k2; + if (diff > 32767) + { + REMIMU_LOG_ERROR("too long..."); + return -1; // too long + } + + if (tokens[k2].kind == REMIMU_KIND_OR) + tokens[k2].pair_offset = diff; + else + tokens[k2].mask[15] = diff; + } + } + + #undef _REGEX_PUSH_TOKEN + #undef _REGEX_SET_MASK + #undef _REGEX_CLEAR_TOKEN + + return 0; +} + +typedef struct _RegexMatcherState { + uint32_t k; + uint32_t group_state; // quantified group temp state (e.g. number of repetitions) + uint32_t prev; // for )s, stack index of corresponding previous quantified state +#ifdef REGEX_STACK_SMOL + uint32_t i; + uint32_t range_min; + uint32_t range_max; +#else + uint64_t i; + uint64_t range_min; + uint64_t range_max; +#endif +} RegexMatcherState; + +// NOTE: undef'd later +#define _REGEX_CHECK_MASK(K, byte) (!!(tokens[K].mask[((uint8_t)byte)>>4] & (1 << ((uint8_t)byte & 0xF)))) + +// Returns match length if text starts with a regex match. +// Returns -1 if the text doesn't start with a regex match. +// Returns -2 if the matcher ran out of memory or the regex is too complex. +// Returns -3 if the regex is somehow invalid. +// The first cap_slots capture positions and spans (lengths) will be written to cap_pos and cap_span. If zero, will not be written to. +// SAFETY: The text variable must be null-terminated, and start_i must be the index of a character within the string or its null terminator. +// SAFETY: Tokens array must be terminated by a REMIMU_KIND_END token (done by default by regex_parse). +// SAFETY: Partial capture data may be written even if the match fails. +REMIMU_FUNC_VISIBILITY int64_t regex_match(const RegexToken * tokens, const char * text, size_t start_i, uint16_t cap_slots, int64_t * cap_pos, int64_t * cap_span) +{ + (void)text; + +#ifdef REGEX_VERBOSE + const uint8_t verbose = 1; +#else + const uint8_t verbose = 0; +#endif + +#define IF_VERBOSE(X) { if (verbose) { X } } + +#ifdef REGEX_STACK_SMOL + const uint16_t stack_size_max = 256; +#else + const uint16_t stack_size_max = 1024; +#endif + const uint16_t aux_stats_size = 1024; + if (cap_slots > aux_stats_size) + cap_slots = aux_stats_size; + + // quantified group state + uint8_t q_group_accepts_zero[aux_stats_size]; + uint32_t q_group_state[aux_stats_size]; // number of repetitions + uint32_t q_group_stack[aux_stats_size]; // location of most recent corresponding ) on stack. 0 means nowhere + + uint16_t q_group_cap_index[aux_stats_size]; + memset(q_group_cap_index, 0xFF, sizeof(q_group_cap_index)); + + uint64_t tokens_len = 0; + uint32_t k = 0; + uint16_t caps = 0; + + while (tokens[k].kind != REMIMU_KIND_END) + { + if (tokens[k].kind == REMIMU_KIND_OPEN && caps < cap_slots) + { + q_group_cap_index[tokens[k].mask[0]] = caps; + q_group_cap_index[tokens[k + tokens[k].pair_offset].mask[0]] = caps; + cap_pos[caps] = -1; + cap_span[caps] = -1; + caps += 1; + } + k += 1; + if (tokens[k].kind == REMIMU_KIND_CLOSE || tokens[k].kind == REMIMU_KIND_OPEN || tokens[k].kind == REMIMU_KIND_NCOPEN) + { + if (tokens[k].mask[0] >= aux_stats_size) + { + REMIMU_LOG_ERROR("too many qualified groups. returning"); + return -2; // OOM: too many quantified groups + } + + q_group_state[tokens[k].mask[0]] = 0; + q_group_stack[tokens[k].mask[0]] = 0; + q_group_accepts_zero[tokens[k].mask[0]] = 0; + } + } + + tokens_len = k; + + RegexMatcherState rewind_stack[stack_size_max]; + uint16_t stack_n = 0; + + uint64_t i = start_i; + + uint64_t range_min = 0; + uint64_t range_max = 0; + uint8_t just_rewinded = 0; + + #define _P_TEXT_HIGHLIGHTED() do { \ + IF_VERBOSE(printf("\033[91m"); \ + for (uint64_t q = 0; q < i; q++) printf("%c", text[q]); \ + printf("\033[0m"); \ + for (uint64_t q = i; text[q] != 0; q++) printf("%c", text[q]); \ + printf("\n");) \ + } while (0) + + #define _REWIND_DO_SAVE_RAW(K, ISDUMMY) do { \ + if (stack_n >= stack_size_max) \ + { \ + REMIMU_LOG_ERROR("out of backtracking room. returning"); \ + return -2; \ + } \ + RegexMatcherState s; \ + memset(&s, 0, sizeof(RegexMatcherState)); \ + s.i = i; \ + s.k = (K); \ + s.range_min = range_min; \ + s.range_max = range_max; \ + s.prev = 0; \ + if (ISDUMMY) s.prev = 0xFAC7; \ + else if (tokens[s.k].kind == REMIMU_KIND_CLOSE) \ + { \ + s.group_state = q_group_state[tokens[s.k].mask[0]]; \ + s.prev = q_group_stack[tokens[s.k].mask[0]]; \ + q_group_stack[tokens[s.k].mask[0]] = stack_n; \ + } \ + rewind_stack[stack_n++] = s; \ + _P_TEXT_HIGHLIGHTED(); \ + IF_VERBOSE(printf("-- saving rewind state k %u i %zd rmin %zu rmax %zd (line %d) (depth %d prev %d)\n", s.k, i, range_min, range_max, __LINE__, stack_n, s.prev);) \ + } while (0) + #define _REWIND_DO_SAVE_DUMMY(K) _REWIND_DO_SAVE_RAW(K, 1) + #define _REWIND_DO_SAVE(K) _REWIND_DO_SAVE_RAW(K, 0) + + #define _REWIND_OR_ABORT() do { \ + if (stack_n == 0) \ + return -1; \ + stack_n -= 1; \ + while (stack_n > 0 && rewind_stack[stack_n].prev == 0xFAC7) stack_n -= 1; \ + just_rewinded = 1; \ + range_min = rewind_stack[stack_n].range_min; \ + range_max = rewind_stack[stack_n].range_max; \ + REMIMU_ASSERT(rewind_stack[stack_n].i <= i); \ + i = rewind_stack[stack_n].i; \ + k = rewind_stack[stack_n].k; \ + if (tokens[k].kind == REMIMU_KIND_CLOSE) \ + { \ + q_group_state[tokens[k].mask[0]] = rewind_stack[stack_n].group_state; \ + q_group_stack[tokens[k].mask[0]] = rewind_stack[stack_n].prev; \ + } \ + _P_TEXT_HIGHLIGHTED(); \ + IF_VERBOSE(printf("-- rewound to k %u i %zd rmin %zu rmax %zd (kind %d prev %d)\n", k, i, range_min, range_max, tokens[k].kind, rewind_stack[stack_n].prev);) \ + k -= 1; \ + } while (0) + // the -= 1 is because of the k++ in the for loop + + // used in boundary anchor checker + uint64_t w_mask[16]; + memset(w_mask, 0, sizeof(w_mask)); + w_mask[3] = 0x03FF; + w_mask[4] = 0xFFFE; + w_mask[5] = 0x87FF; + w_mask[6] = 0xFFFE; + w_mask[7] = 0x07FF; + #define _REGEX_CHECK_IS_W(byte) (!!(w_mask[((uint8_t)byte)>>4] & (1 << ((uint8_t)byte & 0xF)))) + + int limit = REMIMU_ITERATION_LIMIT; + for (k = 0; k < tokens_len; k++) + { + if (REMIMU_ITERATION_LIMIT) + { + if (limit-- == 0) + { + REMIMU_LOG_ERROR("iteration limit exceeded. returning"); + return -2; + } + } + IF_VERBOSE(printf("k: %u\ti: %zu\tl: %zu\tstack_n: %d\n", k, i, limit, stack_n);) + _P_TEXT_HIGHLIGHTED(); + if (tokens[k].kind == REMIMU_KIND_CARET) + { + if (i != 0) + _REWIND_OR_ABORT(); + continue; + } + else if (tokens[k].kind == REMIMU_KIND_DOLLAR) + { + if (text[i] != 0) + _REWIND_OR_ABORT(); + continue; + } + else if (tokens[k].kind == REMIMU_KIND_BOUND) + { + if (i == 0 && !_REGEX_CHECK_IS_W(text[i])) + _REWIND_OR_ABORT(); + else if (i != 0 && text[i] == 0 && !_REGEX_CHECK_IS_W(text[i-1])) + _REWIND_OR_ABORT(); + else if (i != 0 && text[i] != 0 && _REGEX_CHECK_IS_W(text[i-1]) == _REGEX_CHECK_IS_W(text[i])) + _REWIND_OR_ABORT(); + } + else if (tokens[k].kind == REMIMU_KIND_NBOUND) + { + if (i == 0 && _REGEX_CHECK_IS_W(text[i])) + _REWIND_OR_ABORT(); + else if (i != 0 && text[i] == 0 && _REGEX_CHECK_IS_W(text[i-1])) + _REWIND_OR_ABORT(); + else if (i != 0 && text[i] != 0 && _REGEX_CHECK_IS_W(text[i-1]) != _REGEX_CHECK_IS_W(text[i])) + _REWIND_OR_ABORT(); + } + else + { + // deliberately unmatchable token (e.g. a{0}, a{0,0}) + if (tokens[k].count_hi == 1) + { + if (tokens[k].kind == REMIMU_KIND_OPEN || tokens[k].kind == REMIMU_KIND_NCOPEN) + k += tokens[k].pair_offset; + else + k += 1; + continue; + } + + if (tokens[k].kind == REMIMU_KIND_OPEN || tokens[k].kind == REMIMU_KIND_NCOPEN) + { + if (!just_rewinded) + { + IF_VERBOSE(printf("hit OPEN. i is %zd, depth is %d\n", i, stack_n);) + // need this to be able to detect and reject zero-size matches + //q_group_state[tokens[k].mask[0]] = i; + + // if we're lazy and the min length is 0, we need to try the non-group case first + if ((tokens[k].mode & REMIMU_MODE_LAZY) && (tokens[k].count_lo == 0 || q_group_accepts_zero[tokens[k + tokens[k].pair_offset].mask[0]])) + { + IF_VERBOSE(puts("trying non-group case first.....");) + range_min = 0; + range_max = 0; + _REWIND_DO_SAVE(k); + k += tokens[k].pair_offset; // automatic += 1 will put us past the matching ) + } + else + { + range_min = 1; + range_max = 0; + _REWIND_DO_SAVE(k); + } + } + else + { + IF_VERBOSE(printf("rewinded into OPEN. i is %zd, depth is %d\n", i, stack_n);) + just_rewinded = 0; + + uint64_t orig_k = k; + + IF_VERBOSE(printf("--- trying to try another alternation, start k is %d, rmin is %zu\n", k, range_min);) + + if (range_min != 0) + { + IF_VERBOSE(puts("rangemin is not zero. checking...");) + k += range_min; + IF_VERBOSE(printf("start kind: %d\n", tokens[k].kind);) + IF_VERBOSE(printf("before start kind: %d\n", tokens[k-1].kind);) + if (tokens[k-1].kind == REMIMU_KIND_OR) + k += tokens[k-1].pair_offset - 1; + else if (tokens[k-1].kind == REMIMU_KIND_OPEN || tokens[k-1].kind == REMIMU_KIND_NCOPEN) + k += tokens[k-1].mask[15] - 1; + + IF_VERBOSE(printf("kamakama %d %d\n", k, tokens[k].kind);) + + if (tokens[k].kind == REMIMU_KIND_END) // unbalanced parens + return -3; + + IF_VERBOSE(printf("---?!?! %d, %d\n", k, q_group_state[tokens[k].mask[0]]);) + if (tokens[k].kind == REMIMU_KIND_CLOSE) + { + IF_VERBOSE(puts("!!~!~!~~~~!!~~!~ hit CLOSE. rewinding");) + // do nothing and continue on if we don't need this group + if (tokens[k].count_lo == 0 || q_group_accepts_zero[tokens[k].mask[0]]) + { + IF_VERBOSE(puts("continuing because we don't need this group");) + q_group_state[tokens[k].mask[0]] = 0; + + if (!(tokens[k].mode & REMIMU_MODE_LAZY)) + q_group_stack[tokens[k].mask[0]] = 0; + + continue; + } + // otherwise go to the last point before the group + else + { + IF_VERBOSE(puts("going to last point before this group");) + _REWIND_OR_ABORT(); + continue; + } + } + + REMIMU_ASSERT(tokens[k].kind == REMIMU_KIND_OR); + } + + IF_VERBOSE(printf("--- FOUND ALTERNATION for paren at k %zd at k %d\n", orig_k, k);) + + ptrdiff_t k_diff = k - orig_k; + range_min = k_diff + 1; + + IF_VERBOSE(puts("(saving in paren after rewinding and looking for next regex token to check)");) + IF_VERBOSE(printf("%zd\n", range_min);) + _REWIND_DO_SAVE(k - k_diff); + } + } + else if (tokens[k].kind == REMIMU_KIND_CLOSE) + { + // unquantified + if (tokens[k].count_lo == 1 && tokens[k].count_hi == 2) + { + // for captures + uint16_t cap_index = q_group_cap_index[tokens[k].mask[0]]; + if (cap_index != 0xFFFF) + _REWIND_DO_SAVE_DUMMY(k); + } + // quantified + else + { + IF_VERBOSE(puts("closer test.....");) + if (!just_rewinded) + { + uint32_t prev = q_group_stack[tokens[k].mask[0]]; + + IF_VERBOSE(printf("qrqrqrqrqrqrqrq------- k %d, gs %d, gaz %d, i %zd, tklo %d, rmin %zd, tkhi %d, rmax %zd, prev %d, sn %d\n", k, q_group_state[tokens[k].mask[0]], q_group_accepts_zero[tokens[k].mask[0]], i, tokens[k].count_lo, range_min, tokens[k].count_hi, range_max, prev, stack_n);) + + range_max = tokens[k].count_hi; + range_max -= 1; + range_min = q_group_accepts_zero[tokens[k].mask[0]] ? 0 : tokens[k].count_lo; + //REMIMU_ASSERT(q_group_state[tokens[k + tokens[k].pair_offset].mask[0]] <= i); + //if (prev) REMIMU_ASSERT(rewind_stack[prev].i <= i); + IF_VERBOSE(printf("qzqzqzqzqzqzqzq------- rmin %zd, rmax %zd\n", range_min, range_max);) + + // minimum requirement not yet met + if (q_group_state[tokens[k].mask[0]] + 1 < range_min) + { + IF_VERBOSE(puts("continuing minimum matches for a quantified group");) + q_group_state[tokens[k].mask[0]] += 1; + _REWIND_DO_SAVE(k); + + k += tokens[k].pair_offset; // back to start of group + k -= 1; // ensure we actually hit the group node next and not the node after it + continue; + } + // maximum allowance exceeded + else if (tokens[k].count_hi != 0 && q_group_state[tokens[k].mask[0]] + 1 > range_max) + { + IF_VERBOSE(printf("hit maximum allowed instances of a quantified group %d %zd\n", q_group_state[tokens[k].mask[0]], range_max);) + range_max -= 1; + _REWIND_OR_ABORT(); + continue; + } + + // fallback case to detect zero-length matches when we backtracked into the inside of this group + // after an attempted parse of a second copy of itself + uint8_t force_zero = 0; + if (prev != 0 && rewind_stack[prev].i > i) + { + // find matching open paren + size_t n = stack_n - 1; + while (n > 0 && rewind_stack[n].k != k + tokens[k].pair_offset) + n -= 1; + REMIMU_ASSERT(n > 0); + if (rewind_stack[n].i == i) + force_zero = 1; + } + + // reject zero-length matches + if ((force_zero || (prev != 0 && rewind_stack[prev].i == i))) // && q_group_state[tokens[k].mask[0]] > 0 + { + IF_VERBOSE(printf("rejecting zero-length match..... %d %zd %zd\n", force_zero, rewind_stack[prev].i, i);) + IF_VERBOSE(printf("%d (k: %d)\n", q_group_state[tokens[k].mask[0]], k);) + + q_group_accepts_zero[tokens[k].mask[0]] = 1; + _REWIND_OR_ABORT(); + //range_max = q_group_state[tokens[k].mask[0]]; + //range_min = 0; + } + else if (tokens[k].mode & REMIMU_MODE_LAZY) // lazy + { + IF_VERBOSE(printf("nidnfasidfnidfndifn------- %d, %d, %zd\n", q_group_state[tokens[k].mask[0]], tokens[k].count_lo, range_min);) + if (prev) + IF_VERBOSE(printf("lazy doesn't think it's zero-length. prev i %zd vs i %zd (depth %d)\n", rewind_stack[prev].i, i, stack_n);) + // continue on to past the group; group retry is in rewind state + q_group_state[tokens[k].mask[0]] += 1; + _REWIND_DO_SAVE(k); + q_group_state[tokens[k].mask[0]] = 0; + } + else // greedy + { + IF_VERBOSE(puts("wahiwahi");) + // clear unwanted memory if possessive + if ((tokens[k].mode & REMIMU_MODE_POSSESSIVE)) + { + uint32_t k2 = k; + + // special case for first, only rewind to (, not to ) + if (q_group_state[tokens[k].mask[0]] == 0) + k2 = k + tokens[k].pair_offset; + + if (stack_n == 0) + return -1; + stack_n -= 1; + + while (stack_n > 0 && rewind_stack[stack_n].k != k2) + stack_n -= 1; + + if (stack_n == 0) + return -1; + } + // continue to next match if sane + if ((uint32_t)q_group_state[tokens[k + tokens[k].pair_offset].mask[0]] < (uint32_t)i) + { + IF_VERBOSE(puts("REWINDING FROM GREEDY NON-REWIND CLOSER");) + q_group_state[tokens[k].mask[0]] += 1; + _REWIND_DO_SAVE(k); + k += tokens[k].pair_offset; // back to start of group + k -= 1; // ensure we actually hit the group node next and not the node after it + } + else + IF_VERBOSE(puts("CONTINUING FROM GREEDY NON-REWIND CLOSER");) + } + } + else + { + IF_VERBOSE(puts("IN CLOSER REWIND!!!");) + just_rewinded = 0; + + if (tokens[k].mode & REMIMU_MODE_LAZY) + { + // lazy rewind: need to try matching the group again + _REWIND_DO_SAVE_DUMMY(k); + q_group_stack[tokens[k].mask[0]] = stack_n; + k += tokens[k].pair_offset; // back to start of group + k -= 1; // ensure we actually hit the group node next and not the node after it + } + else + { + // greedy. if we're going to go outside the acceptable range, rewind + IF_VERBOSE(printf("kufukufu %d %zd\n", tokens[k].count_lo, range_min);) + //uint64_t old_i = i; + if (q_group_state[tokens[k].mask[0]] < range_min && !q_group_accepts_zero[tokens[k].mask[0]]) + { + IF_VERBOSE(printf("rewinding from greedy group because we're going to go out of range (%d vs %zd)\n", q_group_state[tokens[k].mask[0]], range_min);) + //i = old_i; + _REWIND_OR_ABORT(); + } + // otherwise continue on to past the group + else + { + IF_VERBOSE(puts("continuing past greedy group");) + q_group_state[tokens[k].mask[0]] = 0; + + // for captures + uint16_t cap_index = q_group_cap_index[tokens[k].mask[0]]; + if (cap_index != 0xFFFF) + _REWIND_DO_SAVE_DUMMY(k); + } + } + } + } + } + else if (tokens[k].kind == REMIMU_KIND_OR) + { + IF_VERBOSE(printf("hit OR at %d. adding %d\n", k, tokens[k].pair_offset);) + k += tokens[k].pair_offset; + k -= 1; + } + else if (tokens[k].kind == REMIMU_KIND_NORMAL) + { + if (!just_rewinded) + { + uint64_t n = 0; + // do whatever the obligatory minimum amount of matching is + uint64_t old_i = i; + while (n < tokens[k].count_lo && text[i] != 0 && _REGEX_CHECK_MASK(k, text[i])) + { + i += 1; + n += 1; + } + if (n < tokens[k].count_lo) + { + IF_VERBOSE(printf("non-match A. rewinding (token %d)\n", k);) + i = old_i; + _REWIND_OR_ABORT(); + continue; + } + + if (tokens[k].mode & REMIMU_MODE_LAZY) + { + range_min = n; + range_max = tokens[k].count_hi - 1; + _REWIND_DO_SAVE(k); + } + else + { + uint64_t ilimit = tokens[k].count_hi; + if (ilimit == 0) + ilimit = ~ilimit; + range_min = n; + while (text[i] != 0 && _REGEX_CHECK_MASK(k, text[i]) && n + 1 < ilimit) + { + IF_VERBOSE(printf("match!! (%c)\n", text[i]);) + i += 1; + n += 1; + } + range_max = n; + IF_VERBOSE(printf("set rmin to %zd and rmax to %zd on entry into normal greedy token with k %d\n", range_min, range_max, k);) + if (!(tokens[k].mode & REMIMU_MODE_POSSESSIVE)) + _REWIND_DO_SAVE(k); + } + } + else + { + just_rewinded = 0; + + if (tokens[k].mode & REMIMU_MODE_LAZY) + { + uint64_t ilimit = range_max; + if (ilimit == 0) + ilimit = ~ilimit; + + if (_REGEX_CHECK_MASK(k, text[i]) && text[i] != 0 && range_min < ilimit) + { + IF_VERBOSE(printf("match2!! (%c) (k: %d)\n", text[i], k);) + i += 1; + range_min += 1; + _REWIND_DO_SAVE(k); + } + else + { + IF_VERBOSE(printf("core rewind lazy (k: %d)\n", k);) + _REWIND_OR_ABORT(); + } + } + else + { + //IF_VERBOSE(printf("comparing rmin %zd and rmax %zd token with k %d\n", range_min, range_max, k);) + if (range_max > range_min) + { + IF_VERBOSE(printf("greedy normal going back (k: %d)\n", k);) + i -= 1; + range_max -= 1; + _REWIND_DO_SAVE(k); + } + else + { + IF_VERBOSE(printf("core rewind greedy (k: %d)\n", k);) + _REWIND_OR_ABORT(); + } + } + } + } + else + { + fprintf(stderr, "unimplemented token kind %d\n", tokens[k].kind); + REMIMU_ASSERT(0); + } + } + //printf("k... %d\n", k); + } + + if (caps != 0) + { + //printf("stack_n: %d\n", stack_n); + fflush(stdout); + for (size_t n = 0; n < stack_n; n++) + { + RegexMatcherState s = rewind_stack[n]; + int kind = tokens[s.k].kind; + if (kind == REMIMU_KIND_OPEN || kind == REMIMU_KIND_CLOSE) + { + uint16_t cap_index = q_group_cap_index[tokens[s.k].mask[0]]; + if (cap_index == 0xFFFF) + continue; + if (tokens[s.k].kind == REMIMU_KIND_OPEN) + cap_pos[cap_index] = s.i; + else if (cap_pos[cap_index] >= 0) + cap_span[cap_index] = s.i - cap_pos[cap_index]; + } + } + // re-deinitialize capture positions that have no associated capture span + for (size_t n = 0; n < caps; n++) + { + if (cap_span[n] == -1) + cap_pos[n] = -1; + } + } + + #undef _REWIND_DO_SAVE + #undef _REWIND_OR_ABORT + #undef _REGEX_CHECK_IS_W + #undef _P_TEXT_HIGHLIGHTED + #undef IF_VERBOSE + + return i; +} + +REMIMU_FUNC_VISIBILITY void print_regex_tokens(RegexToken * tokens) +{ + const char * kind_to_str[] = { + "NORMAL", + "OPEN", + "NCOPEN", + "CLOSE", + "OR", + "CARET", + "DOLLAR", + "BOUND", + "NBOUND", + "END", + }; + const char * mode_to_str[] = { + "GREEDY", + "POSSESS", + "LAZY", + }; + for (int k = 0;; k++) + { + printf("%s\t%s\t", kind_to_str[tokens[k].kind], mode_to_str[tokens[k].mode]); + + int c_old = -1; + for (int c = 0; c < (tokens[k].kind ? 0 : 256); c++) + { + #define _PRINT_C_SMART(c) { \ + if (c >= 0x20 && c <= 0x7E) \ + printf("%c", c); \ + else \ + printf("\\x%02x", c); \ + } + + if (_REGEX_CHECK_MASK(k, c)) + { + if (c_old == -1) + c_old = c; + } + else if (c_old != -1) + { + if (c - 1 == c_old) + { + _PRINT_C_SMART(c_old) + c_old = -1; + } + else if (c - 2 == c_old) + { + _PRINT_C_SMART(c_old) + _PRINT_C_SMART(c_old + 1) + c_old = -1; + } + else + { + _PRINT_C_SMART(c_old) + printf("-"); + _PRINT_C_SMART(c - 1) + c_old = -1; + } + } + } + + /* + printf("\t"); + for (int i = 0; i < 16; i++) + printf("%04x", tokens[k].mask[i]); + */ + + printf("\t{%d,%d}\t(%d)\n", tokens[k].count_lo, tokens[k].count_hi - 1, tokens[k].pair_offset); + + if (tokens[k].kind == REMIMU_KIND_END) + break; + } +} + +#undef _REGEX_CHECK_MASK + +#endif //INCLUDE_REMIMU