Describe the bug
Any file with CRLF (\r\n) line endings fails to parse. This is not specific to one construct — a bare attribute, a block, and a quoted string all fail identically. A single CRLF line in an otherwise-LF file is enough to fail the whole file.
Windows-authored .tf files, or files checked out with core.autocrlf=true, are therefore unparseable.
Software:
- python-hcl2:
main (1432aea)
- Python 3.9, lark 1.3.1
- Long-standing, not a v8 regression — see below
Snippet of HCL2 code causing the unexpected behaviour:
...saved with CRLF line endings, i.e. the literal input "a = 1\r\n".
Expected behavior
{'a': 1} — the same result as the LF form.
Actual behavior
lark.exceptions.UnexpectedToken: Unexpected token Token('STRING_CHARS', '\r\n...
Reproduction across constructs:
| Input |
Result |
"a = 1\r\n" |
raises |
"locals {\r\n a = 1\r\n}\r\n" |
raises |
'a = "x"\r\n' |
raises |
"a = 1\nb = 2\r\nc = 3\n" (one CRLF line) |
raises |
"a = 1\n" (LF control) |
{'a': 1} |
Root cause
hcl2/hcl2.lark — no terminal matches \r:
NL_OR_COMMENT: /\n[ \t]*/ | /#.*\n/ | /\/\/.*\n/ | /\/\*(.|\n)*?(\*\/)/
%ignore /[ \t]+/
The newline terminal matches \n and then only spaces/tabs, and the ignore rule covers only [ \t]+. A \r preceding the \n matches nothing, so it falls through to STRING_CHARS and the parse fails.
Not a regression. Both lines are byte-identical pre-8.x (git show 21b7cf8^:hcl2/hcl2.lark), so CRLF has never been supported.
Possible directions, neither verified:
- Accept
\r in the newline terminal (e.g. /\r?\n[ \t]*/) and in the comment alternatives.
- Normalize line endings on input in
api.load/loads before handing text to the parser.
The second is less invasive but changes what the reconstructor can reproduce byte-for-byte, so it interacts with round-trip fidelity — a CRLF file would round-trip as LF. Worth a deliberate decision rather than a quick patch.
Impact
High for anyone on Windows or with core.autocrlf=true; zero for everyone else. The failure is loud, so there is no silent data loss.
Workaround
Normalize before parsing:
hcl2.loads(text.replace("\r\n", "\n"))
Found while reviewing #309/#312 (empty heredocs), where CRLF heredocs also failed. I checked whether it was heredoc-specific: it is not, which is why this is filed on its own.
Describe the bug
Any file with CRLF (
\r\n) line endings fails to parse. This is not specific to one construct — a bare attribute, a block, and a quoted string all fail identically. A single CRLF line in an otherwise-LF file is enough to fail the whole file.Windows-authored
.tffiles, or files checked out withcore.autocrlf=true, are therefore unparseable.Software:
main(1432aea)Snippet of HCL2 code causing the unexpected behaviour:
...saved with CRLF line endings, i.e. the literal input
"a = 1\r\n".Expected behavior
{'a': 1}— the same result as the LF form.Actual behavior
Reproduction across constructs:
"a = 1\r\n""locals {\r\n a = 1\r\n}\r\n"'a = "x"\r\n'"a = 1\nb = 2\r\nc = 3\n"(one CRLF line)"a = 1\n"(LF control){'a': 1}Root cause
hcl2/hcl2.lark— no terminal matches\r:The newline terminal matches
\nand then only spaces/tabs, and the ignore rule covers only[ \t]+. A\rpreceding the\nmatches nothing, so it falls through toSTRING_CHARSand the parse fails.Not a regression. Both lines are byte-identical pre-8.x (
git show 21b7cf8^:hcl2/hcl2.lark), so CRLF has never been supported.Possible directions, neither verified:
\rin the newline terminal (e.g./\r?\n[ \t]*/) and in the comment alternatives.api.load/loadsbefore handing text to the parser.The second is less invasive but changes what the reconstructor can reproduce byte-for-byte, so it interacts with round-trip fidelity — a CRLF file would round-trip as LF. Worth a deliberate decision rather than a quick patch.
Impact
High for anyone on Windows or with
core.autocrlf=true; zero for everyone else. The failure is loud, so there is no silent data loss.Workaround
Normalize before parsing:
Found while reviewing #309/#312 (empty heredocs), where CRLF heredocs also failed. I checked whether it was heredoc-specific: it is not, which is why this is filed on its own.