Skip to content

Use sys.stderr.write() to fix stderr interleaving under make -jN#194

Merged
robherring merged 1 commit into
devicetree-org:mainfrom
aditya-qti:fix-interleaving-stderr-output
May 7, 2026
Merged

Use sys.stderr.write() to fix stderr interleaving under make -jN#194
robherring merged 1 commit into
devicetree-org:mainfrom
aditya-qti:fix-interleaving-stderr-output

Conversation

@aditya-qti

Copy link
Copy Markdown

Example interleaved stderr:

/home/patchwise/build/arch/arm64/boot/dts/microchip/sparx5_pcb134_emmc.dtb: syscon@600000000 (microchip,sparx5-cpu-syscon): mux-controller:#mux-control-cells: 1 was expected
	from schema $id: http://devicetree.org/schemas/soc/microchip/microchip,sparx5-cpu-syscon.yaml/home/patchwise/build/arch/arm64/boot/dts/microchip/sparx5_pcb134.dtb: syscon@600000000 (microchip,sparx5-cpu-syscon): mux-controller:#mux-control-cells: 1 was expected
	from schema $id: http://devicetree.org/schemas/soc/microchip/microchip,sparx5-cpu-syscon.yaml

This issue was found when running make -j$(nproc) dtbs_check in PatchWise (automates patch review and static analysis):

https://github.com/qualcomm/PatchWise/blob/main/patchwise/patch_review/static_analysis/dtbs_check.py#L36-L51

This makes it difficult for automations to find the diff between before and after errors of the current commit.

The root cause is that on Python's default line-buffered sys.stderr, print() makes 2 write() syscalls when
the input is a multi-line string, and a sibling process' write() can execute between those two syscalls.

single-line print() -> 1 write() syscall.
ex: print("msg\n") -> 1 write()

multi-line print() -> 2 write() syscalls.
ex: print(" line1 \n line2 \n line3 ") -> 2 write() syscalls

  • one for "line1 \n line2 \n line3"
  • one for "\n" that print() adds at the end

Python's sys.stderr is line-buffered by default, so the pending buffer gets flushed when "\n" is seen.

p1                                             p2
write(" line1 \n line2 \n line3 ")
                                            write("msg\n")
write("\n")

Output:

 line1
 line2
 line3 msg

This can be proven by running strace on something like this:

import sys


def banner(label):
    sys.stderr.write(f"=== {label} ===\n")
    sys.stderr.flush()


banner("print(single-line)")
print("ABCDEFGH" * 4, file=sys.stderr)
sys.stderr.flush()

banner("print(multi-line)")
print("LINE1-XX\nLINE2-XX\nLINE3-XX", file=sys.stderr)
sys.stderr.flush()

banner("write(multi-line + '\\n')")
sys.stderr.write("LINE1-XX\nLINE2-XX\nLINE3-XX" + "\n")
sys.stderr.flush()
$ strace -f -e trace=write -e signal=none -o trace_print.strace python3 trace_print.py 2>trace_print.out
$ grep 'write(2,' trace_print.strace
2804383 write(2, "=== print(single-line) ===\n", 27) = 27
2804383 write(2, "ABCDEFGHABCDEFGHABCDEFGHABCDEFGH"..., 33) = 33
2804383 write(2, "=== print(multi-line) ===\n", 26) = 26
2804383 write(2, "LINE1-XX\nLINE2-XX\nLINE3-XX", 26) = 26
2804383 write(2, "\n", 1)               = 1
2804383 write(2, "=== write(multi-line + '\\n') ==="..., 33) = 33
2804383 write(2, "LINE1-XX\nLINE2-XX\nLINE3-XX\n", 27) = 27

$ cat trace_print.out
=== print(single-line) ===
ABCDEFGHABCDEFGHABCDEFGHABCDEFGH
=== print(multi-line) ===
LINE1-XX
LINE2-XX
LINE3-XX
=== write(multi-line + '\n') ===
LINE1-XX
LINE2-XX
LINE3-XX

Fix the issue by using sys.stderr.write(text + "\n") instead of print(text, file=sys.stderr),
since sys.stderr.write(multi_line_text) makes a single write() call.

@robherring

Copy link
Copy Markdown
Member

Can we just fix the multi-line cases and leave the other print()'s alone.

Comment thread dtschema/schema.py Outdated
@aditya-qti

Copy link
Copy Markdown
Author

single-line print() -> 1 write() syscall.
ex: print("msg\n") -> 1 write()

I meant to write print("msg") here :(

@robherring

Copy link
Copy Markdown
Member

Please squash the reverted parts so I can review it.

single-line print() -> 1 write() syscall.
ex: print("msg") -> 1 write()

multi-line print() -> 2 write() syscalls.
ex: print(" line1 \n line2 \n line3 ") -> 2 write() syscalls - one for
"line1 \n line2 \n line3" and one for "\n" that print() adds at the end.

Python's sys.stderr is line-buffered by default, so the pending buffer
gets flushed when "\n" is seen.

p1                                             p2
write(" line1 \n line2 \n line3 ")
                                            write("msg\n")
write("\n")

Output:
 line1
 line2
 line3 msg

Fix the issue by using sys.stderr.write(text + "\n") instead of
print(text, file=sys.stderr), since sys.stderr.write(multi_line_text)
makes a single write() call.

Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Aditya Chillara <achillar@qti.qualcomm.com>
@aditya-qti aditya-qti force-pushed the fix-interleaving-stderr-output branch from a886edf to ef09115 Compare May 7, 2026 13:29
@robherring robherring merged commit e56d0c4 into devicetree-org:main May 7, 2026
7 checks passed
@aditya-qti aditya-qti deleted the fix-interleaving-stderr-output branch May 8, 2026 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants