MMC: NULL pointer dereference in mmc_complete during sustained eMMC write on SC598

[katiaTsaruk]

Kernel crash during sustained write to eMMC on EV-SC598-SOM-EZKIT, kernel 6.12.0.

Reproduction

Streaming a large image (~7.6 GB) from the network directly to the eMMC crashes the kernel at varying progress points (3%–35%):

wget -O - http://<host>:8000/emmc.img | dd of=/dev/mmcblk0 bs=1M

Workaround

Compressing the image and decompressing inline throttles the write rate enough to avoid the crash. The total transfer drops from 7.6 GB to ~25 MB:

wget -O - http://<host>:8000/emmc.img.gz | gunzip | dd of=/dev/mmcblk0 bs=64K

Implemented in br2-external (to be reverted once fixed):

• bootstrap/overlay/init#L69
• post-image.sh#L4

Trace 1 — NULL pointer dereference

[   68.839741] Mem abort info:
[   68.842381]   ESR = 0x0000000086000006
[   68.846100]   EC = 0x21: IABT (current EL), IL = 32 bits
[   68.851464]   SET = 0, FnV = 0
[   68.854454]   EA = 0, S1PTW = 0
[   68.857548]   FSC = 0x06: level 2 translation fault
[   68.862443] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000092bb0000
[   68.868908] [0000000000000000] pgd=0800000092bd3003, p4d=0800000092bd3003, pud=0800000092bd4003, pmd=0000000000000000
[   68.879445] Internal error: Oops: 0000000086000006 [#1] PREEMPT SMP
[   68.885673] Modules linked in:
[   68.888714] CPU: 0 UID: 0 PID: 30 Comm: kworker/0:1H Not tainted 6.12.0 #4
[   68.895566] Hardware name: ADI 64-bit SC598 SOM EZ Kit (DT)
[   68.901129] Workqueue:  0x0 (mmc_complete)
[   68.905202] pstate: 60000009 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   68.912145] pc : 0x0
[   68.914315] lr : 0x0
[   68.916485] sp : ffff800081413e20
[   68.919783] x29: 0000000000000000 x28: 0000000000000000 x27: 0000000000000000
[   68.926901] x26: 0000000000000000 x25: 0000000000000000 x24: ffff00009df41c80
[   68.934018] x23: ffff000091c257c0 x22: ffff000091cf1040 x21: ffff00009df41c58
[   68.941136] x20: ffff000091cf1040 x19: ffff000091c25800 x18: 0000000000000000
[   68.948254] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800080ab0020
[   68.955371] x14: 0000000efd42eeb0 x13: 0000000000000001 x12: 0000000efd451088
[   68.962489] x11: 0000000000000000 x10: 0000000000000a10 x9 : ffff800081413d60
[   68.969606] x8 : ffff000091cf1ab0 x7 : ffff000091cf10c0 x6 : 0000000000000000
[   68.976724] x5 : 0000000000000001 x4 : ffff0000928a9ae0 x3 : 1000000000000000
[   68.983842] x2 : 0000000004208060 x1 : 0000000004208060 x0 : 0000000000000000
[   68.990960] Call trace:
[   68.993390]  0x0
[   68.995220] Code: ???????? ???????? ???????? ???????? (????????)
[   69.001290] ---[ end trace 0000000000000000 ]---

Trace 2 — workqueue WARNING

[   80.542703] WARNING: CPU: 0 PID: 124 at kernel/workqueue.c:788 set_work_data+0x8/0x14
[   80.550511] Modules linked in:
[   80.553553] CPU: 0 UID: 0 PID: 124 Comm: dd Not tainted 6.12.0 #4
[   80.559624] Hardware name: ADI 64-bit SC598 SOM EZ Kit (DT)
[   80.565181] pstate: 600000c9 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   80.572124] pc : set_work_data+0x8/0x14
[   80.575943] lr : __queue_work+0x2ac/0x318
[   80.579935] sp : ffff800080003860
[   80.583234] x29: ffff800080003860 x28: ffff000092899040 x27: 000000000670003e
[   80.590351] x26: ffff800080f1c408 x25: 0000000000000200 x24: ffff00009214d668
[   80.597469] x23: ffff000092173368 x22: ffff000092173360 x21: ffff00009df41c58
[   80.604587] x20: ffff00009214d600 x19: ffff00009214d600 x18: 0000000000000000
[   80.611704] x17: 7ceadbfb87030a08 x16: 01010000d0e30320 x15: 1080127062d825ca
[   80.618822] x14: 0000000000000000 x13: ffff000098006ec0 x12: 0000000000000000
[   80.625939] x11: 000000000000dd86 x10: ffff800080f2c150 x9 : 0000000000000003
[   80.633057] x8 : ffff800080f2c148 x7 : 0000000000000000 x6 : 0000000000000000
[   80.640175] x5 : ffff00009df41c58 x4 : ffff000091800040 x3 : ffff00009df41c80
[   80.647292] x2 : 0000000000600000 x1 : ffff00009214d605 x0 : ffff000092173360
[   80.654411] Call trace:
[   80.656842]  set_work_data+0x8/0x14
[   80.660312]  queue_work_on+0x90/0x98
[   80.663871]  mmc_blk_mq_req_done+0x8c/0x130
[   80.668037]  mmc_request_done+0x100/0x110
[   80.672030]  sdhci_irq+0xab0/0xad8
[   80.675415]  __handle_irq_event_percpu+0x4c/0x100
[   80.680103]  handle_irq_event_percpu+0x14/0x44
[   80.684530]  handle_irq_event+0x44/0x78
[   80.688349]  handle_fasteoi_irq+0xb4/0x13c
[   80.692428]  handle_irq_desc+0x1c/0x30
[   80.696161]  generic_handle_domain_irq+0x18/0x20
[   80.700761]  gic_handle_irq+0xa0/0x100
[   80.704493]  do_interrupt_handler+0x7c/0x84
[   80.708660]  el1_interrupt+0x30/0x5c
[   80.712219]  el1h_64_irq_handler+0x14/0x1c
[   80.716298]  el1h_64_irq+0x64/0x68
[   80.719683]  stmmac_napi_poll_rx+0x2d8/0xb60
[   80.723937]  __napi_poll.constprop.0+0x30/0x154
[   80.728450]  net_rx_action+0x164/0x278
[   80.732183]  handle_softirqs+0x1b8/0x1fc
[   80.736089]  __do_softirq+0x10/0x18
[   80.739561]  ____do_softirq+0xc/0x14
[   80.743119]  call_on_irq_stack+0x24/0x34
[   80.747026]  do_softirq_own_stack+0x18/0x20
[   80.751192]  __irq_exit_rcu+0x60/0x9c
[   80.754837]  irq_exit_rcu+0xc/0x14
[   80.758223]  el1_interrupt+0x34/0x5c
[   80.761781]  el1h_64_irq_handler+0x14/0x1c
[   80.765861]  el1h_64_irq+0x64/0x68
[   80.769246]  prepare_to_wait+0x10/0x74
[   80.772979]  __wait_on_bit+0x38/0xb4
[   80.776537]  out_of_line_wait_on_bit+0x60/0x88
[   80.780964]  __wait_on_buffer+0x24/0x30
[   80.784783]  wait_on_buffer+0x14/0x20
[   80.788429]  __block_write_begin_int+0x5a0/0x5dc
[   80.793029]  iomap_write_begin+0x210/0x3f0
[   80.797109]  iomap_file_buffered_write+0x160/0x33c
[   80.801883]  blkdev_write_iter+0x1cc/0x1f8
[   80.805963]  vfs_write+0x154/0x1d0
[   80.809348]  ksys_write+0x78/0xe0
[   80.812646]  __arm64_sys_write+0x14/0x1c
[   80.816552]  invoke_syscall+0x60/0xe4
[   80.820198]  el0_svc_common.constprop.0+0xb0/0xcc
[   80.824885]  do_el0_svc+0x18/0x20
[   80.828184]  el0_svc+0x58/0x84
[   80.831221]  el0t_64_sync_handler+0x80/0x124
[   80.835475]  el0t_64_sync+0x14c/0x150
[   80.839122] ---[ end trace 0000000000000000 ]---

[CalebEthridgeADI]

I can replicate the work queue warning around 50% of the time if I am writing the eMMC from a NFS. I have never seen the Null pointer dereference.

For reference, the simple eMMC init that I run with the fitImage from Yocto

[nunojsa]

I can replicate the work queue warning around 50% of the time if I am writing the eMMC from a NFS. I have never seen the Null pointer dereference.

Maybe worth trying to backport the following:

https://lore.kernel.org/all/20251117000132.12131-1-jszhang@kernel.org/

git log does not relate to the above but this kind of stale interrupts issue could very well be the reason for this nasty race on handling the work item