Task Cleanup leaks 160B of Memory

[smallbraineng]

Bug description

If a FreeRTOS task is created with ThreadSpawnConfig, it leaks 160 bytes in memory when cle

• Would you like to work on a fix? [y/n]

To Reproduce

1. Create any task with threadspawnconfig, then join and drop it
2. Check the memory usage before and after

 loop {
        log::info!("\n");
        log::info!("============== BEFORE TEST TASK ===========");
        print_memory_info();
        let test_task = spawn("test\0", 1024 * 10, 1, Some(Core::Core0), move || {
            sleep(Duration::from_secs(3));
        });
        test_task.join().unwrap();
        log::info!("\n");
        log::info!("============== AFTER TEST TASK ===========");
        print_memory_info();
    }

and in print_memory_info,

 log::info!(
            " Free: {} bytes",
            heap_caps_get_free_size(MALLOC_CAP_INTERNAL)
        );

Expected behavior

It to clean up the task fully

Environment

• esp-idf-svc 0.51
• Target device (MCU): esp32s3

[ivmarkov]

Are you sure that you are observing a memory leak caused by spawning (std::thread::spawn) the new thread, and NOT observing the allocation of the thread-local memory for the ThreadSpawnConfig keys itself?

Two experiments you could do to confirm/deny:

• Experiment 1: Do not call spawn but still call the ThreadSpawnConfig setup. If you still see a memory "leak", you are observing the ThreadSpawnConfig local storage allocation in heap, rather than anything else.
• Experiment 2: Do what you do now, but offload the ThreadSpawnConfig as well the subsequent std::thread::spawn call in another, "outer" thread, which is also spawned with std::thread::spawn, but does not have a ThreadSpawnConfig setup. Once both threads are joined, you should not see a "memory leak" anymore, as the ThreadSpawnConfig local storage of the outer thread should die when this thread is joined.

[benhansen-io]

I think I am seeing the same or similar issue with plain std::thread::spawn

Reposting from the closed esp-rs/rust issue:

    for i in 0..1000 {
        let handle = std::thread::Builder::new()
            .stack_size(6000)
            .spawn(move || {
                // No op
            })
            .unwrap();
        handle.join().unwrap();
        memory::print_mem_info(&format!("after {i}"));
    }

I expected to be able to spawn and then join a thread to reclaim all memory.

The loop is unable to finish. Each iteration consumes about 150 bytes that is not freed. Eventually the device runs out of memory.

Logs:

2025-05-30 15:40:14 I (11073) ten_os::commands: executing command: test
2025-05-30 15:40:14 I (11073) rust_boppo::memory: HEAP: 450.1/2419k (19%) DMA: 258.6/317k (81%) Stack HWM: 25848: after 0
2025-05-30 15:40:14 I (11083) rust_boppo::memory: HEAP: 450.2/2419k (19%) DMA: 258.7/317k (81%) Stack HWM: 25848: after 1
2025-05-30 15:40:14 I (11083) rust_boppo::memory: HEAP: 450.4/2419k (19%) DMA: 258.9/317k (82%) Stack HWM: 25848: after 2
2025-05-30 15:40:14 I (11083) rust_boppo::memory: HEAP: 450.6/2419k (19%) DMA: 259.0/317k (82%) Stack HWM: 25848: after 3
2025-05-30 15:40:14 I (11093) rust_boppo::memory: HEAP: 450.7/2419k (19%) DMA: 259.2/317k (82%) Stack HWM: 25848: after 4
....
....
2025-05-30 15:40:15 I (12333) rust_boppo::memory: HEAP: 504.7/2415k (21%) DMA: 313.1/313k (100%) Stack HWM: 25816: after 350
2025-05-30 15:40:15 I (12333) rust_boppo::memory: HEAP: 504.8/2415k (21%) DMA: 313.1/313k (100%) Stack HWM: 25816: after 351
2025-05-30 15:40:15 I (12343) rust_boppo::memory: HEAP: 505.0/2415k (21%) DMA: 313.1/313k (100%) Stack HWM: 25816: after 352
2025-05-30 15:40:15 I (12343) rust_boppo::memory: HEAP: 505.1/2415k (21%) DMA: 313.1/313k (100%) Stack HWM: 25816: after 353
2025-05-30 15:40:15 I (12343) rust_boppo::memory: HEAP: 505.3/2415k (21%) DMA: 313.1/313k (100%) Stack HWM: 25816: after 354
2025-05-30 15:40:15 E (12353) pthread: Failed to create task!
2025-05-30 15:40:15
2025-05-30 15:40:15 thread '<unnamed>' panicked at src/commands.rs:134:14:
2025-05-30 15:40:15 called `Result::unwrap()` on an `Err` value: Os { code: 12, kind: OutOfMemory, message: "Not enough space" }
2025-05-30 15:40:15 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2025-05-30 15:40:15
2025-05-30 15:40:15 abort() was called at PC 0x420aa5de on core 0
--- 2025-05-30 15:40:15 0x420aa5de: panic_abort::__rust_start_panic::abort at ??:?

Meta

rustc --version --verbose:

rustc 1.86.0-nightly (5944fc5cd 2025-04-01) (1.86.0.0)
binary: rustc
commit-hash: 5944fc5cd231b5aaf09b471def70ecefefa4737d
commit-date: 2025-04-01
host: x86_64-unknown-linux-gnu
release: 1.86.0-nightly
LLVM version: 19.1.2

[Vollbrecht]

you run out of memory in under 1 sec with this approach right?

I think before trying to investigate to deep, i would test to a) allocate a bunch of threads b) drop them c) go to sleep ( such that you give the 0 priority idle thread a chance to run) d) check again.

The idle thread does a lot of cleanup, and in your example it has never a change to run because the system crashes after 1 sec.

[benhansen-io]

Gave that a shot, no real change. Still seeing unbounded memory growth.

Code:

    for batch_num in 0..1000 {
        let num_threads_per_batch = 100;
        for i in 0..num_threads_per_batch {
            let handle = std::thread::Builder::new()
                .stack_size(6000)
                .spawn(move || {
                    // do nothing
                })
                .unwrap();
            handle.join().unwrap();
            let thread_num = batch_num * num_threads_per_batch + i;
            memory::print_mem_info(&format!("after {thread_num}"));
        }
        std::thread::sleep(std::time::Duration::from_millis(5000));
    }

Output

2025-06-04 11:24:17 I (11433) ten_os::commands: executing command: test
2025-06-04 11:24:17 I (11443) rust_boppo::memory: HEAP: 448.9/2419k (19%) DMA: 258.4/317k (81%) Stack HWM: 25960: after 0
2025-06-04 11:24:17 I (11443) rust_boppo::memory: HEAP: 449.1/2419k (19%) DMA: 258.6/317k (81%) Stack HWM: 25960: after 1
2025-06-04 11:24:17 I (11443) rust_boppo::memory: HEAP: 449.2/2419k (19%) DMA: 258.7/317k (81%) Stack HWM: 25960: after 2
2025-06-04 11:24:17 I (11453) rust_boppo::memory: HEAP: 449.4/2419k (19%) DMA: 258.9/317k (81%) Stack HWM: 25960: after 3
....
2025-06-04 11:24:17 I (11773) rust_boppo::memory: HEAP: 464.5/2418k (19%) DMA: 273.9/316k (87%) Stack HWM: 25960: after 99
2025-06-04 11:24:22 I (16773) rust_boppo::memory: HEAP: 464.5/2418k (19%) DMA: 274.0/316k (87%) Stack HWM: 25960: after 100
...
2025-06-04 11:24:33 I (27683) rust_boppo::memory: HEAP: 504.3/2415k (21%) DMA: 313.2/313k (100%) Stack HWM: 25960: after 355
2025-06-04 11:24:33 I (27693) rust_boppo::memory: HEAP: 504.5/2415k (21%) DMA: 313.2/313k (100%) Stack HWM: 25960: after 356
2025-06-04 11:24:33 E (27693) pthread: Failed to create task!
2025-06-04 11:24:33
2025-06-04 11:24:33 thread '<unnamed>' panicked at src/commands.rs:136:18:
2025-06-04 11:24:33 called `Result::unwrap()` on an `Err` value: Os { code: 12, kind: OutOfMemory, message: "Not enough space" }
2025-06-04 11:24:33 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2025-06-04 11:24:33
2025-06-04 11:24:33 abort() was called at PC 0x420bb712 on core 0

[benhansen-io]

Let me know if I can help try anything else.

Maybe raw pthread create to see if it leaks? https://docs.esp-rs.org/esp-idf-svc/esp_idf_sys/fn.pthread_create.html

[Vollbrecht]

yeah technically that is what the std library is calling through newlib, but of course you can try to have a go.

[Vollbrecht]

In terms of "where all the code lives" You can have a look at the newlib part of the rust libc here. That enables the biggest part of the rust std support.

Inside the the std library itself its not much if you look here

[ivmarkov]

JFYI this is the implementation of the unix flavor of std::thread that is also used with ESP-IDF. Can't see anything suspicious there, but just for reference.

[Vollbrecht]

to close the cycle on the C side, and to answer the question what newlib version is used. It depends on esp-idf / ROM version.

All rom versions can be found here, and for the rest you can look inside the esp-idf newlib dir in the respective esp-idf version you are using for example here for esp-idf master

[benhansen-io]

pthread_create match with pthread_join does not leak:

    for batch_num in 0..1000 {
        let num_threads_per_batch = 100;
        for i in 0..num_threads_per_batch {
            let mut id: sys::pthread_t = 0;
            let mut attr = sys::pthread_attr_t::default();
            unsafe { assert!(sys::pthread_attr_init(&mut attr as *mut _) == 0) };
            unsafe {
                assert!(
                    sys::pthread_create(
                        &mut id as *mut u32,
                        &attr as *const _,
                        Some(thread_main),
                        null_mut(),
                    ) == 0
                );
            }
            let mut ret_val: *mut std::ffi::c_void = null_mut();
            unsafe { assert!(sys::pthread_join(id, &mut ret_val as *mut _) == 0) };
            unsafe { assert!(sys::pthread_attr_destroy(&mut attr as *mut _) == 0) };

            let thread_num = batch_num * num_threads_per_batch + i;
            memory::print_mem_info(&format!("after {thread_num}"));
        }
        std::thread::sleep(std::time::Duration::from_millis(1000));
    }

extern "C" fn thread_main(_arg: *mut std::ffi::c_void) -> *mut std::ffi::c_void {
    null_mut()
}

Output

2025-06-04 12:16:44 I (3506) ten_os::commands: executing command: test
2025-06-04 12:16:44 I (3506) rust_boppo::memory: HEAP: 448.4/2419k (19%) DMA: 257.5/317k (81%) Stack HWM: 25848: after 0
2025-06-04 12:16:44 I (3516) rust_boppo::memory: HEAP: 448.4/2419k (19%) DMA: 257.5/317k (81%) Stack HWM: 25848: after 1
2025-06-04 12:16:44 I (3516) rust_boppo::memory: HEAP: 448.4/2419k (19%) DMA: 257.5/317k (81%) Stack HWM: 25848: after 2
2025-06-04 12:16:44 I (3516) rust_boppo::memory: HEAP: 448.4/2419k (19%) DMA: 257.5/317k (81%) Stack HWM: 25848: after 3
...
2025-06-04 12:17:10 I (29146) rust_boppo::memory: HEAP: 448.9/2419k (19%) DMA: 257.4/317k (81%) Stack HWM: 25848: after 2011
2025-06-04 12:17:10 I (29146) rust_boppo::memory: HEAP: 448.9/2419k (19%) DMA: 257.4/317k (81%) Stack HWM: 25848: after 2012
2025-06-04 12:17:10 I (29146) rust_boppo::memory: HEAP: 448.9/2419k (19%) DMA: 257.4/317k (81%) Stack HWM: 25848: after 2013
...
2025-06-04 12:17:38 I (57546) rust_boppo::memory: HEAP: 448.9/2419k (19%) DMA: 257.4/317k (81%) Stack HWM: 25848: after 4297
2025-06-04 12:17:38 I (57546) rust_boppo::memory: HEAP: 448.9/2419k (19%) DMA: 257.4/317k (81%) Stack HWM: 25848: after 4298
2025-06-04 12:17:38 I (57556) rust_boppo::memory: HEAP: 448.9/2419k (19%) DMA: 257.4/317k (81%) Stack HWM: 25848: after 4299

[ivmarkov]

Then it must be something that the STD wrapper I referenced above does, which your raw pthread calls don't.

When debugging such problems, I've found the easiest to be to just comment/modify bits and pieces of code in the STD itself (you have its sources in the active toolchain) and then do cargo clean prior to cargo rebuild so that the changes to STD are picked up.

(To make sure you are modifying the correct STD sources, deliberately put an erroneous code in there, in the STD thread.rs impl you would be modifying, so that you could see the compiler bailing out.)

[benhansen-io]

Ok. Thanks for the pointers on how to test std code.

When I comment out set_current inside thread spawn the leak goes away.

The code for set_current uses thread locals and says:

    // Make sure that `crate::rt::thread_cleanup` will be run, which will
    // call `drop_current`.
    crate::sys::thread_local::guard::enable();
    CURRENT.set(thread.into_raw().cast_mut());

Does esp-idf clean up thread_locals differently?